How Data is Used Can Mean the Difference Between Improving or Undermining Educational Outcomes

In the past few weeks, there have been a barrage of media reports about educational achievement and, more generally, life outcomes for the youth of Durham.

The positive news is that these issues are receiving attention, but the downside is that the reports may be more harmful than helpful. At its best, data optimizes decision-making, but at its worst data can be deceptive and divisive.

Specialized knowledge is required to leverage data for decision-making, whereas selectively reporting figures requires some effort but no expertise. In the latter scenario, the ambiguity of statistical assumptions predisposes the audience to personal, as well as, framing bias. Those who go through the effort to produce data often have an agenda, and therefore, have incentives to make claims which imply causes and solutions. Data is dangerous when misused. It can create tension, undermine trust and unity, and result in costly adverse decision-making.

One key characteristic of amateur statistics, aside from lacking an experimental design, is that they do not account for the fact that outcomes are a function of many different variables. For example, schools clearly play a crucial role in influencing academic attainment, but a report drawing relative comparisons between attainment outcomes within or across cities usually implicates unidentified failures of one school district versus another while all but ignoring the effects of transportation, affordable housing, food, healthcare, and social support accessibility, as well as people’s different lived experiences, including traumatic exposure of various kinds.

Reactivity to outcomes is strongly linked to bias and emotion. Making decisions about problems and solutions based exclusively on outcomes is the logical equivalent to going with your gut. Descriptive statistics alone have a tendency to reinforce what we already think we know rather than helping us to gain an objective understanding of the issues because we often overestimate our understanding of the context. Shards of truth may be buried in our presumptions or between the different storylines, but other times the truth isn’t within sight.

If one wanted to know what public schools are doing right and what positive changes could be made, the reported outcomes would not meaningfully increase understanding. This would be like a college basketball coach using the Ratings Percentage Index (RPI) to make game plans. The RPI is simply a function of outcome variables that are influenced by other, more influential variables over a team’s success, such as shot selection, rebounding, ball control and many others.

Similarly, objective inference about the determinants of academic achievement is impossible when we simply have some measure of the output, like grade level proficiency, graduation rates or achievement gaps. Summarized outcomes do not even begin to untangle the multifaceted causal factors of student achievement, or even point to which factors are within the schools’ control and which are shaped by other institutions that govern infrastructure, real estate development, credit markets and criminal justice.

Good intentions often lead to unintended consequences. Calculating outcomes or deriving slightly new definitions of them does not enhance the cultural or intellectual competence of our community, its citizens or the institutions within it.

This is troubling because the extent of harm done with every report that subjectively frames and selectively reports data will never be known. A symptomatic obsession can enable data to have a negative social impact, leading to the proliferation of economic and racial segregation, adverse selection of people and funds from public schools, victim blaming and the marginalization of objectivity. The focus needs to shift from symptoms to solutions.

Data should be collected and analyzed in a way that enables us to separately identify effects on outcomes, including those determinants within the school’s control and those outside so that all can be addressed in order of impact and feasibility. Robust evaluations should yield insight, pointing out specific causal factors that affect outcomes that the schools, nonprofits policy and citizens can address.

Applying a scientific lens to social issues transforms data from punitive to instructive. Careful investigation using valid quantitative methods can help us gain an understanding of the inferences that the data will and will not permit. Through empirical analysis, we have the opportunity to disentangle the effects that different factors have on certain outcomes. This is powerful because it enables us to create informed strategies.

Subsequently, when we know how our potential actions will affect an outcome, a cost-benefit analysis can help decide which evidence should be brought to action. Operating in the public and nonprofit sectors, the cost-benefit analysis goes beyond fiscal considerations to examine social returns. Combining these empirical tools puts us in a position to optimize social welfare. Data or analysis vacant of these characteristics will result in suboptimal decision-making.

An empirical basis for decision-making that respects the complexity of determinants on outcomes and the tradeoffs between various actions or lack of action should be utilized at all levels – from the systemic to the programmatic. A symptomatic focus and a preoccupation with a single area will not result in systemic improvement. As institutions, organizations and programs, our goal should be to improve, which can only be achieved through learning.

Durham has great potential to grow while enhancing the well-being of all, including the most marginalized. Continuous improvement requires the commitment of people in the public, private, and social sectors to work together.

Part of analytical integrity is the acknowledgment that sometimes our data tells us nothing at all. If we truly care about addressing systemic issues, lack of information is a strong argument for why we should build more robust datasets that incorporate variables across institutions and the socio-economic environment. This requires a willingness to coordinate and to learn. Importantly, these actions imply the willingness to change.

The Made in Durham partnership exists to address issues of the highest importance. It is the job of data is to increase the role of evidence in the partnership’s decision-making, and because of the gravity of these decisions, I also feel an ethical accountability to this work.

If we aren’t asking the right questions, data can lead to costly decisions that undermine improvement. As members of the community, we should all be able to ask the right questions to hold decision-makers accountable to analytical standards that drive improvement.

Regardless of what the outcomes show now, or anytime in the future, what we should be asking is: what are the causes of these outcomes, what are their magnitudes, and thus, what can we do to improve.

see the original post on the website: http://madeindurham.org/made/news/perspective-data-for-decision-making-or-deceptive-and-divisive/

If you do not know how to ask the right question, you discover nothing.

In God we trust, all others must bring data.

– W. Edwards Deming

(Image © Benjamin Anderson 2016)

My Graduation Address

I’m very late in adding this, but such has been the case with all of my blogging—especially throughout the duration of my Master’s studies. In fact, I debated whether or not to post this at all. We always want our best to shine through. I think my graduation speech was just one of those occasions in which that simply wasn’t the case. Nonetheless, as my own harshest critic, I know that expectations I hold for myself can be too ambitious, or even unrealistic. From the very moment I was asked to speak, I felt tremendously honored and couldn’t pass on this unique opportunity. Despite preparing it while concurrently juggling final exams, completing my thesis, transitioning into a new job, and reuniting with my parents for the first time in about a year, I still had fun with it—even though, if you notice my anxious demeanor in the video, “fun” might not be the adjective that comes to mind.

A broader graduation ceremony recap can be found here.

As economists, we try to model reality. But the most important model we’ll ever construct is that of who we choose to be and what we desire to become.

—BJA

Personality: How to Build It – Lessons from the Past

I didn’t unlock the secret to success. Although, it’s likely that I’m in a slightly better position than I was yesterday. On a recent and brief trip to Cadaqués for the purpose of a little relaxation and recuperation, I embarked on an unplanned adventure through time that unexpectedly gave me a much-needed lift on my perspective. The mechanism through which most of this change occurred was through the relaxation of the mind by virtue of one of the most fascinating and curious reads I’ve ever had.

It was a book on personality, lent to me by one of my best friends who I met while studying my master’s degree in Barcelona this past year. While skeptical at first, the catalyst for me actually opening the book was my understanding that that this book was published in 1915 and that it had been in the family of my classmate (who hails from Germany) for at least 3 generations. Humorously, I had misunderstood the origins of the book. It was actually a gift to him during his undergraduate studies in the States from one of his professors who was an avid book collector. The fact that the book was a portable size and just 120 pages also helped my cause.

This book, called “Personality: How to Build It,” was written from the perspective of a French man, who presumably was himself, “successful”. In this book, the author is concerned with conveying what he believes to be the recipe for success in terms of creating for oneself an aura, a reputation, and a unique identity focused around cunning, sensible discretion, winning ways, and various categories of achievement relative to others. The key element of the author’s taxonomy of a winner, who he refers to as “adroit arrivers”, which I could not reconcile, was this conditionality and relativity to others.

The author, to be fair, in all of his brazen, omniscient authority, certainly conveys some advice which is helpful and relevant today – perhaps even timeless. The way he eloquently speaks of exposing something original to the world and knowing oneself well enough to identify special qualities and to develop them, is inspiring. Care and vigilance, energetic sincerity, perspicacity, determination, and perseverance are all astutely analyzed as elements necessary to form a personality. While pride, envy, jealousy, and conceit are graphically described and demonstrated as defects to be overcome.

But what is most fascinating is that valuable insights can be derived even from the elements that aren’t accurate or relevant today. By combining a contemporary perspective, a reflective mind and a little multidisciplinary academic knowledge, it’s possible to infer from the contextual setting of the author, valuable lessons to better understand the world of yesterday and today, and the people in it. For one thing, the world was much simpler back then, or at least our understanding of it was. Now globalism drives our world economy, almost everything we know about our world is available online, and greater attention gets paid to social issues such as poverty, equality, human/civil rights, and climate change. Also, Psychology (the study of the human mind and behavior) was in its infancy – and still is, really; although it has come a long way since 1915.

Despite the author’s elegant early 20th century language and convincing tact when conveying his point of view, I still disagree with a couple of concepts. Although the author masterfully sells the ideology of “arriving” and that of narcissism, using subtle descriptions, intriguing anecdotes, and profound assurance in his tone, I couldn’t come to grips with his outlook. Maybe it’s not so obvious at first, what’s so bad about “arriving” or being a “winner”. Aside from the conditionality of the author’s definition on one’s position relative to others; is there really a definitive end, a point at which we can confidently say, “I’ve arrived; I’ve won; I’ve done IT!”?

I do not agree that this belief in a definite goal – which still requires maintenance to retain one’s position over others – is superior to the approach that life is a journey with no predefined script where the only instructions for this role are those related to self-improvement. Not to mention that progress or achievement defined dependent upon one’s initial position across the spectrum of others, may not maximize one’s potential. Rather, competing with one’s self is a strategy better designed to maximize one’s potential. Aside from fulfilling our potential, happiness and meaning can come from other sources – often those we least expect – like everyday wonders that occur when we stop to smell the roses. This highlights the importance of also enjoying the journey; cherishing the present.

Perhaps another thing that our overwhelmingly complex world of today impresses upon us is the consciousness that each of us is but a blip in the grand scheme of time, generations, cultures, and societies. To live with this realization, it is essential to have humility. If we have great faith in something, it must come from some source more profound, more enduring than our selfish ego. Trusting too much in one’s mind can be tempting but dangerous.

The author seductively paints the portrait of the ideal personality as one who must necessarily think critically and strategically about how to advance one’s position via self-centered manipulation of the mind. Depending on how you look at it, this can work out well as in the case of the Medici family with the help of Niccolo Machiavelli during the Italian Renaissance period. These teachings make me think that the author must have been skilled in his ability to overpower his cognitive dissonance with confirmation bias. I further hypothesize that he lived within society in which his state of mind actually served him quite well. Perhaps the author’s stance still maintains great advantages at an individual level, but on an aggregate scale, I don’t think the general equilibrium outcome would be so wonderful if too many in the population subscribed to his outlook.

Our brains are an indispensable tool, but we shouldn’t fool ourselves by overestimating its ability to unlock the answers to some of the greatest challenges we’ll face in life. Excessive stroking of the mental ego can actually build a toxic immunity to one’s humanity. Ego-centrism puts self-interest ahead of building relationships, serving others, improving the livelihoods of those we touch and the welfare of the greater global society. Our humanitarian instincts can mysteriously tap into the unknown, and more likely build a connection to something infinitely vaster than one’s self.

The read was fascinating, helpful, humorous, curious, and even a little disconcerting (at the moments I was hoping that not everyone in the world is thinking the same way as the author with respect to “arriving” on a narcissistic shuttle). At the dismaying moments it seemed possible that the elite establishment of this previous era were the precursor to the dominance of unfettered global capitalism, and the seeds of the now robust and rising tide of neoliberal politics and academics we observe in the US and in parts of Europe? I clearly connect the author’s categorical mindset to statistical discrimination and virtuous cycles for the upper classes and vicious cycles for the poor.

Developmental psychology tells us that some of the personality traits the author addresses are not so easy to change, and traumatic life experiences during sensitive stages of life can have a much more formative impact on one’s psychology than a self-help book or coach as an adult, when one’s personality is all but cemented. Nonetheless, we should always celebrate and foster the full evolution of our character, personality and unique identity, while not failing to take lessons from the past; and let’s motivate ourselves to maximize our potential in ways that can only be defined more broadly.

The first thing to avoid is that chronic and contagious folly, fashion, which changes our habits, our thoughts, our body and our life. Accept it only in reasonable form, follow it from a distance and under the least enslaving form.

Conserve your innate originality. Don’t be dragged into tastes which are not your own. Defend yourself against any characteristic of others. Learn to judge everything for yourself without being the perfect repeater of the judgements of others.

… Live stoically on the borders of a promised land, in spite of revolts and unbeliefs which mutter within you. Be the captain of your ship of destiny launched on the ocean of life.”

— H. Laurent, “Personality: How to Build It”

 

(Image © Benjamin Anderson 2015)

Small Blessings, Big Wonders

Every day I have woken up in the magical city of Barcelona, located in the extraordinary world that we call “earth”, I’ve felt brand new. Like a baby captivated by the ordinary, I am awed by small things that in aggregate create an existence that is inspiring beyond the limits of imagination. Whether it’s the cooling, gentle sea breeze that whispers through the elegant palms on my walk home, or the people and animals all frolicking harmoniously together in the park on a clear, Mediterranean fall day; I am reminded of all the blessing bestowed upon me. These blessings, big and small, amount to something wondrous.

When I see what many small things can become, I extrapolate that concept to my purpose and vision; I imagine what small daily achievements can make my long-term accomplishments spectacular. Perhaps this is why expectations are sometimes met, but other times, they are positively shattered. It’s good to set goals, but I refuse to subscribe to the belief that a goal is a limit.

Yesterday is gone. Tomorrow has not yet come. We have only today. Let us begin.

– Mother Teresa

(Image © Benjamin Anderson 2014)