How Data is Used Can Mean the Difference Between Improving or Undermining Educational Outcomes

In the past few weeks, there have been a barrage of media reports about educational achievement and, more generally, life outcomes for the youth of Durham.

The positive news is that these issues are receiving attention, but the downside is that the reports may be more harmful than helpful. At its best, data optimizes decision-making, but at its worst data can be deceptive and divisive.

Specialized knowledge is required to leverage data for decision-making, whereas selectively reporting figures requires some effort but no expertise. In the latter scenario, the ambiguity of statistical assumptions predisposes the audience to personal, as well as, framing bias. Those who go through the effort to produce data often have an agenda, and therefore, have incentives to make claims which imply causes and solutions. Data is dangerous when misused. It can create tension, undermine trust and unity, and result in costly adverse decision-making.

One key characteristic of amateur statistics, aside from lacking an experimental design, is that they do not account for the fact that outcomes are a function of many different variables. For example, schools clearly play a crucial role in influencing academic attainment, but a report drawing relative comparisons between attainment outcomes within or across cities usually implicates unidentified failures of one school district versus another while all but ignoring the effects of transportation, affordable housing, food, healthcare, and social support accessibility, as well as people’s different lived experiences, including traumatic exposure of various kinds.

Reactivity to outcomes is strongly linked to bias and emotion. Making decisions about problems and solutions based exclusively on outcomes is the logical equivalent to going with your gut. Descriptive statistics alone have a tendency to reinforce what we already think we know rather than helping us to gain an objective understanding of the issues because we often overestimate our understanding of the context. Shards of truth may be buried in our presumptions or between the different storylines, but other times the truth isn’t within sight.

If one wanted to know what public schools are doing right and what positive changes could be made, the reported outcomes would not meaningfully increase understanding. This would be like a college basketball coach using the Ratings Percentage Index (RPI) to make game plans. The RPI is simply a function of outcome variables that are influenced by other, more influential variables over a team’s success, such as shot selection, rebounding, ball control and many others.

Similarly, objective inference about the determinants of academic achievement is impossible when we simply have some measure of the output, like grade level proficiency, graduation rates or achievement gaps. Summarized outcomes do not even begin to untangle the multifaceted causal factors of student achievement, or even point to which factors are within the schools’ control and which are shaped by other institutions that govern infrastructure, real estate development, credit markets and criminal justice.

Good intentions often lead to unintended consequences. Calculating outcomes or deriving slightly new definitions of them does not enhance the cultural or intellectual competence of our community, its citizens or the institutions within it.

This is troubling because the extent of harm done with every report that subjectively frames and selectively reports data will never be known. A symptomatic obsession can enable data to have a negative social impact, leading to the proliferation of economic and racial segregation, adverse selection of people and funds from public schools, victim blaming and the marginalization of objectivity. The focus needs to shift from symptoms to solutions.

Data should be collected and analyzed in a way that enables us to separately identify effects on outcomes, including those determinants within the school’s control and those outside so that all can be addressed in order of impact and feasibility. Robust evaluations should yield insight, pointing out specific causal factors that affect outcomes that the schools, nonprofits policy and citizens can address.

Applying a scientific lens to social issues transforms data from punitive to instructive. Careful investigation using valid quantitative methods can help us gain an understanding of the inferences that the data will and will not permit. Through empirical analysis, we have the opportunity to disentangle the effects that different factors have on certain outcomes. This is powerful because it enables us to create informed strategies.

Subsequently, when we know how our potential actions will affect an outcome, a cost-benefit analysis can help decide which evidence should be brought to action. Operating in the public and nonprofit sectors, the cost-benefit analysis goes beyond fiscal considerations to examine social returns. Combining these empirical tools puts us in a position to optimize social welfare. Data or analysis vacant of these characteristics will result in suboptimal decision-making.

An empirical basis for decision-making that respects the complexity of determinants on outcomes and the tradeoffs between various actions or lack of action should be utilized at all levels – from the systemic to the programmatic. A symptomatic focus and a preoccupation with a single area will not result in systemic improvement. As institutions, organizations and programs, our goal should be to improve, which can only be achieved through learning.

Durham has great potential to grow while enhancing the well-being of all, including the most marginalized. Continuous improvement requires the commitment of people in the public, private, and social sectors to work together.

Part of analytical integrity is the acknowledgment that sometimes our data tells us nothing at all. If we truly care about addressing systemic issues, lack of information is a strong argument for why we should build more robust datasets that incorporate variables across institutions and the socio-economic environment. This requires a willingness to coordinate and to learn. Importantly, these actions imply the willingness to change.

The Made in Durham partnership exists to address issues of the highest importance. It is the job of data is to increase the role of evidence in the partnership’s decision-making, and because of the gravity of these decisions, I also feel an ethical accountability to this work.

If we aren’t asking the right questions, data can lead to costly decisions that undermine improvement. As members of the community, we should all be able to ask the right questions to hold decision-makers accountable to analytical standards that drive improvement.

Regardless of what the outcomes show now, or anytime in the future, what we should be asking is: what are the causes of these outcomes, what are their magnitudes, and thus, what can we do to improve.

see the original post on the website:

If you do not know how to ask the right question, you discover nothing.

In God we trust, all others must bring data.

– W. Edwards Deming

(Image © Benjamin Anderson 2016)


Economic Inequalities – Where to Start?

Entering my brush-up courses for graduate school in one week, I ponder the most pressing challenges for the future of a policy economist focused on development and more equitable economic opportunity for all. The (unsatisfactory) growth of human capital with relation to aggregate incomes and the distribution of personal incomes within and across societies plays an increasingly central role in political debates and the demands of citizens worldwide, as information improves global awareness of forces marginalizing the conditions of those with the lowest incomes.

I wrestle with what seems like an overlooked starting point to this issue. As a person born, raised, and educated in the developed world, isolated from extreme poverty, what is more critical – to focus on alleviation/elimination of extreme poverty that affects the developing world, or relative poverty that persists in the developed world alongside massive wealth? Will decimating relative poverty exacerbate the poverty plaguing the undeveloped world by contributing to a concentration of capital that creates a dichotomous world, similar to the growing inequality trends occurring within many of the world’s richest nations (in particular the U.S. case); or will a disproportionately large increase in the wealth and well-being of the relative poor create an economic climate of socially responsible globalism and development aid that fulfills the economic proverb preached by some since the dawn of the modern conservative economic era in the U.S., that “a rising tide lifts all boats”? Both ends have good intentions, but which produces the greatest net gain (measured in all indicators of quality of living)? In order to answer this question, both require carefully researched, coordinated, and executed policies to achieve the optimal effect. Both must holistically assess and measure costs and benefits, seen and unseen, such as:

  • Number of lives positively affected
  • The net effect on said lives
  • The magnitude and direction of effects on differential aspects of life
  • The marginal returns to improvement in factors contributing to quality of life stratified by the individuals’ and populations’ utility preferences
  • Risks of failures and implementation
  • Who is hurt in the short-run
  • What is the “breakeven point”, or the point at which people have opportunity for growth, a “foot on the ladder” of development, and development is sustainable with no further need of planned aid because people have the tools they need to grow
  • Might rich countries and multinational corporate powers perceive potential threats coming from new competition, and might this pose a political barrier to action
  • Might controversy and class “warfare” regarding relative power and influence somehow emerge as a result of more equitable societies; or other negative unforeseen consequences

Secondarily, but necessarily, the costs and benefits to the financiers/rich for investing what’s necessary to fight poverty must be accounted for. Theory would support a long-run gain for those investing the capital – opening and expanding new markets and bolstering demand – via empowering growth.

These are complex issues, not to be oversimplified, and necessitating analysis by multiple disciplinary perspectives working together; not to mention political cooperation and united, competent implementation. Truthfully, I think some of these questions are beyond the capabilities of human understanding, and are much deeper and more significant. What is certain is that love is at the roots of the solution. What is also certain is we must not simply pontificate, but we must labor to understand and analyze in the most efficient way we know how. We must utilize and build upon the academic arsenal available to take on these challenges.

Laziness, selfishness, and apathy are the enemy of the fight against poverty and severe inequalities in capital. Unfortunately, these traits are part of human nature and a significant component of the inertia of the rich world today. How can one begin to tackle problems so complex and with innumerable interrelated drivers? Most importantly, we must act! Bring on the books, and bring it on, world!

Social distinctions can be based only on common utility.

– Declaration of the Rights of Man and the Citizen, article 1, 1789

(written 8/24/14; image © Benjamin Anderson 2014)