CAUSE, EFFECT, AND THE STRUCTURE OF THE SOCIAL WORLD
MEGAN T. STEVENSON
Claim: most reforms and interventions in the criminal legal space have little to no lasting effect when evaluated by RCTs (randomized controlled trials), and the occasional success usually fails to replicate when evaluated in other settings.
…people are only exposed to research that has made it through the distorting filter of research and publication incentives.15 This filter suppresses research that isn’t statistically significant, sufficiently novel, or otherwise exciting.16 Most people are only aware of the tiny set of studies that made it through the sieve. And these studies are biased toward showing that the intervention evaluated was more successful than it actually was…
In the 1980s, sociologist Peter Rossi argued that the failure of social programs was so ubiquitous that it should be known as the Iron Law of Evaluation.
What gets published is the single instance where the spurious causal effect is found. With an infinite supply of research questions, and thousands of scholars looking for interesting research, the literature will be full of false causal claims.
The first scholar to demonstrate a causal relationship between X and Y can reap large professional benefits: prestigious publications, respect from peers,increased likelihood of tenure, etc. But, in a field that rewards novelty, attempts to replicate the original result have little upside for the researcher’s career. If a subsequent study confirms the initial result, it will generally publish in a journal of lower prestige, if at all. If a subsequent study yields opposite results, it may earn the resentment of the original scholar whose work was challenged.54 Young scholars are often advised to avoid attempting to replicate the work of others.
If change is cumulative, then small interventions can grow in influence over time.99 Such low-cost, highbenefit interventions are the holy grail of social engineers.
Under the engineer’s view, social processes are structured and manipulable. RCTs and other causal inference methods are used to map the functioning of themachine, to see what impact a particular lever has. They can be used to identify interventions that yield consistent and replicable success. The uncertainty of reform is minimized because interventions can be piloted before scaling up….When it comes to the type of limited-scope interventions evaluable viaRCT and other quasi-experimental methods, the engineer’s view appears to be mostly a myth. More than fifty years of RCT evidence shows the limits in ourability to engineer change with this type of intervention.
… it teaches us that the social world is full of what I call “stabilizers” and short on what I refer to as “cascades.” …Stabilizers are the set of socioeconomic forces that resist externally-imposed change….” Cascades are forces that magnify small changes, that turn a small intervention into a large and lasting effect…
Types of interventions considered:
- financial (UBI)
- Big Brother type mentoring, My Brother’s Keeper (MBK, an Obama era initiative)
- education, job training
- variations in jail time
- Swift, Certain and fair (rapid minor penalty)
- Multisystemic therapy
- Project HOPE