Dustin Houchin: Sorry, but it’s a law: Social intervention programs don’t work

In a prior column I reviewed the flaws with criminal justice program evaluation, including unsound testing methods, conflicts of interest, selection bias and publication bias. I then posed the question, “Then how do we know what works?” The answer is you evaluate the programs using the highest standards in the research field. It turns out that when you rely on the highest standards — and why wouldn’t you? — no programs are shown to work.

In general with social science, as opposed to criminology, it is well known that almost no programs work. This is so well-established that they have a name for it. They call it the Iron Law of Evaluation: The expected value of any net impact assessment of any large-scale social program is zero.

To put it more simply, social programs fail to work so often that when researchers set out to study one, they should expect to find that the program makes zero impact. Unfortunately, the Iron Law of Evaluation is not understood among people who work in the criminal justice system.

A new article in the Boston University Law Review by Megan T. Stevenson, professor of law and economics at University of Virginia School of Law, tests the Iron Law of Evaluation on criminal justice programs. Her findings support the Iron Law’s application in this area. She found that no programs to reform offenders in the criminal justice system work.

To come to this conclusion, Stevenson performed what is called a meta-analysis. This means that she didn’t conduct new research herself. Rather, she analyzed studies of criminal justice programs that have been performed over the last 50 years. Not just any studies. Stevenson only reviewed studies that relied on randomized controlled trials because randomized controlled trials are the gold standard in the research field. They yield the most reliable results.

Here is a list of programs that don’t work, as evidenced by the highest standard of research available:

• Counseling and therapy programs
• Criminal legal supervision, including intensive probation
• Scared straight programs
• Job training programs
• Drug testing programs
• Substance abuse counseling programs
• Drug courts
• Juvenile diversion programs
• Policing “hot spots”
• Boot camps

So, pretty much everything your court system does has been shown by the highest standards of research available in the field to simply not work. But you probably think it works. Why?

Well, we want something to work. Crime and substance abuse are huge problems in our society. It would be nice if we could fix them. So, we have hope, and we rely on anecdotal success stories to convince ourselves that our favorite program works.

We are also told that things work. We go to conferences where exciting or passionate or entertaining or powerful people tell us stuff works. But the stuff doesn’t work. These speakers are relying on their own hope or anecdotal evidence or flawed research or are trying to manipulate us in their own self-interest.

But we are judges, lawyers and policymakers. We should not be swayed by feel-good stories or flawed evidence. When we are making decisions that drastically intervene in people’s lives and when the safety of our communities is at stake, we should be relying only on the best available evidence. The best available evidence shows that the expected value of large-scale social intervention programs is zero. Nothing works.•

__________

Houchin is the Washington County Superior Court judge in Salem. He also is the publisher of Judex, a Substack newsletter on conservative judicial issues at judex.substack.com. Opinions expressed are those of the author.