By Scott Burris
A couple weeks ago, I was in a conference room at a global health organization, all ready to give my talk on monitoring and evaluating legal health interventions. The chief of the organization’s formidable M&E operation was my host, and after briefly going through my bio he wound up his introduction by describing me as “the guy who will be telling us how to measure the un-measurable.”
Perfect.
In that one flourish, he captured the biggest barrier to more and better research on the impact of laws and legal practices on health: the cultural belief that law is different from other forms of individual and institutionalized human behavior and belief, so that it, alone, must perforce remain an evidence-free zone. This is certainly a tragedy of low expectations if ever there was one. Uncertainty is part of any hard decision, but if people in the organization I was visiting were talking about defining the optimum treatment regimen for a particular disease, they would take for granted that the deliberations of the decision-makers would be guided by a substantial evidence base. Yet when the question is what package of laws and legal practices create the best environment for preventing the same disease, or encouraging people to seek treatment, they see nothing strange about proceeding entirely on intuition and experience.
As Evan Anderson and I have recently written, the importance of law to health, and the overall success in properly evaluating its impact, belie this continued cultural prejudice. Law can be hard to evaluate, but so are most other influences on our behavior and environment. In a number of areas of legal intervention, researchers have found ways to measure the hard-to-measure and produce credible findings that have shaped policy. They have done so in ways that respect the prosaic realities of practical science work: developing reliable measures and data and deploying them within robust designs is not the work of individuals, it’s not cheap, and it is not quick. Where legal evaluation has thrived, it has done so because enough money was available for long enough to support multiple lines of inquiry by multiple teams of researchers. Careers, or stands of careers, could be built, and competition and disagreement could drive rigor and relevance.
This week, PHLR is celebrating one very tangible result of investment in the field: the publication of Public Health Law Research: Theory and Methods, which was conceived by the PHLR Methods Core and edited by Alex Wagenaar and me. The book, which was written both as a methods class text book and a general reference work, is an important piece of field-building, in that it tries to define the basic good practices of PHLR. But I think it does more: Alex, whose work on crash law exemplifies all that legal monitoring and evaluation can be, has led the production of a book that we can drop on the desk of every person in every funding and health services organization who thinks that measuring law is measuring the un-measurable.
Nice post, Scott. I think that your commentator may have mis-named the problem. I don’t think that the problem is usually measurability per se. We can measure whether a given state has adopted law A versus law B (the independent variable), and can measure whether the people in that state have better or worse health (the dependent variable), along with measuring various mediating variables such as the number of law enforcement actions and the number of car wrecks. That measurement task requires work, but is not rocket science.
What is really tricky is not the measurement, but the causal inference, primarily due to other differences in the state that may have caused it to adopt law B and also may cause it to have different health outcomes. That’s a gigantic problem for anyone who does observational empirical legal research (including many economists), one that I think is more profoundly difficult than many scholars and consumers of scholarship realize. That (along with my aversion to calculus) is why I am an experimentalist , using random assignment. Now, if only we could persuade policymakers to use random assignment in whether, or at least when, to adopt policies, then we could have gold-standard causal inferences.
Thanks, Chris, but I stand by my title. While you are right about the inference problem, in general, the problem the people I was talking to were most concerned about was measuring implementation and intermediate effects of legal interventions. My argument is that that is not a unique law problem, and is one that can be solved with reasonable resources and ingenuity.
I wonder if the law-and-economics folks have the same problem, with everyone objecting that law is un-measurable? I suspect not.