Related
Working Paper
/ May 21,2015
The Political Economy of Investment in Pre-K
Oct 10,2012
Measuring Outcomes from Educational Training
+ Kari Kohn
When it comes to learning—whether in a classroom or on-the-job—measurement is critical for tracking progress. And though measuring inputs to learning (such as hours in the classroom) is often easier than measuring outcomes (such as performance on tasks or problem-solving), it is the outcomes that will tell us more about the efficacy of education. From the summary of a podcast with Lant Pritchett on economic development, the Center for Global Development's Lawrence MacDonald writes:
Lant suggests that it’s important to develop indicators that showcase real progress (and are resistant to the mere appearance of progress). In the education sector for example, instead of measuring enrollment rates, he suggests more effort to measure learning (see Charles Kenny).
In the New Yorker, Lisa Rosenbaum delivers a similar message in the context of medical education. Recent changes in the rules governing shift limits for medical residents may be impacting the quality of patient care and residency training.
As we seek to understand what this new training system means for the future of health care, the ease of monitoring certain outcomes sometimes keeps us from understanding equally important but little-studied factors. Rivka Galchen describes this challenge in her recent essay about the legendary Elmhurst Hospital physician and clinical educator Dr. Joseph Lieber: “The field of medicine has advanced through measuring: weight, blood pressure, dosage, cost, days until discharge, years until death… But sometimes the medical field makes the mistake of valuing most what is most easily measured.”
Our approach to duty-hour limits for residents has been no exception. Everyone knows how it feels to be tired, and there is nothing easier to count than hours worked or slept. I have lost count of the number of conversations I’ve had with non-doctor friends that have begun, “You do realize sleep deprivation is akin to being drunk?” This sentiment is echoed in a recently published survey of U.S. citizens that found that eighty per cent of people would prefer a different doctor if they knew theirs had been working more than twenty-four hours. This conviction—that a rested doctor who doesn’t know you would be better than a tired doctor who does—fueled the 2003 and 2011 reforms.
This is not to suggest that we shouldn’t measure everything we can. But the most important things to quantify are hard to measure: outcomes relating to quality and education. And although it will be a while before we can really understand the effect of the 2011 reforms, two recently published studies suggest that, right now, both quality of care and quality of education are suffering.
So, in this case, perceptions about the impact of an easily measured input (hours per shift) drove reforms that may turn out to have an adverse impact on more important if more difficult to measure outcomes: the quality of patient care and the quality of residence training. Thinking about it from a patient: would you rather know the shift limit at your hospital or would you rather know how it ranks in terms of quality of care?