The intervention and the checklist: two paradigms for improvement

I’m working on a project involving the evaluation of social service innovations, and the other day one of my colleagues remarked that in many cases, we really know what works, the issue is getting it done. This reminded me of a fascinating article by Atul Gawande on the use of checklists for medical treatments, which in turn made me think about two different paradigms for improving a system, whether it be health, education, services, or whatever.

The first paradigm–the one we’re taught in statistics classes–is of progress via “interventions” or “treatments.” The story is that people come up with ideas (perhaps from fundamental science, as we non-biologists imagine is happening in medical research, or maybe from exploratory analysis of existing data, or maybe just from somebody’s brilliant insight), and then these get studied (possibly through randomized clinical trials, but that’s not really my point here; my real focus is on the concept of the discrete “intervention”), and then some ideas are revealed to be successful and some are not (with allowances taken for multiple testing or hierarchical structure in the studies), and the successful ideas get dispersed and used widely. There’s then a secondary phase in which interventions can get tested and modified in the wild.

The second paradigm, alluded to by my colleague above, is that of the checklist. Here the story is that everyone knows what works, but for logistical or other reasons, not all these things always get done. Improvement occurs when people are required (or encouraged or bribed or whatever) to do the 10 or 12 things that, together, are known to improve effectiveness. This “checklist” paradigm seems much different than the “intervention” approach that is standard in statistics and econometrics.

The two paradigms are not mutually exclusive. For example, the items on a checklist might have had their effectiveness individually demonstrated via earlier clinical trials–in fact, maybe that’s what got them on the checklist in the first place. Conversely, the procedure of “following a checklist” can itself be seen as an intervention and be evaluated as such.

And there are other paradigms out there, such as the self-experimentation paradigm (in which the generation and testing of new ideas go together) and the “marketplace of ideas” paradigm (in which more efficient systems are believed to evolve and survive through competitive pressures).

I just think it’s interesting that the intervention paradigm, which is so central to our thinking in statistics and econometrics (not to mention NIH funding), is not the only way to think about process improvement. A point that is obvious to nonstatisticians, perhaps.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://thomblake.com Thom Blake

    I think someone stole your paragraph breaks.

  • Anna Salamon

    Atul Gawande’s article is fascinating. Rationality seems like a promising area for checklists — we know a fair amount (though much less than would be useful) about what common human error patterns look like, etc., but it is difficult to consistently be on the look out. (We also don’t know what happens if one *is* constantly on the lookout in various ways, but it seems plausible that it would help.) Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?

  • Kyle

    This is not obvious stuff.
    The process improvement folks from Taylor forward have been pushing this thought process…and yet…it remains termendously difficult to get anyone to change process regardless the evidences of rather impressive benefits.
    The issue seems to be multifaceted:
    1. Folks lie with statistics, so a statistical improvement is disbelieved.
    2. Incentives do not line up well with results, so improvements not adopted.
    3. Personal needs don’t line up well with mechanistic behavior (people think, but are wrong on this point…if you’ve moved past Taylor to Deming or past)
    4. Inertia.
    5. some other stuff, but I’ve got other work to do now. 🙂

  • http://retiredurologist.com retired urologist

    If Atul Gawande’s miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their “checklists”. Most American medical decisions work on the “checklist” principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it’s just the “second half” of unnecessary healthcare.

  • http://retiredurologist.com retired urologist

    If Atul Gawande’s miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their “checklists”. Most American medical decisions work on the “checklist” principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it’s just the “second half” of unnecessary healthcare.

  • Tom

    Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?

    I would avoid applying checklists to anything involving sex or relationships, but they might be useful for financial decisions. Such a list ought to be relatively easy to produce based on standard rationality texts, I think. I have a copy of Predictably Irrational here in front of me, and it looks like it could be converted into a checklist without too much difficulty (eg. “Are you overvalueing this free product/service? Calculate how much it is actually worth.” or “Are you trying too hard to keep your options open?”) The problem is that in day-to-day life we seem to make the most irrational choices when overcome with emotion or responding in a knee-jerk way (Cialdini’s *click-whirr* decision making) so it would be difficult to remember to use a checklist even if you had one. Perhaps you would remember to use it if it were attached to something you were habitually checking anyway, like an iPhone application, or if it were part of a useful application like home budgeting software.

  • frelkins

    @Andrew Gelman

    the issue is getting it done

    Well, I am surprised to hear you say this. Why would the intense business of healthcare be different than any other complex process model? Immediately I would like to ask you to consider a Toyota lean process for medicine. Really, it should be called GTD for Doctors. Why is this not obvious? This is a serious question to you, not a rhetorical one.

    Reading the original article, I was struck by a number of factors. Let me quote from the article here:

    A study of forty-one thousand trauma patients—just trauma patients—found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations for teams to attend to. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for each is not possible

    Does he mean that there are 32,261 possible steps to follow for these 1,224 kinds of injuries? Or does he mean there are 32,261 different diagnoses, each with its own set of say, 10 steps? Am I wrong? I could be wrong. Tell me if I’m wrong please.

    If I understand what he is saying, of course it’s possible either way. How do you think we make cars and build modern airplanes? Does he truly believe that building a stealth bomber is any less complex?

    In fact, as a person who primarily works an an information architect, I can immediately see a nice VC-funded business in creating computerized process-model interaction diagrams a la Garrett in swimlanes (because they’re quite simple; other systems are more complex and may be harder for busy doctors to learn but I’m probably agnostic on that) that run on touchscreen handhelds over wireless.

    The nurses/doctors just touch the nicely colored boxes to walk themselves thru the proper flow. You start with the most commonly seen and gradually add the the more exotic. The enormous costs involved in medical errors makes the revenue possibilities for such a venture attractive. I can’t believe no one has started this business yet, actually. Andrew, is the incubator still housed in the Mudd Terrace? Is Columbia still doing that? ‘Cuz if so we should meet there, like, now.

    All have steps that are worth putting on a checklist and testing in routine care. The question—still unanswered—is whether medical culture will embrace the opportunity

    And the issue that leaps out here immediately is status: the nurses – of whom there obviously need to be more – must be empowered to challenge doctors, and hospital executives must be empowered to back them up, and ask for equipment changes (like the new kits with drapes). The article constantly makes status differences apparent – “community hospitals,” “urban hospitals” – as opposed to higher-status hospital like Johns Hopkins.

    Yet the author finds the nurses in the even the lowly urban hospital make the system work with just pencil and paper! Didn’t Robin once cite some study saying that regular community hospitals seem to give just as good treatment as fancy ones? Clearly for patient care to improve in this regard it seems that these artificial status barriers need to end all around?

  • John Maxwell

    Hmmm… What if you had a clinical trial that was less discrete and more continuous?

    For example, you could have every doctor participating in the trial get a different subset of the 50 items you are considering for your checklist. This would allow you to do things like write patient health as a function of checklist length and write out the checklist items in order from least effective to most effective.

    Instead of testing the safety function at a few points (e.g. giving every doctor a 12-item checklist), you could test the safety function at a wide variety of points (6, 12, 18, 24, 30, and 36-item checklists) and then write a curve to match your data.

    What are the disadvantages of this approach?

  • Nick Tarleton

    Anna: Not a checklist, but I had the idea to take one bias that seems problematic at a time, write a note of it somewhere prominent, and try to keep it in my mind and watch out just for it, in the hope that I come to (sometimes) catch it automatically. It’s too early to say how it works.