I’m working on a project involving the evaluation of social service innovations, and the other day one of my colleagues remarked that in many cases, we really know what works, the issue is getting it done.
Anna: Not a checklist, but I had the idea to take one bias that seems problematic at a time, write a note of it somewhere prominent, and try to keep it in my mind and watch out just for it, in the hope that I come to (sometimes) catch it automatically. It's too early to say how it works.
Hmmm... What if you had a clinical trial that was less discrete and more continuous?
For example, you could have every doctor participating in the trial get a different subset of the 50 items you are considering for your checklist. This would allow you to do things like write patient health as a function of checklist length and write out the checklist items in order from least effective to most effective.
Instead of testing the safety function at a few points (e.g. giving every doctor a 12-item checklist), you could test the safety function at a wide variety of points (6, 12, 18, 24, 30, and 36-item checklists) and then write a curve to match your data.
Well, I am surprised to hear you say this. Why would the intense business of healthcare be different than any other complex process model? Immediately I would like to ask you to consider a Toyota lean process for medicine. Really, it should be called GTD for Doctors. Why is this not obvious? This is a serious question to you, not a rhetorical one.
Reading the original article, I was struck by a number of factors. Let me quote from the article here:
"A study of forty-one thousand trauma patients—just trauma patients—found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations for teams to attend to. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for each is not possible"
Does he mean that there are 32,261 possible steps to follow for these 1,224 kinds of injuries? Or does he mean there are 32,261 different diagnoses, each with its own set of say, 10 steps? Am I wrong? I could be wrong. Tell me if I'm wrong please.
If I understand what he is saying, of course it's possible either way. How do you think we make cars and build modern airplanes? Does he truly believe that building a stealth bomber is any less complex?
In fact, as a person who primarily works an an information architect, I can immediately see a nice VC-funded business in creating computerized process-model interaction diagrams a la Garrett in swimlanes (because they're quite simple; other systems are more complex and may be harder for busy doctors to learn but I'm probably agnostic on that) that run on touchscreen handhelds over wireless.
The nurses/doctors just touch the nicely colored boxes to walk themselves thru the proper flow. You start with the most commonly seen and gradually add the the more exotic. The enormous costs involved in medical errors makes the revenue possibilities for such a venture attractive. I can't believe no one has started this business yet, actually. Andrew, is the incubator still housed in the Mudd Terrace? Is Columbia still doing that? 'Cuz if so we should meet there, like, now.
"All have steps that are worth putting on a checklist and testing in routine care. The question—still unanswered—is whether medical culture will embrace the opportunity"
And the issue that leaps out here immediately is status: the nurses - of whom there obviously need to be more - must be empowered to challenge doctors, and hospital executives must be empowered to back them up, and ask for equipment changes (like the new kits with drapes). The article constantly makes status differences apparent - "community hospitals," "urban hospitals" - as opposed to higher-status hospital like Johns Hopkins.
Yet the author finds the nurses in the even the lowly urban hospital make the system work with just pencil and paper! Didn't Robin once cite some study saying that regular community hospitals seem to give just as good treatment as fancy ones? Clearly for patient care to improve in this regard it seems that these artificial status barriers need to end all around?
Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?
I would avoid applying checklists to anything involving sex or relationships, but they might be useful for financial decisions. Such a list ought to be relatively easy to produce based on standard rationality texts, I think. I have a copy of Predictably Irrational here in front of me, and it looks like it could be converted into a checklist without too much difficulty (eg. "Are you overvalueing this free product/service? Calculate how much it is actually worth." or "Are you trying too hard to keep your options open?") The problem is that in day-to-day life we seem to make the most irrational choices when overcome with emotion or responding in a knee-jerk way (Cialdini's *click-whirr* decision making) so it would be difficult to remember to use a checklist even if you had one. Perhaps you would remember to use it if it were attached to something you were habitually checking anyway, like an iPhone application, or if it were part of a useful application like home budgeting software.
If Atul Gawande's miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their "checklists". Most American medical decisions work on the "checklist" principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it's just the "second half" of unnecessary healthcare.
If Atul Gawande's miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their "checklists". Most American medical decisions work on the "checklist" principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it's just the "second half" of unnecessary healthcare.
This is not obvious stuff.The process improvement folks from Taylor forward have been pushing this thought process...and yet...it remains termendously difficult to get anyone to change process regardless the evidences of rather impressive benefits.The issue seems to be multifaceted:1. Folks lie with statistics, so a statistical improvement is disbelieved.2. Incentives do not line up well with results, so improvements not adopted.3. Personal needs don't line up well with mechanistic behavior (people think, but are wrong on this point...if you've moved past Taylor to Deming or past)4. Inertia.5. some other stuff, but I've got other work to do now. :)
Atul Gawande's article is fascinating. Rationality seems like a promising area for checklists -- we know a fair amount (though much less than would be useful) about what common human error patterns look like, etc., but it is difficult to consistently be on the look out. (We also don't know what happens if one *is* constantly on the lookout in various ways, but it seems plausible that it would help.) Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?
Anna: Not a checklist, but I had the idea to take one bias that seems problematic at a time, write a note of it somewhere prominent, and try to keep it in my mind and watch out just for it, in the hope that I come to (sometimes) catch it automatically. It's too early to say how it works.
Hmmm... What if you had a clinical trial that was less discrete and more continuous?
For example, you could have every doctor participating in the trial get a different subset of the 50 items you are considering for your checklist. This would allow you to do things like write patient health as a function of checklist length and write out the checklist items in order from least effective to most effective.
Instead of testing the safety function at a few points (e.g. giving every doctor a 12-item checklist), you could test the safety function at a wide variety of points (6, 12, 18, 24, 30, and 36-item checklists) and then write a curve to match your data.
What are the disadvantages of this approach?
@Andrew Gelman
"the issue is getting it done"
Well, I am surprised to hear you say this. Why would the intense business of healthcare be different than any other complex process model? Immediately I would like to ask you to consider a Toyota lean process for medicine. Really, it should be called GTD for Doctors. Why is this not obvious? This is a serious question to you, not a rhetorical one.
Reading the original article, I was struck by a number of factors. Let me quote from the article here:
"A study of forty-one thousand trauma patients—just trauma patients—found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations for teams to attend to. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for each is not possible"
Does he mean that there are 32,261 possible steps to follow for these 1,224 kinds of injuries? Or does he mean there are 32,261 different diagnoses, each with its own set of say, 10 steps? Am I wrong? I could be wrong. Tell me if I'm wrong please.
If I understand what he is saying, of course it's possible either way. How do you think we make cars and build modern airplanes? Does he truly believe that building a stealth bomber is any less complex?
In fact, as a person who primarily works an an information architect, I can immediately see a nice VC-funded business in creating computerized process-model interaction diagrams a la Garrett in swimlanes (because they're quite simple; other systems are more complex and may be harder for busy doctors to learn but I'm probably agnostic on that) that run on touchscreen handhelds over wireless.
The nurses/doctors just touch the nicely colored boxes to walk themselves thru the proper flow. You start with the most commonly seen and gradually add the the more exotic. The enormous costs involved in medical errors makes the revenue possibilities for such a venture attractive. I can't believe no one has started this business yet, actually. Andrew, is the incubator still housed in the Mudd Terrace? Is Columbia still doing that? 'Cuz if so we should meet there, like, now.
"All have steps that are worth putting on a checklist and testing in routine care. The question—still unanswered—is whether medical culture will embrace the opportunity"
And the issue that leaps out here immediately is status: the nurses - of whom there obviously need to be more - must be empowered to challenge doctors, and hospital executives must be empowered to back them up, and ask for equipment changes (like the new kits with drapes). The article constantly makes status differences apparent - "community hospitals," "urban hospitals" - as opposed to higher-status hospital like Johns Hopkins.
Yet the author finds the nurses in the even the lowly urban hospital make the system work with just pencil and paper! Didn't Robin once cite some study saying that regular community hospitals seem to give just as good treatment as fancy ones? Clearly for patient care to improve in this regard it seems that these artificial status barriers need to end all around?
Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?
I would avoid applying checklists to anything involving sex or relationships, but they might be useful for financial decisions. Such a list ought to be relatively easy to produce based on standard rationality texts, I think. I have a copy of Predictably Irrational here in front of me, and it looks like it could be converted into a checklist without too much difficulty (eg. "Are you overvalueing this free product/service? Calculate how much it is actually worth." or "Are you trying too hard to keep your options open?") The problem is that in day-to-day life we seem to make the most irrational choices when overcome with emotion or responding in a knee-jerk way (Cialdini's *click-whirr* decision making) so it would be difficult to remember to use a checklist even if you had one. Perhaps you would remember to use it if it were attached to something you were habitually checking anyway, like an iPhone application, or if it were part of a useful application like home budgeting software.
If Atul Gawande's miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their "checklists". Most American medical decisions work on the "checklist" principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it's just the "second half" of unnecessary healthcare.
If Atul Gawande's miracle example child had died instead or become a permanent vegetable, surely Robin Hanson would have used her story as an example of the enormous waste of healthcare dollars versus health benefits achieved. Such care as described cost the Austrian taxpayers well over a million US dollars (they have a socialized healthcare system), and in almost every other instance, their efforts would have been either futile or actually harmful (if you consider rendering a state of persistent vegetation harmful), in spite of their "checklists". Most American medical decisions work on the "checklist" principle, and many of the items on the list are designed to avoid malpractice lawsuits brought under the US contingency fee arrangement (not allowed in Austria, nor in most of the rest of the world). Checklists, in this light, seem to be best evaluated after the fact, at least medically: when the result is good, who can argue the value of a life or good health?; when the result is bad, it's just the "second half" of unnecessary healthcare.
This is not obvious stuff.The process improvement folks from Taylor forward have been pushing this thought process...and yet...it remains termendously difficult to get anyone to change process regardless the evidences of rather impressive benefits.The issue seems to be multifaceted:1. Folks lie with statistics, so a statistical improvement is disbelieved.2. Incentives do not line up well with results, so improvements not adopted.3. Personal needs don't line up well with mechanistic behavior (people think, but are wrong on this point...if you've moved past Taylor to Deming or past)4. Inertia.5. some other stuff, but I've got other work to do now. :)
Atul Gawande's article is fascinating. Rationality seems like a promising area for checklists -- we know a fair amount (though much less than would be useful) about what common human error patterns look like, etc., but it is difficult to consistently be on the look out. (We also don't know what happens if one *is* constantly on the lookout in various ways, but it seems plausible that it would help.) Anyhow: are any of you fellow OB readers using checklists, or a similar gimmick, to improve your practice of rationality? Any results? Any ideas for how such a checklist should go?
I think someone stole your paragraph breaks.