Free Docs Not Help Poor Kids

I don't find it surprising when studies of American/European health care consumption show little relationship between consumption and health outcomes. … But I confess I am shocked that studies show the same thing in the developing world:

That is Megan McArdle, on this Gates-funded randomized test of free medicine:

2,194 households containing 2,592 Ghanaian children under 5 y old were randomised into a prepayment scheme allowing free primary care including drugs, or to a control group whose families paid user fees for health care (normal practice) … The primary outcome was moderate anaemia (haemoglobin [Hb] < 8 g/dl); major secondary outcomes were health care utilisation, severe anaemia, and mortality. At baseline the randomised groups were similar. Introducing free primary health care altered the health care seeking behaviour of households; those randomised to the intervention arm used formal health care [12%] more and nonformal care [10%] less than the control group. Introducing free primary health care did not lead to any measurable difference in any health outcome. … Anaemia was chosen as the primary outcome because it is the most commonly used objective outcome of community interventions on malaria morbidity, with malaria the most common life-threatening disease of children under 5 y of age in West Africa.

I am, alas, not surprised.

Added:  PLOS Medicine couldn't publish the above study without also publishing a criticism:

Several biases have led the authors to judge its success on a very limited basis: (1) although the scheme benefits all members of participating households, the study only took into account a sub-population of beneficiaries (children); (2) in this sub-population, only health-related impacts were considered, and among all possible health benefits, only the potential gains in malaria-related outcomes were considered; and (3) among malaria-related outcomes, the analysis was restricted solely to one indicator: the prevalence of severe and moderate anaemia. … The study's authors conclude: "This lack of any effect, including on secondary outcomes such as Hb for which the study had good power, challenges the assumption that where introducing free health care leads to changes in utilisation, it can safely be assumed to translate into health benefits. Given the potential size of resources involved in providing free health care that could be diverted from other priorities on the basis of that assumption, this finding is potentially important for policymakers." But given the methodological limitations of the study, we believe that the trial provides no scientific evidence on the effectiveness of the pre-payment scheme.

Geez.  Translated: as long as any possible studies have not yet been done, there can be no evidence that med $ doesn't help.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://google.com/ Daniel Reeves

    Instead of saying, “randomized test of free medicine,” you should say “randomized test of Ghanaian free medicine.” The latter is less misleading, as their country is certainly a lot different than ours economically speaking.

  • http://google.com/ Daniel Reeves

    Oops. In reference to my last post, I concede I had forgotten the first block quote by the time I finished reading the post. And I was also referencing the low level at which Ghana’s marginal returns on health care spending hit around zero. In America, I’m sure low and equivalent levels of spending have some value attached to it.

  • frelkins

    @Robin

    Since it seems that health benefits can come simply from having authority figures pay individual attention to you, I wonder if the study would have had a better result if say, the children had been sent to traditional healers or other culturally appropriate folk working in tandem with the Gates clinic people to deliver treatment.

    Perhaps the children & families didn’t see the free health care doctors as having the proper authority, or the doctors didn’t spend enough time with them individually?

    Or do people just value the authority’s time and attention more highly because they have to pay for it, thus increasing the health benefit thru even the smallest fees?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Daniel, what makes you so sure?

    frelkins, maybe the authority attention gain is countered by real health harms.

  • http://diogenes42.blogspot.com Nz

    Robin, it might help to read the abstract — the number of events in both groups is surprisingly low. The study is underpowered and they probably overestimated the benefit on the outcome they looked at. A less biased conclusion would probably just consider this specific intervention rather than medicine as a whole.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Nz, when I hear about a bigger study, I’ll talk about it. Till then, this is what we have.

  • http://profile.typekey.com/EWBrownV/ Billy Brown

    As usual, your own anti-mdicine bias leads you to give a contrarian study an unwarranted level of credence. Check the literature and you’ll find there have been numerous studies specifically about the payoff of various health care interventions in African nations, showing benifits ranging from moderate to huge depending on the study and the nature of the intervention.

    So some skepticism about this result is quite justified – the authors are contradicting a fairly substantial body of research based on rather thin data. This is especially true since malaria treatments tend to show especially high payoffs in other studies.

  • frelkins

    @frelkins

    maybe the authority attention gain is countered by real health harms

    Then beyond building clean water wells, covered sewers, and offering basic immunizations for measles etc. (the malaria vaccine is said to be coming), perhaps the best thing the Gates Foundation can do for poor children is train more “respected traditional healers” and give them sugar pills to dispense (authority benefits + no harm), as well as build more free schools (education level appears to raise health status)?

  • jamie

    Again with the Tarzan grammar. Should that be “Free docs do not help kids”? Or maybe “Free docs not helping kids”?

  • frelkins

    @jamie

    Tarzan

    For some reason, a lot of people feel a strange need to try to “correct” Robin on these things. May I direct your attention to the perspective of a professional headline writer?

    “As with the last time we explored grammatical errors, I feel compelled to mention that copywriting and blogging should be conversational and engaging, and breaking formal grammatical and spelling conventions can often be a good thing. Every time I see a comment complaining about something like, oh, I don’t know…the improper use of an ellipsis or one-sentence paragraphs, I shake my head with sadness.”

    Deliberately breaking rules in headlines is a time-honored attention grabber, practiced by nearly all popular newspapers. Notice that OB now has in excess of 2.75 million hits. I’d say it’s working for ’em.

  • http://www.mutualinformation.org/ luispedro

    I think you are too soft on the critique. The following sentence is especially telling: “While it is clear fees must be abolished,…”. Also, notice how they critique the original study in that “only health-related impacts were considered”.

    *

    Nz, The study is not so underpowered when it comes to measuring mean Hb concentration, which is done for the whole population. They also have other indicators. The focus on a single under-powered indicator is misleading.

  • a student of economics

    Hey Robin,

    Your blog is fun to read but on this one, I think you need to either read the study more carefully or brush up on you statistics before jumping to conclusions, especially those fed to you by bloggers like Megan who have a known ideological axe to grind. Here’s what I wrote at your colleagues’ Marginal Revolution blog about this study:

    Health utilization was only 12% higher in the experimental group, yet, as per David Levine’s comment, we cannot reject the hypothesis that the primary outcome improved by as much as 34% in the experimental group. Given the very low power of the test, it seems premature to conclude that additional health care had no effect. In fact, we can’t rule out an effect more than 2x the amount of treatment.

    Meanwhile, due to selection or the Hawthorne effect or some other unanticipated experimental design issue, even the control group apparently had a staggering 70% better outcome than the authors predicted before the study.

    So what can we conclude from all this? Not much about healthcare in Ghana and even less about healthcare in the U.S.

    But we can make some inferences about the criteria Megan McArdle uses for deciding which of the many medical studies done each year are important for readers to know about, and which are echoed by bloggers with sympathetic ideologies.

    Best of luck in your laudable project to “overcome bias”.

  • Carl Shulman

    “Then beyond building clean water wells, covered sewers, and offering basic immunizations for measles etc. (the malaria vaccine is said to be coming), perhaps the best thing the Gates Foundation can do”

    Fortune,

    The Gates Foundation spends on agricultural research, vaccine development, etc. Even with tens of billions of dollars, it’s unlikely that paying for traditional healers (is there such a shortage now?), except for experimental studies, will ever be reasonable given their opportunity costs.

  • jamie

    frelkins:

    I know that we are off topic, so this will be my only response.

    Broken grammar might help attract attention. All I know is that it looks unintelligent to me, and if the headline had not been written by someone that I considered smart, I would not have bothered reading the content. Am I alone in judging the content by the headline?

    Also, grammatical rules allow you to say precisely what you mean. Without them, you must rely on context and common expectations. This limits your ability to say the unexpected.

  • billswift

    I read something a few years ago that said providing window and door screens would have more impact on health in Africa than all the other medical spending combined, but that it isn’t done because it isn’t fancy or prestigious enough for the providers of aid to actually do. (Sorry can’t remember the source, not even whether it was Web or a book.)

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Billy, care to cite a better randomized trial on “the payoff of various health care interventions in African nations, showing benefits”?

    student, I don’t understand why you think their excessively-pessimistic initial estimate of outcomes means we can’t conclude much from the study. Chicken-little overestimates of problems are pretty standard in this area.

  • Grant

    student, I think Robin’s point is that while the study isn’t terribly conclusive, people’s reaction to it shows their biases that more health care must be a good thing.

  • a student of economics

    Grant says: “Robin’s point is that while the study isn’t terribly conclusive…”

    Is that why his bold-face headline states this unambiguous, if ungrammatical, conclusion: “Free Docs Not Help Poor Kids”?

    “…it shows their biases that more health care must be a good thing”

    Uh, yes, my bias is that health care usually has an effect on health. Studies which claim otherwise are counterintuitive to most people though not to Robin (“I am, alas, not surprised”). One would expect a high powered, well designed study with clear results before making such a claim.

    On the contrary, as noted, according to conventional statistical standards, this study cannot conclude that a small amount (12%) of additional health care does not improve health by as much as 34%, contrary to Robin’s misleading headline.

    My prior belief before seeing study: health care probably improves health.
    My updated belief after seeing study and Robin’s analysis: health care probably improves health.

  • frelkins

    @Carl

    Fortune

    Indeed. Why does even Ozawa cheap out on all 8 kettledrums & 2 full-sized concert gongs? It’s the only way Orff’s lack of taste can at least acquire grandeur! As Empress of the World I deserve no less – does he really expect me to make do with only 6 kettledrums, 1 small gong, and 1 medium cymbal? C’mon everybody, weep with me.

    with tens of billions of dollars

    No way it’s gonna cost that much, Carl, which is the point. Sugar pills & grandmothers are cheap and at least do no harm. Let’s have the courage to benefit society by actually Being Hansonian. Health care appears at our current stage to be as much signaling as science. Maybe more signaling than science. Robin can weigh in on that, not I.

    Fulfilling all the requisite signals seems easy to do, and would thus improve many if not most health outcomes. So why don’t we? Are we not truth seeking here or what?

  • a student of economics

    Robin @ 1:08pm: Apparently, the expected rate of malaria-induced anemia is 10% in this population. The experimental treatment group (with subsidized care) did much, much better than this — about 3%. But that’s not the headline. The reason it was not statistically significantly different from the control group is that, for reasons unexplained and not discussed by the authors, the control group also did much much better than expected. This is a red flag that there is some large, unexplained, phenomenon affecting the groups, e.g. the Hawthorne effect, or the there is a major flaw in the experimental design. If you run an controlled experiment, and the control behaves far differently than expected and you don’t know why, then it raises questions about the reliability of the study design, execution and results.

    Furthermore, the difference in healthcare utilization was only 12% between the groups. But we can’t reject a difference of as much as 34% in outcomes. So, even taking the study as face value, it would be an abuse of statistics to say that they ruled out any effect from treatment. The study is fully consistent with the small difference in treatment providing a large (30%) improvement in the outcome, making your headline false.

    To be precise: this study FAILS to reject the hypothesis that health care has an economically significant (i.e. 30%) effect in this situation.

    If you wanted to write an accurate headline, you could say “A published study does not show that free docs help kids and also does not show that they don’t” but I guess that would hardly be worth blogging about.

  • Grant

    student, I actually hadn’t noticed the headline. Obviously you’re correct.

    Though to be fair to Robin, the issue isn’t whether “health care is good”, its whether or not health care at the margin is good; i.e. what sort of returns does more health care give the patient? Obviously I’d agree that most health care improves health (though this probably wasn’t true throughout most of history).

  • steven

    If a value is near the edge of the study’s 95% confidence interval, that still means the study provides a fair amount of evidence against that value (relative to the values in the middle of the confidence interval).

  • http://profile.typekey.com/greyorm/ Raven Daegmorgan

    I think Robin’s point is that while the study isn’t terribly conclusive, people’s reaction to it shows their biases that more health care must be a good thing.

    I think people’s reaction in reading Robin’s statement in the most positive light possible, regardless of what the statement is, shows their biases towards Robin as an in-group figure especially towards forgiving and overlooking his own biases, foibles, or tunnel-vision.

  • Michael Bishop

    I think this post deserves an additional “Added:” to point out what cannot be concluded on the basis of this study. I thank Student of Economics for the comments.

    Clarification in the comments is insufficient for such important info.