I don't find it surprising when studies of American/European health care consumption show little relationship between consumption and health outcomes. … But I confess I am shocked that studies show the same thing
I think this post deserves an additional "Added:" to point out what cannot be concluded on the basis of this study. I thank Student of Economics for the comments.
Clarification in the comments is insufficient for such important info.
I think Robin's point is that while the study isn't terribly conclusive, people's reaction to it shows their biases that more health care must be a good thing.
I think people's reaction in reading Robin's statement in the most positive light possible, regardless of what the statement is, shows their biases towards Robin as an in-group figure especially towards forgiving and overlooking his own biases, foibles, or tunnel-vision.
If a value is near the edge of the study's 95% confidence interval, that still means the study provides a fair amount of evidence against that value (relative to the values in the middle of the confidence interval).
student, I actually hadn't noticed the headline. Obviously you're correct.
Though to be fair to Robin, the issue isn't whether "health care is good", its whether or not health care at the margin is good; i.e. what sort of returns does more health care give the patient? Obviously I'd agree that most health care improves health (though this probably wasn't true throughout most of history).
Robin @ 1:08pm: Apparently, the expected rate of malaria-induced anemia is 10% in this population. The experimental treatment group (with subsidized care) did much, much better than this -- about 3%. But that's not the headline. The reason it was not statistically significantly different from the control group is that, for reasons unexplained and not discussed by the authors, the control group also did much much better than expected. This is a red flag that there is some large, unexplained, phenomenon affecting the groups, e.g. the Hawthorne effect, or the there is a major flaw in the experimental design. If you run an controlled experiment, and the control behaves far differently than expected and you don't know why, then it raises questions about the reliability of the study design, execution and results.
Furthermore, the difference in healthcare utilization was only 12% between the groups. But we can't reject a difference of as much as 34% in outcomes. So, even taking the study as face value, it would be an abuse of statistics to say that they ruled out any effect from treatment. The study is fully consistent with the small difference in treatment providing a large (30%) improvement in the outcome, making your headline false.
To be precise: this study FAILS to reject the hypothesis that health care has an economically significant (i.e. 30%) effect in this situation.
If you wanted to write an accurate headline, you could say "A published study does not show that free docs help kids and also does not show that they don't" but I guess that would hardly be worth blogging about.
Indeed. Why does even Ozawa cheap out on all 8 kettledrums & 2 full-sized concert gongs? It's the only way Orff's lack of taste can at least acquire grandeur! As Empress of the World I deserve no less - does he really expect me to make do with only 6 kettledrums, 1 small gong, and 1 medium cymbal? C'mon everybody, weep with me.
"with tens of billions of dollars"
No way it's gonna cost that much, Carl, which is the point. Sugar pills & grandmothers are cheap and at least do no harm. Let's have the courage to benefit society by actually Being Hansonian. Health care appears at our current stage to be as much signaling as science. Maybe more signaling than science. Robin can weigh in on that, not I.
Fulfilling all the requisite signals seems easy to do, and would thus improve many if not most health outcomes. So why don't we? Are we not truth seeking here or what?
Grant says: "Robin's point is that while the study isn't terribly conclusive..."
Is that why his bold-face headline states this unambiguous, if ungrammatical, conclusion: "Free Docs Not Help Poor Kids"?
"...it shows their biases that more health care must be a good thing"
Uh, yes, my bias is that health care usually has an effect on health. Studies which claim otherwise are counterintuitive to most people though not to Robin ("I am, alas, not surprised"). One would expect a high powered, well designed study with clear results before making such a claim.
On the contrary, as noted, according to conventional statistical standards, this study cannot conclude that a small amount (12%) of additional health care does not improve health by as much as 34%, contrary to Robin's misleading headline.
My prior belief before seeing study: health care probably improves health.My updated belief after seeing study and Robin's analysis: health care probably improves health.
student, I think Robin's point is that while the study isn't terribly conclusive, people's reaction to it shows their biases that more health care must be a good thing.
Billy, care to cite a better randomized trial on "the payoff of various health care interventions in African nations, showing benefits"?
student, I don't understand why you think their excessively-pessimistic initial estimate of outcomes means we can't conclude much from the study. Chicken-little overestimates of problems are pretty standard in this area.
I read something a few years ago that said providing window and door screens would have more impact on health in Africa than all the other medical spending combined, but that it isn't done because it isn't fancy or prestigious enough for the providers of aid to actually do. (Sorry can't remember the source, not even whether it was Web or a book.)
I know that we are off topic, so this will be my only response.
Broken grammar might help attract attention. All I know is that it looks unintelligent to me, and if the headline had not been written by someone that I considered smart, I would not have bothered reading the content. Am I alone in judging the content by the headline?
Also, grammatical rules allow you to say precisely what you mean. Without them, you must rely on context and common expectations. This limits your ability to say the unexpected.
"Then beyond building clean water wells, covered sewers, and offering basic immunizations for measles etc. (the malaria vaccine is said to be coming), perhaps the best thing the Gates Foundation can do"
Fortune,
The Gates Foundation spends on agricultural research, vaccine development, etc. Even with tens of billions of dollars, it's unlikely that paying for traditional healers (is there such a shortage now?), except for experimental studies, will ever be reasonable given their opportunity costs.
Your blog is fun to read but on this one, I think you need to either read the study more carefully or brush up on you statistics before jumping to conclusions, especially those fed to you by bloggers like Megan who have a known ideological axe to grind. Here's what I wrote at your colleagues' Marginal Revolution blog about this study:
Health utilization was only 12% higher in the experimental group, yet, as per David Levine's comment, we cannot reject the hypothesis that the primary outcome improved by as much as 34% in the experimental group. Given the very low power of the test, it seems premature to conclude that additional health care had no effect. In fact, we can't rule out an effect more than 2x the amount of treatment.
Meanwhile, due to selection or the Hawthorne effect or some other unanticipated experimental design issue, even the control group apparently had a staggering 70% better outcome than the authors predicted before the study.
So what can we conclude from all this? Not much about healthcare in Ghana and even less about healthcare in the U.S.
But we can make some inferences about the criteria Megan McArdle uses for deciding which of the many medical studies done each year are important for readers to know about, and which are echoed by bloggers with sympathetic ideologies.
Best of luck in your laudable project to "overcome bias".
I think you are too soft on the critique. The following sentence is especially telling: "While it is clear fees must be abolished,...". Also, notice how they critique the original study in that "only health-related impacts were considered".
*
Nz, The study is not so underpowered when it comes to measuring mean Hb concentration, which is done for the whole population. They also have other indicators. The focus on a single under-powered indicator is misleading.
For some reason, a lot of people feel a strange need to try to "correct" Robin on these things. May I direct your attention to the perspective of a professional headline writer?
"As with the last time we explored grammatical errors, I feel compelled to mention that copywriting and blogging should be conversational and engaging, and breaking formal grammatical and spelling conventions can often be a good thing. Every time I see a comment complaining about something like, oh, I don’t know…the improper use of an ellipsis or one-sentence paragraphs, I shake my head with sadness."
Deliberately breaking rules in headlines is a time-honored attention grabber, practiced by nearly all popular newspapers. Notice that OB now has in excess of 2.75 million hits. I'd say it's working for 'em.
I think this post deserves an additional "Added:" to point out what cannot be concluded on the basis of this study. I thank Student of Economics for the comments.
Clarification in the comments is insufficient for such important info.
I think Robin's point is that while the study isn't terribly conclusive, people's reaction to it shows their biases that more health care must be a good thing.
I think people's reaction in reading Robin's statement in the most positive light possible, regardless of what the statement is, shows their biases towards Robin as an in-group figure especially towards forgiving and overlooking his own biases, foibles, or tunnel-vision.
If a value is near the edge of the study's 95% confidence interval, that still means the study provides a fair amount of evidence against that value (relative to the values in the middle of the confidence interval).
student, I actually hadn't noticed the headline. Obviously you're correct.
Though to be fair to Robin, the issue isn't whether "health care is good", its whether or not health care at the margin is good; i.e. what sort of returns does more health care give the patient? Obviously I'd agree that most health care improves health (though this probably wasn't true throughout most of history).
Robin @ 1:08pm: Apparently, the expected rate of malaria-induced anemia is 10% in this population. The experimental treatment group (with subsidized care) did much, much better than this -- about 3%. But that's not the headline. The reason it was not statistically significantly different from the control group is that, for reasons unexplained and not discussed by the authors, the control group also did much much better than expected. This is a red flag that there is some large, unexplained, phenomenon affecting the groups, e.g. the Hawthorne effect, or the there is a major flaw in the experimental design. If you run an controlled experiment, and the control behaves far differently than expected and you don't know why, then it raises questions about the reliability of the study design, execution and results.
Furthermore, the difference in healthcare utilization was only 12% between the groups. But we can't reject a difference of as much as 34% in outcomes. So, even taking the study as face value, it would be an abuse of statistics to say that they ruled out any effect from treatment. The study is fully consistent with the small difference in treatment providing a large (30%) improvement in the outcome, making your headline false.
To be precise: this study FAILS to reject the hypothesis that health care has an economically significant (i.e. 30%) effect in this situation.
If you wanted to write an accurate headline, you could say "A published study does not show that free docs help kids and also does not show that they don't" but I guess that would hardly be worth blogging about.
@Carl
"Fortune"
Indeed. Why does even Ozawa cheap out on all 8 kettledrums & 2 full-sized concert gongs? It's the only way Orff's lack of taste can at least acquire grandeur! As Empress of the World I deserve no less - does he really expect me to make do with only 6 kettledrums, 1 small gong, and 1 medium cymbal? C'mon everybody, weep with me.
"with tens of billions of dollars"
No way it's gonna cost that much, Carl, which is the point. Sugar pills & grandmothers are cheap and at least do no harm. Let's have the courage to benefit society by actually Being Hansonian. Health care appears at our current stage to be as much signaling as science. Maybe more signaling than science. Robin can weigh in on that, not I.
Fulfilling all the requisite signals seems easy to do, and would thus improve many if not most health outcomes. So why don't we? Are we not truth seeking here or what?
Grant says: "Robin's point is that while the study isn't terribly conclusive..."
Is that why his bold-face headline states this unambiguous, if ungrammatical, conclusion: "Free Docs Not Help Poor Kids"?
"...it shows their biases that more health care must be a good thing"
Uh, yes, my bias is that health care usually has an effect on health. Studies which claim otherwise are counterintuitive to most people though not to Robin ("I am, alas, not surprised"). One would expect a high powered, well designed study with clear results before making such a claim.
On the contrary, as noted, according to conventional statistical standards, this study cannot conclude that a small amount (12%) of additional health care does not improve health by as much as 34%, contrary to Robin's misleading headline.
My prior belief before seeing study: health care probably improves health.My updated belief after seeing study and Robin's analysis: health care probably improves health.
student, I think Robin's point is that while the study isn't terribly conclusive, people's reaction to it shows their biases that more health care must be a good thing.
Billy, care to cite a better randomized trial on "the payoff of various health care interventions in African nations, showing benefits"?
student, I don't understand why you think their excessively-pessimistic initial estimate of outcomes means we can't conclude much from the study. Chicken-little overestimates of problems are pretty standard in this area.
I read something a few years ago that said providing window and door screens would have more impact on health in Africa than all the other medical spending combined, but that it isn't done because it isn't fancy or prestigious enough for the providers of aid to actually do. (Sorry can't remember the source, not even whether it was Web or a book.)
frelkins:
I know that we are off topic, so this will be my only response.
Broken grammar might help attract attention. All I know is that it looks unintelligent to me, and if the headline had not been written by someone that I considered smart, I would not have bothered reading the content. Am I alone in judging the content by the headline?
Also, grammatical rules allow you to say precisely what you mean. Without them, you must rely on context and common expectations. This limits your ability to say the unexpected.
"Then beyond building clean water wells, covered sewers, and offering basic immunizations for measles etc. (the malaria vaccine is said to be coming), perhaps the best thing the Gates Foundation can do"
Fortune,
The Gates Foundation spends on agricultural research, vaccine development, etc. Even with tens of billions of dollars, it's unlikely that paying for traditional healers (is there such a shortage now?), except for experimental studies, will ever be reasonable given their opportunity costs.
Hey Robin,
Your blog is fun to read but on this one, I think you need to either read the study more carefully or brush up on you statistics before jumping to conclusions, especially those fed to you by bloggers like Megan who have a known ideological axe to grind. Here's what I wrote at your colleagues' Marginal Revolution blog about this study:
Health utilization was only 12% higher in the experimental group, yet, as per David Levine's comment, we cannot reject the hypothesis that the primary outcome improved by as much as 34% in the experimental group. Given the very low power of the test, it seems premature to conclude that additional health care had no effect. In fact, we can't rule out an effect more than 2x the amount of treatment.
Meanwhile, due to selection or the Hawthorne effect or some other unanticipated experimental design issue, even the control group apparently had a staggering 70% better outcome than the authors predicted before the study.
So what can we conclude from all this? Not much about healthcare in Ghana and even less about healthcare in the U.S.
But we can make some inferences about the criteria Megan McArdle uses for deciding which of the many medical studies done each year are important for readers to know about, and which are echoed by bloggers with sympathetic ideologies.
Best of luck in your laudable project to "overcome bias".
I think you are too soft on the critique. The following sentence is especially telling: "While it is clear fees must be abolished,...". Also, notice how they critique the original study in that "only health-related impacts were considered".
*
Nz, The study is not so underpowered when it comes to measuring mean Hb concentration, which is done for the whole population. They also have other indicators. The focus on a single under-powered indicator is misleading.
@jamie
"Tarzan"
For some reason, a lot of people feel a strange need to try to "correct" Robin on these things. May I direct your attention to the perspective of a professional headline writer?
"As with the last time we explored grammatical errors, I feel compelled to mention that copywriting and blogging should be conversational and engaging, and breaking formal grammatical and spelling conventions can often be a good thing. Every time I see a comment complaining about something like, oh, I don’t know…the improper use of an ellipsis or one-sentence paragraphs, I shake my head with sadness."
Deliberately breaking rules in headlines is a time-honored attention grabber, practiced by nearly all popular newspapers. Notice that OB now has in excess of 2.75 million hits. I'd say it's working for 'em.
Again with the Tarzan grammar. Should that be "Free docs do not help kids"? Or maybe "Free docs not helping kids"?