Medicine As Scandal

To me, the RAND Experiment seems clear evidence that medicine is a huge scandal.  Readers of the medical literature, as well as readers of medical media coverage and students in health and medicine, have all been given the strong impression that in the aggregate, more medicine produces more health.  This was the impression thirty years ago as much as today.

Yet our single clearest data point, the RAND experiment, confirms the typical result of aggregate correlation studies: we see no such relation.  Thus the medical research literature must suffer from severe biases, such as fraud, funding bias, treatment selection bias, publication selection bias, leaky placebo effects, misapplied statistics, and so on.  How else can we square the usual positive benefit found in medical publications with a net zero benefit?  Furthermore, what else but education and media biases can explain why this experiment, very expensive, well published, and the most important medical study ever, remains mostly unknown to medical students, professionals and the public? 

Further evidence of bias is found in the shameful way many try to claim the RAND experiment shows that medicine helps, via noting "significant" results, using statistical tests that do not correct for the data mining required to find those results.   For example, if you look at thirty outcomes, and are willing to break subjects down by both poor/rich and good/bad initial health, you should on average see six results "significant" at the 5% level, even if there were no effects. 

Tomorrow, I will post a petition to redo the RAND experiment today, and comments there will be devoted to letting people to sign that petition.

Added: The book Fiefdom Syndrome gives an example of a sales manager whose regular presentations to his manager usually focused on how sales were up for particular customers groups on particular products, and who didn’t want to talk about why total sales continued to fall.  This sales manager might complain "I keep telling you all this good news; why do you focus on that one piece of bad news?"  Similarly, studies of the aggregate effects of medicine are not just one more kind of medical study; they are a crucial check on biases in all the other studies.    

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://jamesdmiller.blogspot.com/ James D. Miller

    This is sort of like the scandal of actively managed mutual funds. The evidence against them is overwhelming. They are clearly a horrible investment compared to index funds, yet investors still spend billions in fees on them each year.

  • anonymous

    Robin,

    Usually people claim advances in medicine are responsible for the increases in life expectancy we have seen over the past couple hundred years, as well as decreases in child death rates, and decreases in diseases such as polio, small pox, and many others.

    What makes you place so much faith in one study. How biased are you by using this one study to confirm your beliefs? Guilty of confirmation bias yourself perhaps?

    Maybe you should think a little more about the evidence for the benefit of medicine before you jump off the deepend because of this ONE STUDY.

  • http://scienceblogs.com/denialism/ MarkH

    I’d have to agree with Anon here. You’re putting way too much emphasis on one study that was really studying something very different – medical utilization – and it showed a decreasing return for increasing utilization. What does that have to do with the efficacy of evidenced-based medicine?

    Every week, you see in NEJM, JAMA, and other major medical journals the results of trials which show benefits of medical interventions ranging from statin use to monitoring of breast cancer by mammography. We are seeing for the first time, an overall decrease in cancer mortality within the last few years. People are living incredibly long times and are healthier for greater portions of their life. Interventions like coronary artery bypass grafting, and other forms of interventional radiology are extending people’s lives by decades when they otherwise would have been dead, dead, dead. I don’t get this desire to cast away medicine over a single utilization study. It’s just nuts.

    Before you doubt the efficacy of EBM you should instead have a study that shows people are healthier without medicine compared to those that have care (comparison across countries would suggest they are not). And before you throw out the idea of allowing increased utilization or creating a more socialized version like Britain, France or Canada you should consider the US spends more on healthcare per capita, yet receives less care, and we have poorer outcomes. Our system is non-ideal, but throwing away medicine or calling it a “scandal” reeks of crankery to me. You seem, if anything, biased towards libertarianism at the cost of ignoring all other data in the world except your one study.

  • Chi

    Seriously then, as you say, “How else can we square the usual positive benefit found in medical publications with a net zero benefit?”

    Can you think of no other explanations? I am interested in your fervor for this study, but it doesn’t seem like you are really rationally dismissing the alternative lessons.

  • Biomed Tim

    MarkH, I think you misunderstood the post. The RAND experiment showed that more medical care does not produce more health “at the margin.” I don’t think Robin is trying to denounce medicine altogether.

  • joe

    Tim,

    What about the statements,

    “How else can we square the usual positive benefit found in medical publications with a net zero benefit? ”

    or

    “To me, the RAND Experiment seems clear evidence that medicine is a huge scandal.”

    or

    “the medical research literature must suffer from severe biases, such as fraud, funding bias, treatment selection bias, publication selection bias, leaky placebo effects, misapplied statistics, and so on.”

    Robin seems to have pretty much lost faith in ALL of the medical literature.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    All, I had a two year health policy postdoc, have published in health econ, and have been teaching health econ for seven years. I know of many other studies, and I hope to have more posts in the future about explaining health time trend. But I cannot communicate everything I know in a few short blog posts. The best I can see to do here is to point to the strongest describable items I see and summarize each one, one at a time.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    Robin, you should mention the part about whenever other studies control for other variables in a multiple regression, the weight of medicine comes out zero. Or link to other blog posts that do so. This is an important category of evidence and it is necessary to emphasize that this study is merely the largest and best, not that it accounts for a majority of the weight of accumulated evidence. Likewise for your previous blog posts that mention the mystery of increasing lifespan, etc.

    There will be newcomers and people with short memory spans who automatically assume that all the evidence you mention is all the evidence there is; if there is a huge mound of other evidence you must refer to it on every occasion when you refer to the RAND study. Get used to being quoted out of context.

  • joe

    Robin,
    That was a very weak rebuttal. We are not asking for you to “communicate everything you know.”
    But to “summarize each one, one at a time” while ignoring a large body of evidence is highly irresponsible. There has to be other conclusions we can make while taking into consideration the amount of evidence we have for the efficacy of medicine. Though the literature is surely plagued by some of the biases you mentioned such as publication bias, that does not imply that medicine is a scandal. Rather, we need to understand the potential for these biases and be weary of significant results achieved even in large scale clinical trials. We need better designed studies and more confirmatory analyses before accepting any revolutionary finding in a study as fact.
    Data mining is precisely the reason why the FDA doesn’t allow secondary analyses for the approval of a new drug… it is too easy to find significant results.

  • joe

    Eliezer,

    You are right, there is a huge problem with mulitple regression and their interpretation. You have to be really careful about which variables to include. Throw them all in and you are almost assuredly to find no effect for treatment. Multiple regression sometimes doesn’t answer the question we thought we were asking by proposing the model… we have to be really careful. Statistical tools are like a loaded gun in the hands of people who don’t know really understand how to use them.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Joe, I summarized one large body of evidence when mentioning “the typical result of aggregate correlation studies”, but apparently that does not have enough detail to engage readers here. I gave detail on one study and readers complain “that’s only one study.” I’m not sure what you want from me.

  • joe

    It’s one thing to talk about one study and to see what we can learn from it, to try to explain the results or find weaknesses in the study design, and to discern the generalizability of the results, all of which are things that you should be done with every study published, but it’s quite another to make the conclusions you did in your original post up above. Perhaps you would like to clarify your seemingly drastic conclusions?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Joe, if you’ll tell me what you think is unclear, then perhaps I can clarify.

    Mark, I don’t see what this has to do with “libertarianism” or why you accuse me of that. Yes, lots of positive sounding studies of specific treatments come out all the time; the question is how to aggregate all those studies and correct for biases.

    I put an “Added” on the post above.

  • Keith Elis

    I would like to propose something I thought about a few months ago, what we might call ‘good story’ bias. When faced with a decision between two alternatives, a person is more likely to choose the alternative with anecdotal support over the alternative with statistical support. (I don’t know if this idea already exists in the literature, so stop me if I’m not in new territory here).

    In the case of health care, statistics support the view that more medicine does not equal better health. However, good stories — from case studies in medical schools, to Uncle Jim talking about Grandma’s stroke — indicate that checkups, diagnosis, treatment, and follow-up are strongly related to health outcomes. Even when medical intervention exacerbates existing health problems or even causes death, a lawyer can explain to a jury exactly how the *right* checkup/diagnosis/treatment/follow-up could have changed the outcome.

    These are good stories. Good stories are pervasive in health care and most of the time the stories are completely factual and incontrovertible. ‘There is no question that the doctor misdiagnosed mom’s stroke and that she died two days later.’ These stories are hard to ignore, and aggregate in people’s minds (including the minds of doctors) to create seemingly strong evidence that medical treatment has a fantastic effect on health or un-health.

    I wonder if someone could render an opinion as to whether ‘good story’ bias as I describe it really needs its own label. I never brought it up because my initial reaction was that anchoring and availability bias adequately explain what I’m talking about. Now I’m not so sure.

    I’ll finish with a comment on this:

    “the medical research literature must suffer from severe biases, such as fraud, funding bias, treatment selection bias, publication selection bias, leaky placebo effects, misapplied statistics…”

    Robin, I don’t think it’s necessary to posit these complicated institutional biases. The simpler explanation is that the aggregation of certain standard cognitive biases is largely responsible for the way consensus has built on this issue. An effective treatment of the issue would deal with these reasonably well-understood and accepted facts of psychology first before building a somwhat weaker case regarding biases at the level of society. Maybe I’m taking you too literally and your list above is describable at the level of individual biases.

    Keith

  • TGGP

    MarkH, the only reason for you to assume Robin is “biased towards libertarianism” is if you are already biased AGAINST it. Consider “Money Driven Medicine” by Maggie Mahar. Tyler Cowen says of it “The book has the most coherent, supportable, and fleshed out anti-market story I’ve seen“. Yet the book fits in quite well with Robin Hanson’s point. Spending more on medicine is not resulting in better outcomes.

  • joe

    I don’t think your views on medicine could possibly be as bad as you made them seem up above in your original post. Correct me if I’m wrong, in the case of medicine, it seems clear that we have a case of diminishing marginal returns and that more medicine may not always be as much better as we are led to believe. However, there must be certain medical treatments which are in fact beneficial as evidenced by trends in diseases and life-expectancy. I suppose the real question from a medical perspective is which treatments really are beneficial and which aren’t. Would you agree?

    Clinical trials attempt to answer the question of whether or not the treatment is better than the standard of care. The result of this is the finding of significant results for treatments which might be no better than the standard of care. Especially because of publication bias, it makes it difficult to discern which are truly positive results and which are merely false positives. I share your frustration in that area and think that there need to be more confirmatory analyses for positive results before we conclude a treatment effective. However, I wouldn’t go so far as to conclude from the rand study:
    “How else can we square the usual positive benefit found in medical publications with a net zero benefit?”

    If we are interested in the net benefits of more medicine and discerning effective treatments, I don’t think that assigning people to either healthcare or no healthcare answers the question of interest… it brings in too many factors including psychological factors such as self-selection for treatment as well as leaving treatment decisions in the hands of doctors. Clinical trials attempt to control for these factors in order to isolate the effects of the treatment. The rand experiment should really be called more of an observational study if trying to determine the net benefit of more medicine. However, I would say that it is a valid experiment from a social policy perspective since it attempts to address a different question. The conclusions drawn should be related to the question asked as dictated by the design of the study. Do you agree?

  • Keith Elis

    I should clarify one thing in my comment above. When I say “a person is more likely to choose the alternative with anecdotal support over the alternative with statistical support” I have no evidence that this is true. I am suggesting that the problem posed by the RAND study might lead us to posit good story bias as a hypothetical explanation of why the study has not led to an updating of the consensus. That is, good stories about medicine indicate the study must be wrong.

    Good story bias might also be employed to explain the preference for active money management in mutual funds over passive strategies.

    Keith

  • Jor

    _Thirty_ year old study. Like I said in another post, most preventive medications (i.e. medications for non-acute care) — really are very new. Most preventive regimes, for things like Diabetes, Hypertension, Heart Failure, etc. have all changed dramatically in the 80’s and 90’s.

    The US medical system does incentivze many diagnostic tests and interventions that are mostly useless (some even harmful) and have an net-zero aggregate outcome. However, that does not mean all of medicine has a net-zero outcome.

    I find it hard to believe that even you, yourself take this seriously. Are you seriously never getting your cholesterol, blood pressure, or blood sugar checked?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Jor, we have many more recent aggregate correlation studies, and even a thirty year old bias this large is big news.

    Joe, it is far from easy to figure out what has caused long term health improvements. I’m not sure what you have in mind for an alternative non-price way to test aggregate medical effects.

  • Jor

    Robin, post a link to what you think is a good recent observational study, showing net-zero outcome. Not cross-national. In terms of the bias from 30 years ago, — modern medicine is only 100 years old (i.e. with Osler & John Hopkins). Let’s just say, if you treated people the same way as was done 30 years ago, you would get sued for mal-practice. So, I don’t find it such a big deal.

    From my perspective, the scandal in medicine is expensive diagnostics and therapuetics of very questionalbe utility that are incentivzed by the current reimbursement schedule for physicians.

    If there is a net-zero effect of medicine on health, I would doubt it’s because of lack of efficacy (or bias) on EBM-based medicine (many recommendations, which have been tested by multiple people, in many large trials). Instead, I would guess its because people doing questionable procedures or questionable therapuetics wind up wiping out the benefit of proven therapuetics with complications (esp. where procedures have no improvement on mortality in the first place, e.g. PTCA in asx. CAD, CABG in many cases, Spinal Surgery for back pain (in many cases), Anti-Arrythmics, etc.)

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Jor, my post above does link to a good recent correlation study, at “typical result.” I don’t doubt that medicine has changed; spending has tripled. The question is whether the ratio of helpful to harmful medicine has changed.

  • Jor

    Thought it was going to RAND Study. I’m peripherally aware of the dartmouth geographical variation stuff — but from what I understand, they don’t doubt the veractiy of the published literature, but believe that the problem is useless tests and interventions, as I mentioned.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Jor, the geographic variation data shows zero correlation between more medicine and more health, for recent times as well as past, as I claimed. I’m less interested in the opinions of the authors than in the actual data results.

  • http://www.thebandarlog.com Ben A

    I, like Joe, would be eager for more links to studies suggesting that increased treatment does not lead to better health outcomes. This is not a literature with which I am familiar.

    The linked study on last 6 months of life seems plausible. It may well be that a great deal of end-of-life therapy is minimally effective. It is highly implausible, however, that the multitude of studies showing consistent survival benefit for chemotherapy across tumor types result from bias. Are there any studies that look at specific conditions (rheumatoid arthritis, prostate cancer, diabetes, etc) and seek to correlate spending to results?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Here is another of the many aggregate correlation studies.

  • David J. Balan

    As I understand it, all the RAND study shows is that the marginal health care consumed by people who have better insurance over and above that consumed by people who have worse insurance (nobody in the study had no insurance) doesn’t get you any health benefits. That’s an interesting result (that may or may not still hold if the study was done today). But I don’t see how that demonstrates that there is anything wrong with all of the individual studies that show that this or that medical treatment works. Those studies compare *all* the people who got the treatment with *all* the people who didn’t. Why isn’t it possible that lots of the treatments really do work for the inframarginal people (and so for the average people), who more or less get them regardless of the quality of their insurance, while it’s only worthless for the marginal people?

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    A bias I need to write on at some point: In a hunter-gatherer society of 200 people, you were rarely more than one inferential step away from everyone else. Everyone had the same background knowledge. Someone who made statements that could not be perceived as true in one step was a lunatic or a liar.

    Robin, you’ve been working in this field too long. You’ve forgotten how strange it sounds – in a world full of anecdotes and advertisements, which are the shared background beliefs – to claim that marginal increases in health spending have net null effect.

    If you’re going to post the petition tomorrow, it had better include a huge heap of studies showing no effect – a nice bucket of damning evidence – because you’ve forgotten how strange the basic thesis sounds.

    You may get new readers for the petition who haven’t been following this blog. Take their fresh minds into consideration.

    Because if it’s not obvious in one step, you’re just a lunatic.

  • joe

    When a clinical randomly assigns 5000 patients to either the new treatment or the standard of care, they have the statistical power to see a significant difference. Also, each of the people usually has something in common such as a pre-existing condition.

    If you only look at 7700 people, many of which are not going to be a match for any one condition, it is going to be impossible to see the effect of increased health from any one particular treatment… not to mention the fact that you may not even have a subpopulation for comparison of two similar treatments ( one more expensive than the other) for the purpose of a valid comparison.

    I bet that there were enough healthy people in the study that it would be nearly impossible to see an overall effect in any one category…. it is only when we concentrate on a large population of diseased people that we are able to see any positive significant effect in a clinical trial. Maybe there were just too many average, relatively healthy people in the study.

    I think I am onto something.

  • http://scienceblogs.com/denialism/ MarkH

    Maybe I jumped the gun on the libertarian link. I guess I was generally suspicious of the aim of attacking decreased healthcare utilization as a result of this study (from 30 years ago). It seems anti-socialized medicine to me, when it seems clear that the socialized systems are doing a lot better job on primary outcomes. Anecdotes aside (in about 1 minute people will show up complaining about MRI access), the socialized medical systems which provide increased access for prevention and maintenance are doing a lot better job in metrics of medical success.

    As far as saying that I’m over-reading an attack against evidence based medicine in this post – no, it seems quite clear that that’s the implication. As other commenters have noted, the writing of the post suggests an attack on medicine as a whole as a “scandal”.

    It probably wouldn’t hurt to repeat this study in today’s medical climate. A lot has changed in 30 years, and there are a lot of things that are currently underutilized due to economics. I would be curious as to the outcome to such a study. However, using one study, from 30 years ago, to suggest that medicine is a “scandal” is troublesome. It just reeks of something, maybe I can’t quite put my finger on it yet, but I don’t like it. Smells like a precursor to woo.

  • http://yorkshire-ranter.blogspot.com Alex

    Does Robin Hanson have private health insurance? I think we should be told.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Eliezer, I added lots of links to correlation studies in today’s petition post, but honestly, most of my students don’t believe me after an entire semester course; I’m stuck with this sounding strange no matter how much evidence I muster.

    Joe, many of the correlation studies include far more people.

    Mark, my claim is about medicine as actually practiced, not about evidence-based medicine as it might be practiced. Most medical practice has little solid evidence behind it.

    All you who think the RAND study too small and old for comfort, please sign today’s petition asking for a bigger study today.

  • Stuart Armstrong

    Robin, what solution would you suggest, if your point is correct (as it seems to be)?

    HMO’s, state selected health care, or entirely private health care would all start to address the issue – do you have any preferred direction?

  • joe

    Robin,

    Did you stop for two seconds and think about the potential problem of including too many healthy people in the study if you are interested in the benefit of medicine.

    Think for 2 seconds about who medicine is supposed to help. Think about who is the population of most clinical trials. It is sick people who need the medicine they receive. Your stupid study tells us nothing about how much healthier a sick person would be if he had better health insurance. You just have too many healthy people to see an effect. So wbat if clinical trials oversample sick people…. that’s what they are supposed to do. They are supposed to concentrate on their poplulation of interest.

    Think for 2 seconds about the populations of people who are relatively healthy in your aggregate effects study. Why would you expect to see a difference at the end? Western medicine is reactive, not proactive, and for the small proportion of people who actually did get sick during the study, I guarantee that the people who had better health insurance were healthier because of it. They probably went to the doctor sooner and recovered faster. More regular checkups inreases the ability for early discovery of a disease. Sometimes, this is the difference between life and death.

    I know that you probably aren’t going to spend 2 seconds actually thinking about this before you respond, but I thought that I would try anyway.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Stuart, I have creative suggestions for better institutions, but at the individual level my simple suggestion was “if you would not pay for medicine out of your own pocket, then don’t bother to go when others offer to pay.”

    Joe, the net effect of medicine includes both making sick people better (and sicker) and making well people sicker (and better). We need both groups in the data to see the net effect.

  • Stuart Armstrong

    I guarantee that the people who had better health insurance were healthier because of it. They probably went to the doctor sooner and recovered faster.

    Do you have any evidence to back this up? The studies Robin linked to seem rather clear that this is not the case; the minimum courtesy in response would be to present studies the other way (the fact that the RAND study is old is evidence against it being true today; it’s not evidence that it is false today. The effects detected in there may well have increased).

  • joe

    “Joe, the net effect of medicine includes both making sick people better (and sicker) and making well people sicker (and better). We need both groups in the data to see the net effect.”

    That is absolutely crazy. The purpose of medicine is to make sick people healthy. Clinical trials establish the efficacy of medicine in treating disease. I see that you didn’t think for 2 seconds about what I said above. About why there shouldn’t be a difference in your aggregate effects study.

    SOMEONE out there must understand what I was trying to say in my above post, or at least give me a better answer as to why I’m looking at it the wrong way.

  • joe

    Stuart,

    For many disesases, early detection, especially for most cancers, is key to survival. People who go to the doctor earlier must have a better chance for early detection.

    In the case of diabetes, the longer you go undiagnosed, the more dangerous it is to your body.

    I am sure there are plenty of other examples.

  • joe

    Stuart,

    The whole point of my previous post was to point out that any benefit sick people receive is going to be washed out by including them with too many healthy people… this is statistics 101. If you compare population totals for various endpoints at the end of one of these studies, they shouldn’t be much different because you are not concentrating on the people who are sick and need treatment.

    Clinical trials concentrate on people who are sick and demonstrate which treatments are more effective.

    Rand-type studies ask a very different question and the answer should not be confused with the benefit seen in the medical literature. The comparison of the two is absolutely RIDICULOUS.

  • Stuart Armstrong

    “if you would not pay for medicine out of your own pocket, then don’t bother to go when others offer to pay.”

    There I do think you’re going farther than the evidence warrants (the Rand study didn’t include any uninsured, for a start). Your statement also implies that patients know what is better for them; but there could be many other reasons why extra health-care doesn’t buy extra health. (I’m not saying your statement is wrong, I’m just saying it lacks evidence). It’s also irrelevant in a fully socialised system – when people get health-care for free, they have no real idea how much they would pay for medicine, were it not free.

    Your idea on http://hanson.gmu.edu/buyhealth.html is impressive, however. Has anything similar to it been implemented anywhere?

  • Michael Sullivan

    Joe: 7700 people over a 3 to 5 year period is a pretty big study, even if it’s mostly healthy people.

    The population of people where I would hear about their serious medical problems is probably less than 7700, and yet I probably hear about one or two such every couple months. There is no shortage of people with real health problems in almost any random sample of 7700 people over 3 years.

    Perhaps the study should be bigger, but this study is a significant bit of evidence all by itself if there aren’t any other unpublished studies like it with different results.

    I am not really surprised at all, even though I haven’t done the amount of work that Robin has in these investigations. I’ve read medical literature, and by the standards of other sciences evidence is *profoundly* lacking for many very mainstream medical dogmas. Far too much of medicine is one study science with a great potential for suffering data-mining bias. Statistical significance is mistaken for AbsoluteTruth[tm], a study iis published and nobody bothers to confirm or refute it, because there is no money or glory in repeating some other guy’s experiment. You get money and fame by getting your *own*, *original* statistically signficant result. Well, I can tell you how to get those: Do a whole lot of random experiments. About 1 in 20 will give a result that is significant to the standard 95%CI. This will be true even if your hypothesis and the results have nothing whatsoever to do with each other.

    Ignoring this data mining bias is the very same innumeracy that grips gamers in a tizzy about random events on game servers. “What is the chance of X?” is a question seen commonly on backgammon and poker boards from some poor punter who’s just had a run of bad luck. Well, it depends how many chances you had to get it, and how big the set of X’s that would make you ask this question is! The chance that I would get dealt QsTh6d3s2c in a hand of 5card draw poker is…. 1 in 2.5 million! Can you believe that?

    These studies are the same thing on the medical scene. We do a whole raft of experiments and then pull out the ones that look interesting afterwards and publish those. That’s data-mining, and if a bayesian wants to update her priors based on such a result, the data-mining has to be corrected for, which is very hard, since nobody knows how many studies went unpublished. That’s why replication is so important to building up a body of knowledge. Having a positive study be replicated many times is the correction for data-mining. Now we have a new hypothesis and we will specificallly test *it* and no other and see what happens. Do that two or three times and get significant results each time, and now your evidence is *very* strong. Don’t do it at all, and the evidence is pretty weak for any study where it’s feasible that a whole bunch of similar insignificant results could have been found and gone unpublished.

  • http://scienceblogs.com/denialism/ MarkH

    Something is weird about the comments on this thread. Is anyone else seeing the person responsible for the comments shifted down one?

  • Stuart Armstrong

    NB: one major caveat of the RAND study is that they didn’t (couldn’t because of Medicare) include anyone aged more that 61 in the study. So the elderly may be in a different situation. The RAND II should find a way of including them.

  • http://scienceblogs.com/denialism/ MarkH

    Wait no, I’m an idiot or the display fixed itself. Dammit.

    Anyway, I think joe has made some good points here that haven’t been addressed. You have tunnel vision for this one study, and while it would probably be a good idea to repeat something like this in today’s medical climate (which has changed enormously in 30 years with HMOs, increasing costs and the end of fee for service), it is irresponsible to suggest that EBM doesn’t work because of increased utilization, or that utilization today is comparable to that of 30 years ago. Your comment about medicine making people sicker also has me wondering. Is this the beginning of some woo?

    Michael Sullivan, you would be interested in the work of John Ioannidis (I put a video of one of his lectures up at google video). He addresses your complaint, but I think it is inaccurate to suggest that medicine frequently revolves around single-studies. This is just not the way the medical literature looks. Things get replicated constantly, we routinely monitor and re-check and write systematic reviews and meta-analyses and consider these before changing protocols for patient care. I think your criticism, while based on a good knowledge of the problem of statistical significane, is not valid, since, well, medicine has thought of this problem too.

  • Michael Sullivan

    markh: I should pull back from the implicit suggestion that all Medicine faces this problem. Obviously many or most standard results are reproduced, and I would guess that the opinions of doctors and researchers as a whole about what evidence we have for various propositions is fairly well correlated with what a Bayesian aware of all the public evidence would think. But I don’t think medicine results are anywhere near as hard as in other sciences where experimentation is easier and cheaper or has been working in a smaller problem space for a long time.

    There seems to be lot of medical information based on fairly small amounts of data (even if there is far more that is quite well founded). Where I’ve seen more problem is areas with a small effect on lots of people. When it comes to diagnosis and treatment of serious diseases and injuries, I’m very confident in established medical opinion. I’m not so confident when it comes to things like nutrition and exercise and other lifestyle issues, which recommendations can have a huge effect on overall health in the aggregate, but generally a small effect on any given individual (compared to say, curing scarlet fever with antibiotics).

    I’ll also note that I have an even bigger problem with underinformed studies taken as gospel in the social sciences than in medicine (and I’m much more familiar with some of those domains than with medicine). In both cases, though, the media and general infosphere is a much bigger culprit than expert practitioners, most of whom share my uncertainties and concerns about insufficiently replicated results.

    Thanks for the recommendation on John Ioannidis. I will check him out.

  • billswift

    @ joe “That is absolutely crazy. The purpose of medicine is to make sick people healthy.”

    Many historians and writers on contemporary issues have talked about this problem. I hereby name it “The Fallacy of Intended Results.” The beleif that the intended goal of a law or policy matters to the actual result.

  • billswift

    Actually, anyone even marginally economically literate should be familiar with this in the context of rent control, if nowhere else.

  • joe

    Bill,

    Medicine is not some failed economic policy that is supposed to work in theory but doesn’t actually work in practice. Clinical trials that have been well run and replicated demonstrate the efficacy of medicine to treat certain diseases.

    Medicine is not simply INTENDED to help the sick, it has been proven to actually help the sick, which it is supposed to do by definition. To test a medicine’s efficacy, you actually have to put it to the test on SICK people… that was the whole point of my comment to Robin, which you clearly didn’t understand.

    Nice try though.

    Anyone even marginally statistically literate should know that the question of whether or not a medical treatment is effective must be tested on a population that is actually sick.

    When you need 5000 SICK patients in your study to determine that a SINGLE treatment is significantly effective, how are you possibly going to see an effect in a population of relatively healthy people. You couldn’t possibly have enough sick people to demonstrate the efficacy of more medicine for any outcome of interest… from a medical perspective, you don’t really have a sample of 7700 people. There’s a little thing in statistics called power anyone who’s statistically literate should have heard of it.

    If you want to demonstrate the efficacy of medicine in treatment of a disease, the population of interest must be SICK people with that disease.

    If you are interested in the social policy of free healthcare and price incentives, then run your RAND studies. I think that it is a great question to ask from a social policy perspective. But don’t go off interpreting your results as saying that there is no net benefit to medicine and that medicine is a scandal. Clinical trials set up a valid comparison of two treatments to attempt to demonstate the efficacy of medicine. I am not even sure patients in the rand study who suffered from a disease even recieved two different treatments based on their differing medical coverage… which is actually what you need if you want to establish the efficacy of a treatment over another, which is what clinical trials try to do.

    If your study doesn’t address that question, you shouldn’t try to answer it.

  • Dave

    >Medicine is not simply INTENDED to help the sick, it has been proven to actually help the sick, which it is supposed to do by definition.

    Sure. It’s also been proven to occasionally harm sick people, occasionally harm well people (nosocomial infection, pharmaceutical errors, frank malpractice, etc.), and quite often have no effect on outcomes (sometimes because it is ignored or avoided). None of these are intended, but all occur, and do so more often than most people realize (particularly the “no effect” one). For a study like Rand, all of those factors end up being netted in. For clinical trials of individual treatments, many of them are (correctly) ignored as irrelevant to the experiment.

    The Rand study doesn’t indicate that medicine has no effect. The Rand study indicates (at most) that the sort of medicine that you wouldn’t pay for yourself has no net effect. Given an infection, most people will go to their doctor and get some antibiotics, even if they have to pay for it themselves. This means that the fact that antibiotics are examples of incredibly effective medical treatment doesn’t show up in the Rand study, since both populations are getting antibiotic treatment at basically the same rate. Whether or not you have to pay for it, you’re likely to get your kid vaccinated for rubella, if only so they can go to school. The fact that rubella vaccination is an incredibly effective preventative treatment similarly doesn’t show up in the Rand study. (Moreover, even if your kid doesn’t get vaccinated for rubella, he still probably won’t get it, since nearly everyone else is, because rubella vaccination is incredibly effective!)

    If the sort of medicine that you wouldn’t pay for has no net effect, but the sort of medicine you would pay for has great effect, that would be odd, but not obviously impossible. It could just mean that patients are avoiding overuse of medicine and use of marginal medical interventions because of expense, and are doing so at rates that balance out any underuse of successful medical interventions due to expense. Given the state of the art of medicine, I can’t see how anyone could say such a situation was unthinkable. Given the state of the art of medicine thirty years ago (when surgeries were much more invasive, and pharmaceuticals much less effective), the Rand results seem quite plausible, if not quite so earthshaking as Robin portrays. If you tried to give someone today the sort of medical care available thirty years ago, you’d be performing grevious malpractice.

  • joe

    “It’s also been proven to occasionally harm sick people, occasionally harm well people (nosocomial infection, pharmaceutical errors, frank malpractice, etc.) ”

    I agree that these are all problems in the real world, but should we blame the medical procedures themselves, or rather are they the fault of imcompetent doctors and hospitals not taking enough precautions to prevent mistakes as well as the spread of bacteria in hospitals?

    However, I do tend to agree with most of your other comments about conclusions that can be made from the Rand study.

  • Dave


    I agree that these are all problems in the real world, but should we blame the medical procedures themselves, or rather are they the fault of imcompetent doctors and hospitals not taking enough precautions to prevent mistakes as well as the spread of bacteria in hospitals?

    I don’t believe the Rand study or anyone here was blaming the procedures. The Rand study is about the state of medicine as a whole, rather than any line item. If you go at it from the point of view of individual procedures being effective, you’re just asking to be confused. You’ll miss the fact that the forest seems to be a bit sick by pointing out individual healthy trees. That’s probably why most of the commenters here sound so deeply confused.

    While malpractice and nosocomial infections are easy to dismiss as incompetence, overtreatment is much less so. If you autopsy someone over 60, there’s a good chance you’ll find a small, undiagnosed cancer, even though they died of something completely different. If that cancer had been found (say through more aggressive monitoring), it would have been treated (who lets a known cancer go untreated), even though the patient never had any symptoms from it. Cancer treatments are Not Fun, and indeed are often indirectly lethal. Excessive use of medicine can have negative health impact even if the medicine is the best available.

    Further complications in interpreting the Rand study arise because of the happy circumstance that people some times just “get better” from their conditions. Statistically, the incidence of “just getting better” lessens the net value of medicine found by the Rand study. This is because if you forgo medicine and simply get better, you’re a net null in the Rand data set, while if you would have just gotten better but underwent treatment, you’re a statistical net negative (due to side effects and risks of treatment). Healthy, young people (the Rand study only included those below 61) living in healthy, safe environments (most of the US in 1977) are pretty good at simply getting better from a lot of problems, so this effect is probably significant.

  • michael vassar

    You know though, it seems to me that a cheaper option than RAND II for testing the
    general efficacy of western medicine would be to simply provide cheap
    or free western medical clinics to a random sub-set of African villages
    where nothing similar is available. Natural charitable experiments of
    this sort surely exist already.

  • Mark

    My sister is an epidemiologist at a major research hospital. A few years back her group was awarded a grant to produce a study determining when women should routinely begin annual breast cancer screening (mammograms)–age 40 or 50.

    Their results concluded that the incidence of breast cancer before age 50 did not justify routinely screening women for breast cancer before age 50. Not only did you have to consider the cost in terms of resources, but also the accumulated exposure to radiation, both of which did not warrant annual screening before age 50.

    The doctor who headed the study was told by the NIH that their study countered the conclusions of a study conducted by
    radiologists (age 40, of course, go figure) and would therefore be discarded in favor of the radiologists’ report.

  • joe

    Michael,

    I think that is a great idea if you want to test the efficacy of medicine versus no medicine. However, it asks a very different question to which most reasonable people probably agree that we already know the answer.

    The rand study asks a very different question and I really don’t think that it should be taken as evidence against the efficacy of medicine. Nonetheless, if there really are people out there who don’t think that western medicine is beneficial, then go run a study like Michael is suggesting.

  • joe

    Mark,
    You bring up a very interesting topic, especially since early detection is key to surviving breast cancer and/or preventing metastasis.
    In regards to a recommended age for everyone, I am not sure that can and should be estabished. For people who have breast cancer in their family, they are at higher risk to develop breast cancer than the general population. For these people, the benefits of an earlier screening probably far outweigh the costs.

  • joe

    “I don’t believe the Rand study or anyone here was blaming the procedures. The Rand study is about the state of medicine as a whole, rather than any line item. If you go at it from the point of view of individual procedures being effective, you’re just asking to be confused.”

    I really am not confused about the results of the Rand study. I am not even surprised by the results. I am, however, surprised by some of the conclusions people try to make about medicine as a whole from the Rand study. I think that if you reread the original post by Robin, and some of the earlier posts on this page, you will understand my frustration with people taking the results from the Rand study and concluding that medicine, overall, is a scandal and that all medical journals are only filled with false positives due to biases and running too many experiments.

    If you really want to test the overall efficacy of medicine, you have to randomly prevent a group of people from receiving medicine (which is highly unethical) and see how they fare compared to people who received medicine… or, like Michael suggested above, run an experiment in Africa.

  • Dave

    You bring up a very interesting topic, especially since early detection is key to surviving breast cancer and/or preventing metastasis.

    True, but if you push the recommended mammogram age too early, then most of the positive mammograms will be false positives. This means that a whole lot of women will be subjected to both the (small) risks of the followup biopsy, and the (very large) stresses of hearing “maybe we need to look at this a bit closer…”. These are the sort of tradeoffs that get made with EBM.

  • Dave

    I think that if you reread the original post by Robin, and some of the earlier posts on this page, you will understand my frustration with people taking the results from the Rand study and concluding that medicine, overall, is a scandal and that all medical journals are only filled with false positives due to biases and running too many experiments.

    Medical journals probably are filled with false positives, mostly because .05 is too low of a certainty given the number of studies published. After all, .05 means that every couple of issues, NEJM publishes something flat out wrong, which is nonetheless becomes the conventional wisdom. I also think that enough demonstrated instances of “fraud, funding bias, treatment selection bias, publication selection bias, leaky placebo effects, misapplied statistics, and so on” can be demonstrated that Robin can’t actually be faulted on that point. There are other indications (overly wide inter-hospital and inter-physician variations in treatment recommendations and outcomes, in particular), that indicate that medicine is, if not scandalous, at very least a lot more suspect than the public as a whole recognizes. Robin was probably a bit over the top, but it’s not far from being a fair cop. I won’t be signing the petition, though, mostly because those are billions I can see better Bayesian uses for.

    Of course, I’m just a software engineer, and 95% of us are charlatans and suck-tits (oy, you should see my resume’ pile). It’s not like I’ve got a lot of room to talk.

  • joe

    Dave,

    I don’t deny the existence of these problems. However, the medical community is making strides and the newest treatment doesn’t become widely used just because it found a significant result and made it into a journal. These significant results, however, are important to investigate in large-scale confirmatory analyses. For the purpose of advancing medicine, it is critical that potentially important discoveries be published and investigated. I understand, and well-informed people understand… the media and the public unfortunately don’t understand, and I’m not sure what to do about that one.

    In regards to most of the biases mentioned, a well-designed randomized clinical trial is able to account for most of those problems.

  • joe

    “True, but if you push the recommended mammogram age too early, then most of the positive mammograms will be false positives. This means that a whole lot of women will be subjected to both the (small) risks of the followup biopsy, and the (very large) stresses of hearing “maybe we need to look at this a bit closer…”. These are the sort of tradeoffs that get made with EBM. “

    I completely agree. But wouldn’t you also agree that people who are at higher risk would be well-advised to start receiving routine mammograms earlier than others.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Joe, by “medicine” you appear to mean something that happens in a clinical trial, where by “medicine” I mean what happens when ordinary people go to see a doctor. I was clear to distinguish the common medicine that everyone gets and the extra medicine that some people get when they face a lower price of medicine, and to make my strongest claims about this extra medicine.

  • joe

    Robin,

    On a scale of 1-10, I’d give you maybe a 3 for making that distinction clear…even in your original post.

    “To me, the RAND Experiment seems clear evidence that medicine is a huge scandal.”

    “How else can we square the usual positive benefit found in medical publications with a net zero benefit? ”

    Can you perhaps reference a few articles in medical journals concerning your so-called “extra medicine”… and connect that with the “extra medicine” administered in the Rand study?

  • BillK

    Two comments.

    1) You have to distinguish between the theory of medicine and medical treatments (good) and medical practice as it is actually done in the US (poor).
    The problem in the US is not medical theory. The problem is improving the very variable experiences that customers receive. Some find it wonderful and life-saving, others have terrible experiences. Average = little overall benefit.

    For example
    http://www.sciencedaily.com/releases/2007/05/070515074645.htm
    May 16, 2007
    US Continues To Lag On Health Care, According To New International Comparison

    The U.S. health care system ranks last compared with five other nations on measures of quality, access, efficiency, equity, and outcomes, in the third edition of a Commonwealth Fund report analyzing international health policy surveys. While the U.S. did well on some preventive care measures, the nation ranked at the bottom on measures of safe care and coordinated care.

    2) the second point is the huge scale of medical fraud in the US.
    Both via Medicare and self treatment by the general public, who will buy any snake oil or miracle cure that has a colourful advert.
    This impacts heavily on the poor outcome of those receiving so-called medical treatments.

  • Douglas Knight

    BillK,
    Are these “measures of quality, access, efficiency, equity” correlated with the measures of “outcomes”? Are these measures of outcomes even correlated with actual outcome? The article you link to talks about “potential years of life lost to diabetes.” The gap between what the medical literature predicts for life expectancy and the real world dwarfs the gap between the US and Europe.

    Do snakeoil products really have a negative impact on outcomes? Are they any less popular in Europe?

  • http://profile.typepad.com/halfinney Hal Finney

    I agree with the sentiment in the petition and signed it, but I was confused by the technical jargon in the final sentence:

    “Whatever other purposes such an experiment pursues, it should try to make clear the aggregate health effects of variations in aggregate medical spending, variations induced by feasible regimes of quality control, including free patient choice induced by a varying aggregate price.”

    I think I understand the first part about aggregate spending: if we spend X% more on medicine, how much better health to do we get? But what about “variations induced by feasible regimes of quality control”? This seems to have something to do with the distinction between medicine as practiced in theory (in controlled studies, say) versus how it is practiced in the field. But the verb “induced” is tripping me up. And the last part: are we asking that the study specifically charge subjects different amounts and see how much medicine they are willing to buy? Why do we want to know that; is the point to distinguish whether medicine that people are willing to pay (more) for produces more health than “extra” medicine that comes cheap or free?

    Overall I could benefit from some expansion of the logic behind how one might wish to see such a study constructed. I can see that it is not a trivial matter.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Hal, the first RAND experiment included an HMO variation, where HMO administrators were a “feasible regime of quality control.” Such administrators decided which medicine was good enough to give people.

    Tyler Cowen responded to my petition saying he would like to see “How much healthier would we be if we retargeted expenditures to some commonly recommended areas, such as pre-natal care and prescription drugs?”

    A new experiment could include several such systems for choosing which medicine is good enough to give patients, and see which regime actually works best.

  • BillK

    The new Michael Moore film ‘Sicko’ is getting rave reviews.
    See the BBC for a selection:
    http://news.bbc.co.uk/1/hi/entertainment/6673039.stm

    I’ll quote from the ‘Variety’ review
    http://www.variety.com/review/VE1117933678.html?categoryid=31&cs=1

    Quotes:

    “Sicko,” an affecting and entertaining dissection of the American health care industry, showing how it benefits the few at the expense of the many.

    Employing his trademark personal narration and David vs. Goliath approach, Moore enlivens what is, in essence, a depressing subject by wrapping it in irony and injecting levity wherever possible: a graph shows America’s position in global health care as No. 38 — just above Slovenia — and is followed by film footage of primitive operating conditions; and he offers a long list of health conditions that can deny a person insurance coverage, with the list scrolling into deep space accompanied by the “Stars Wars” theme.

    Pic explores why American health care came to be exploited for profit in the private sector rather than being a government-paid, free-to-consumers service as are education, libraries, fire and police.

    Pic starts by sketching a gamut of health-care horror stories from average Americans: those who can’t afford insurance, those who are denied coverage for various, often ludicrous reasons, and those who believe themselves well-protected, but find that the moment they avail themselves of medical services their insurance provider uses obscure technical reasons to refuse coverage, retroactively deny claims and cancel insurance, or raise rates so astronomically that the patient is forced into the ranks of the nearly 50 million uninsured.

    The congressional testimony of a former Humana medical director provides a devastatingly direct description of what she calls “the dirty work of managed care.” Constantly told that she was not denying care to patients, rather simply denying them Humana’s coverage, her career advanced as she saved her corporation money.
    —————-

    Sounds like a film worth seeing.

    BillK

  • joe

    “A new experiment could include several such systems for choosing which medicine is good enough to give patients, and see which regime actually works best. “

    I think this is a great idea…. unfortunately, you are going to need a MUCH larger sample size. Any time you start looking at individual medicines, you are going to need a large population of people that actually received the medicine… your group of 10-20 thousand people will not have enough people receiving any one particular medicine to make the determination of “which medicine is good enough”… especially if you are going to account for multiple testing.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Joe, an experiment could show that a certain system for choosing medicine did better on average without evaluating individual medicines.

  • joe

    Yeah, but doesn’t that put you down a slippery slope on which positive effects could get lost amongst negative ones. Each system proposed will have many factors and decisions made and at the end of the day, you probably won’t be able to distinguish which of those factors or decisions helped or hurt the final outcome… which is something I assume we would like to know in attempting to create a better, more efficient healthcare system.

    I am not so sure the problem is the system for choosing medicine, but rather the apparent holes in our knowledge regarding what helps and what hurts.

    Shouldn’t information on individual medicines allow for the configuration of an optimal system for choosing medicine.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Joe, the purpose of developing a new better system is different and much more expensive than the purpose of testing the quality of the systems we have.

  • joe

    Robin,

    In your original post, correct me if I’m wrong, you seemed frustrated with the state of healthcare in the United States and concluded that we must be performing many procedures which have net benefit zero.

    All that I am saying, is why don’t we try to find out which procedures have net benefit zero, or are harmful, and which are helpful.

    Your original post spoke to the problem of bad information being used to make medical decisions. Why don’t we try to improve our knowledge of medicine before we create a new system?

  • Carl Shulman

    Joe,

    If a $100 million RAND experiment, Mk II, found that cutting health care expenditures by 30% had negligible health effects, this would suggest a great way to improve our knowledge of medicine: cut Medicare, Medicaid, and the tax subsidies to employer insurance and raise $500 billion to do truly gargantuan experiments and isolate helpful and harmful components of medicine.

  • Jimmy

    Either I’ve misunderstood your conclusions, or you’re not making any sense at all.

    You said: “To me, the RAND Experiment seems clear evidence that medicine is a huge scandal.”

    Medicine? The ENTIRE field of medicine? The basic idea of “This person is sick, we will perform action X and then he will feel better”? Or just most medicine as it is practiced today? Either way, I don’t understand your conclusions.

    The RAND experiment was not a test of “The effect of medicine on random people”; it is better described as a test of “The effect of giving free medicine to random people”. That’s a big difference.

    Perhaps you notice this difference, considering your use of the word “more” in “[we] have all been given the strong impression that in the aggregate, more medicine produces more health.” But I’m not precisely sure what you mean. “More” compared to what? Do you mean “more” compared to nothing, or “more” compared to what most people already have access to? That’s an extremely important distinction. (And it still doesn’t cover which *kinds* of medicine we might need more of vs. which kinds we have enough of, etc. etc.)

    Let me draw an analogy with food. We all know that food is necessary to maintain one’s health, right? We also know that, beyond a certain point, food is not helpful to one’s health. (Eating 2000 Calories a day is much better than eating 500 Calories a day, but eating 8000 Calories at a day isn’t any better than eating 2000 Calories a day)

    Suppose we start with a group of randomly-selected people who all normally pay for their own food, and who all manage to purchase and eat the standard 2000 Calorie diet. Now divide the group in two, and give half of the group 2000 Calories worth of food everyday at no cost (the other half is the control group.) What will happen? Well the group with free food may eat a little more, but they won’t be any healthier as a result. In the meantime they’ll have extra pocket change to spend on movie tickets or whatever, since they won’t need to budget as much (if anything) on food.

    At the end of the study, both groups will have similar levels of health. Someone will then claim that “This experiment seems clear evidence that food is a huge scandal.” and furthermore that “The food research literature must suffer from severe biases, such as fraud, funding bias, treatment selection bias, publication selection bias, leaky placebo effects, misapplied statistics, and so on.”

    An inaccurate conclusion, wouldn’t you say? Or, if the speaker did reach the proper conclusion, then surely his wording is misleading. It *looks* like he’s saying that there is no correlation at all between food and health, which is simply untrue.

    If your original large group had been malnourished, then the giving of free food to half of them would have caused a great increase in health compared to the half that remained malnourished. It is only because the original large group already had sufficient food that the giving of free food had no net effect.

    Similarly, I posit that the average American (at the time of the study) already had access to all the medicine that he could really make use of. Giving him additional free medicine thus had no net effect on his health. But if you had started with a large group of people with no medicinal access to speak of, then the group that received free medicine would have experienced a marked increase in health.

    One vital note: The RAND experiment did NOT study people without any health insurance at all. Applying the food metaphor, this means that malnourished people were EXCLUDED from the study. In other words, the experiment proved that, among people who have basic access to healthcare, giving them free healthcare doesn’t improve their health. It did NOT study anyone who had no access to healthcare to begin with and later received such access via governmental assistance.

    The RAND study is helpful in deciding whether to give out free health care, and it tells us something about the effect of giving the average person *more* medicine than he already has access to. But it tells us next to nothing about the general effect of medicine in all cases. It doesn’t invalidate the idea that sick people benefit from the application of medicine.

    Do you see what I’m saying?

    “what else but education and media biases can explain why this experiment, very expensive, well published, and the most important medical study ever, remains mostly unknown to medical students, professionals and the public? ”

    I have an explanation: The study doesn’t prove what you (apparently) think it proves, and thus it is not nearly as important as you claim.

  • Pingback: Overcoming Bias : Plastination Is Near

  • Mobile Anime

    I love this blog because it is user friendly with appreciative information.Visit us