Too Much Debate

Imagine someone argued:

Stock car races are a huge waste of resources.  To find out which car models are faster, we can just have experts keep track of the speed of the cars they drive past, and write up their observations.  Then we’ll have them debate each other.  Sure some biases might slip in, but we shouldn’t pretend we can escape bias; stock car races can have biases too.  For example, there might be a pebble on one side of the track that isn’t on the other side, or the sun might get in one driver’s eye for a moment but not in another’s.

Or imagine:

Big formal elections are a huge waste of resources.  To find out which candidate is more popular, we can just have experts survey different groups at different times, and write up their observations.  Then we’ll have them debate each other.  Sure some biases might slip in, but we shouldn’t pretend we can escape bias; elections can have biases too.  After all, rain on election day, or certain news the day before, might discourage some kinds of voters but not others.

To me, Austin Frakt on medical experiments talks similarly.  Him:

Randomized experiments differ only in degree from nonexperimental evaluations of causal effects … The half-billion dollars or so that some advocate spending on another RAND HIE would arguably be better spent funding [~1000] well-conceived observational or natural experiment-based studies.

Me:

No doubt a thousand “well-conceived” observational studies, neutrally executed and interpreted, could in principle give more total info than one big experiment.  But … [this] would give many thousands of opportunities for such biases to skew their results. … The main hope for [a clear decisive answer comes] from just a few big experiments focused clear health outcomes agreed on ahead of time.

Him:

The potential for bias in general does not necessarily mean that this randomized study in particular should be preferred. … Contamination of experimental arms, attrition, … [mistaken] statistical corrections … selective reporting of results. … limitations in their generalizability. Even the original RAND HIE has a few imperfections. … Ten years is a very long time in health care. By the time a second RAND HIE study is complete … the new results will be stale. …

No doubt the results of 1,000 such studies would not be unanimous, … But … there would be a general consensus on some questions … To be sure there would be room for debate … just as there is in the case of the RAND HIE. …  And that is really my main point. No study, or collection of studies, can ever be the definitive word on a subject. There will always be debate. … The best we can hope is that they inform, not that they settle, debate.

More likely than not, medicine is on average is near useless or harmful on the margin; that is my best reading of the evidence.  When I try to persuade folks, I start with our single best data point, the old RAND experiment, but people complain it was too small, short, and long ago (and it let folks leave too easily).  When I point to other studies they suggest I must be biased about which studies I cite.  Other experts are cagey about how much they agree with me.  Most agree the effect seems small, but many insist that even tiny effects are oh so important; and we don’t dare cut back, as that would be giving up on maybe improving things.

Lots of small diverse flawed studies, plus lots of diverse researchers each choosing their own criteria for rating and debating them, seems to me a recipe for everyone believing whatever they want.  Without one (or a few) very strong very clear studies, there is simply no way to convince most folks that marginal medicine is on average useless, and we should cut way back.  And with folks like Austin Frakt eager to make sure there is plenty of room for debate, we may never get such clarity.

Added: Austin responds.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • Pingback: In Search of Convincing Evidence? | The Incidental Economist

  • spriteless

    It is hard for you to convince me that when my dad’s diverticuli broke, scooping out all the rot and sewing up another means of defecation while his body healed had only a marginal effect on his health. He was kind of, you know, dying. Medicine is at its most memorable at moments like that, not in the decades of taking medicine that reduces the chance of a heart attack by 10% of .5%, rather than by 10% of 100%.

    • Stuart Armstrong

      I think Robin’s point is about the marginal value of medecine. Getting rid of all of medecine will not make us healthier, obviously. But the question is, does a little more spending on medecine that what we have now increase our health at all? And does a little less decrease it?

      It’s pretty certain that diverticuli operations would not be the first things to be cut when the money goes down.

      • http://lesswrong.com/ CannibalSmith

        What would be cut, for example?

      • Doug S.

        Well, what wouldn’t you get if you had to pay out-of-pocket for it?

    • http://www.cawtech.freeserve.co.uk Alan Crowe

      That reminded me of a counter-anecdote. One commenter’s father had had a routine colonoscopy, but the man was already old and frail. If the colonoscopy had found colon cancer there would not have been anything useful to do about the cancer. Unfortunately there was a rare accident. The endoscope ruptured the man’s colon and being old and frail he died of it.

      That raised a strange question: does Spriteless’ anecdote argue against Robin’s claim, or in favour of it?

      The problem is that the naive reading of “medicine is on average is near useless or harmful on the margin;” is that the margin is the homeopathy zone. The treatments do nothing and only money is at stake. That is not actually what Robin is saying. Useless on average covers both useless in every case and a balance of kill and cure.

      Spriteless’ anecdote offers us an image of the typical medical procedure as a reasonably priced surgery that is directly life saving. If that were the end of the story it would be easy to show that medical expenditures produced a large impact on mortality. It’s not. So what is the rest of the story?

      Accepting Robin’s judgement that it is hard to demonstrate the marginal benefit of medicine in the aggregate figures, the implications of Spriteless’ anecdote are perhaps inverted. If I’m persuaded that reasonably priced surgeries that are directly life saving are common place, then my heart sinks, for the implication is that the average is coming from a balancing factor of common place medical disaster. And Robin’s point becomes about lives as well as money; there are many lives to be saved by cutting back on killer surgery.

  • http://wiredcola.com Ryan Cousineau

    Spriteless: just so. On the other hand, it’s rather hard to convince me that when my father and my mother-in-law both contracted opportunistic infections after semi-successful carpal tunnel and hernia surgeries (respectively), that medicine was especially helpful to either of them.

    Medicine’s successes are memorable, its worst failures are not. Indeed, medicine buries its gravest mistakes.

    The worst of it is that we’re not really sure if a lot of high-end, high-intervention surgeries actually improve medical outcomes more than doing nothing.

    We’ve just come through a long period of prostate testing and surgery that, according to the best opinions today, mostly had no effect on the lifespan of the affected men (but it did frequently render them impotent).

  • http://entitledtoanopinion.wordpress.com TGGP

    I’m just a layman, so I’d like to hear why in principle 1,000 observational studies should be better than a single randomized trial with equivalent funding (let’s assume these are being done by a single person for the benefit of his own pursuit of truth). Is it because of diminishing returns in study size? You should be able to add more variables to observe or treatments to be given, even if this means not having as many experimental subjects, while still pursuing the one-big-study.

  • Lo Statuz

    Would a RAND rerun really cost $500M and take 10 years?

    As a practical matter, who would volunteer to enroll in the study? Today, I figure lots of people would sign up for a 50% chance at free medicine. But consider those who land in the arm where they have to pay almost everything out of pocket. In 2014, they can drop out and buy subsidized insurance instead. Rationally, anyone who expected to save money that way would do so. Those who remain would be healthier than average, skewing the results
    .

  • y81

    If we get enough anecdotes in the comments, it will equal data, right?

    Anyway, when my mom had a glioblastoma, she had two surgeries, radiation treatment and chemotherapy. She died within a year of the initial diagnosis. Interestingly, her GP told her, after the first surgery (which was indicated to confirm the type of brain tumor), not to undergo further treatment, as it would be painful and offer no meaningful benefit. However, the oncologists were much more bullish on the effects of treatment. Had we known then what we know now, we might have foregone treatment, but we deferred to the experts.

    Clearly, in this case, considerable extra expense had no marginal benefit.

    • http://williambswift.blogspot.com/ billswift

      Isn’t that all statistics really is is a lot of boiled down anecdotes? Sort of like the reverse of Stalin’s aphorism. All statistics are is a million murders (or anecdotes).

  • Grant

    Robin,

    Couldn’t 1,000 observational studies give us good ideas as to which treatments are harmful or wasteful at the margin? And which are beneficial (I’m sure some are)? We’d still have a picture of how useful medicine is at the margin, but we’d also get an idea of which treatments were helpful and which ones weren’t.

  • Contemplationist

    Its interesting that a hyper-rationalist like Robin comes to similar conclusions as hyper-skeptic Nassim Taleb regarding the value of Medicine.

  • Patricia

    The counter anecdote is a great example of precisely what we’re trained NOT to do. Investigations are unnecessary, wasteful, harmful and to be avoided like the plague if they won’t change our management/outcome.

    Certainly we would get more bang for our buck from engineering and public health than from medicine on the margin, but the effects of cure are near and the effects of prevention are far, so who’s going to get elected on a platform of diverting those funds?

  • Pingback: TheMoneyIllusion » Tyler Cowen’s curious curiosity