A Valid Concern

In the October Journal of Health Politics, Policy and Law, John Nyman questions the RAND experiment:

Of the 1,294 adult participants who were randomly assigned to the free plan, 5 participants (0.4 percent) left the experiment voluntarily during the observation period, while of the 2,664 who were assigned to any of the cost-sharing plans, 179 participants (6.7 percent) voluntarily left the experiment. … The explanation that makes the most sense is that the dropouts were participants who had just been diagnosed with an illness that would require a costly hospital procedure. … If they dropped out, their coverage would automatically revert to their original insurance policies, which were likely to cover major medical expenses (such as hospitalizations) with no copayments … As a result of dropping out, these participants’ inpatient stays (and associated health care spending) did not register in the experiment, and it appeared as if participants in the cost-sharing group had a lower rate of inpatient use.

This is a valid concern, which the original work tried to address, but perhaps not sufficiently.  While this is the best of the health-medicine studies, it was far from the only one.  But this seems all the more reason to redo the experiment today.  Hat tip to John McDonough, Alex Tabarrok, and Michael Cannon.

Added 10/24: The RAND Experiment folks have responded with vigor.  HT to Alex Tabarrok.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://entitledtoanopinion.wordpress.com/ TGGP

    Your link to John McDonough is broken. Get rid of the “
    ” in the url.

    What are the other studies that give a null result?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    TGGP, my cut medicine in half essay listed many studies. (And the link has been edited.)

  • Jor

    That’s a pretty big problem. Although at the time it was published it was ok, TODAY, I don’t think a study with that problem would get published in NEJM without an intent to treat analysis. It also doesn’t seem to rare that intent-to-treat and as-treated analysis often have reverse outcomes.

  • http://profile.typekey.com/nickbostrom/ Nick Bostrom

    Thanks for posting this, Robin. Let us try to take delight in discovering and disseminating evidence that weakens our arguments.

  • http://www.cato.org/people/cannon.html Michael F. Cannon

    Your link to Cannon is broken. 🙂

  • http://entitledtoanopinion.wordpress.com/ TGGP

    Via Marginal Revolution again, the RAND folks have a reply.