Funding Bias

From a Post article while I was traveling:

Wal-Mart and Toys R Us … will stop selling plastic baby bottles, food containers and other products that contain [BPA]. … One of the eyebrow-raising statistics about the BPA studies is the stark divergence in results, depending on who funded them.  More than 90 percent of the 100-plus government-funded studies performed by independent scientists found health effects from low doses of BPA, while none of the fewer than two dozen chemical-industry-funded studies did.  This striking difference in studies isn’t unique to BPA. When a scientist is hired by a firm with a financial interest in the outcome, the likelihood that the result of that study will be favorable to that firm is dramatically increased. …

Within the scientific community, there is little debate about the existence of the funding effect, but the mechanism through which it plays out has been a surprise.  At first, it was widely assumed that the misleading results … came from shoddy studies done by researchers who manipulated methods and data. … But close examination of the manufacturers’ studies showed that their quality was usually at least as good as, and often better than, studies that were not funded by drug companies. …

"Tricks of the trade" … include testing your drug against a treatment that either does not work or does not work very well; testing your drug against too low or too high a dose of the comparison drug because this will make your drug appear more effective or less toxic; publishing the results of a single trial many times in different forms to make it appear that multiple studies reached the same conclusions; and publishing only those studies, or even parts of studies, that are favorable to your drug, and burying the rest. … Decisions about which articles to include in a meta-analysis and how heavily to weight them have an enormous impact on the conclusions. …

The answer is de-linking sponsorship and research. One model is the Health Effects Institute … [with] an independent governing structure. … HEI conducts studies paid for by corporations, but its researchers are sufficiently insulated from the sponsors that their results are credible.

Alas this solution is mostly wishful thinking. "Government funded" does not mean "unbiased" – it just means a different mix of biases.  Instead we need evaluation institutions, such as prediction markets, which can better resist funding biases of all sorts. 

GD Star Rating
Tagged as:
Trackback URL:
  • A prediction market requires a gold standard to determine the outcome and pay out its bets. If you can’t trust government or manufacturer studies, what do the contracts trust?

  • I have in mind things like forecasts of future mortality rates conditional on whether BPA is banned.

  • Eliezer, you lost me on that one. I think you’re conflating/mixing up a bunch of things rather badly. In an attempt to reply to your comments the “we” in Robin’s statement “we need evaluation institutions” are those of us trying to get the most accurate models of reality. What the contractors trust and why we should trust the information we get from the prediction markets would be separate questions. For the first question, I think the answer is the reputation for the prediction market for paying out predictions fairly according to the rules agreed by the participant contactors. For the second question, we look at what makes a successful prediction market empirically. I think the consensus of experts matters more here than governments or industries as the ultimate evaluator, alternatively, it’s the consensus of us observers discussing using these prediction markets, most narrowly it’s Robin (or me).

  • Tome

    Thomas Kuhn did some great research on the history of science, and found out that these so-called paradigms of science often come out as ‘scientific truth’, while they’re not much more than social convention in the name of a given normality. Science takes too much for granted and believes too deeply in it’s ability to ‘see’ and understand’ when they merely assume and believe.

  • sophiesdad

    Dr. Hanson:

    How strange that you should choose the Health Effects Institute as a “model” for unbiased research. Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors:

    American Suzuki Motor Corporation
    BMW of North America, Inc.
    Caterpillar, Inc.
    Cummins Engine Company
    Detroit Diesel Corporation
    Ford Motor Company
    General Motors Corporation
    Hino Motors, Ltd.
    Honda Motor Company, Ltd.
    Hyundai America Technical Center
    International Truck and Engine Corporation
    Isuzu Motors, Ltd.
    Jaguar Cars, Inc.
    John Deere and Company
    KIA Motors America, Inc.
    Mack Trucks, Inc.
    Mazda Motor Corporation
    Mercedes Benz of North America, Inc.
    Mitsubishi Motors Corporation
    Nissan Motor Company, Ltd.
    Range Rover of North America
    Subaru of America, Inc.
    Toyota Motor Corporation
    Volkswagen of America, Inc.
    Volvo North America Corporation

    Other Sponsors:

    American Petroleum Institute
    American Iron and Steel Institute
    ArvinMeritor A&ET S.p.A.
    California Air Resources Board
    Corning, Inc.
    William and Flora Hewlett Foundation
    U.S. Department of Energy, National Energy Technology Laboratory
    U.S. Department of Transportation, Federal Highway Administration

    Does the internal combusiton engine have any recognized influence on air pollution?

  • sophiesdad: How strange that you should choose the Health Effects Institute as a “model” for unbiased research.’ Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors […]

    I have trouble understanding this comment. Not only does the article claim that the funding is set up in such a way that this sponsorship is not a problem, but Robin didn’t endorse the article, and even argued against it:

    Robin Hanson: Alas this solution is mostly wishful thinking. “Government funded” does not mean “unbiased”

    I have a little trouble understanding this, as well. 🙂 The article doesn’t seem to be arguing for government funded research, but rather says that the solution is a funding model that “insulates” researchers from sponsors.

  • Allan, sounds like you’re making a good point, but not addressing the core of Robin’s argument, and I think Robin’s on the side of the angels on this one. Prediction markets can help “insulate” researchers (and us, the audience of the researchers) from the researchers own biases (or more broadly from many types of biases, beyond sponsor-derived bias).

  • Yeah, I wasn’t arguing against prediction markets. They sound like a good idea to me. It was just a nitpick I suppose.

  • sophiesdad

    Mr. Crossman: The article doesn’t seem to be arguing for government funded research, but rather says that the solution is a funding model that “insulates” researchers from sponsors.

    Firstly, my bad in attributing the model choice to Dr. Hanson, rather than to the article, although his classification of it as “wishful thinking” suggests that he agrees with the concept, were it possible.

    Secondly, it is naive to think that paid researchers can be insulated from the influence of those who are the source of their pay. This concept has virtually eliminated the possibility of getting accurate medical information from the “thought leaders” in my field, erectile dysfunction. Those professors have gone from a relatively low fixed-salary arrangement (making them among the lowest paid urologic surgeons) provided solely by their universities, to the current “consulting” arrangements for the pharmaceutical and device industry, making them the highest paid of all urologists, some into the millions of dollars annually, while still maintaining their professorships. I am acquainted with all these “thought leaders”; not one thinks he has been influenced by industry. My own professional certifying board cautions all journal readers and all conference attendees: decide for yourself what is true. This is worse than knowing for certain that all of it is false.

  • michael vassar

    Robin: When you talk about forecasts conditional on other forecasts, as you often do with prediction markets, I think of the discussion where you asserted, correctly, that very few people actually hedge. It seems psychologically unrealistic to me to expect complex hedging to be widespread in prediction markets when even simple hedging is uncommon in existing markets (though many things are economically justified in the name of hedging).

  • Jeff

    This looks to me like a good example of “perfect is the enemy of good.” Any research will have some bias, but we should draw a distinction between imperfect results and those that are nearly worthless. Government-sponsored research has had its ups and downs, but there isn’t a single overwhelming source of bias, and in practice it’s worked reasonably well.

    Industry-sponsored research is a whole different story. When a drug company pays for research, they always want to find out that their new drug is safe and effective. No matter how honorable and careful their scientists might be, the fact is that 100% of the sponsorship is hoping for a particular result; for smaller companies, if the drug isn’t found safe and effective, it means there won’t be any money to finish the research. It’s going to be damned hard to keep that from influencing the reported results.

    (Note: I’m not arguing against prediction markets or the HEI model; just pointing out that lumping together government- and industry-sponsored research as equally biased is just ludicrous.)

  • Jay

    99.9% of government studies support more government intervention.

    I’d like to hear somone name one government funded study in which the result was something other than a “politician” pontificating to us on how said study proves that the government needs to reduce our liberties so that the government can protect us from ourselves.

  • Thanatos Savehn

    Having been involved with this and similar issues I offer the following:

    1) Government funded academic research is highly prone to produce positives. My favorite (or rather least favorite) example is the funding of so-called Gulf War Syndrome research. Those institutions which didn’t find a link to heretofore unknown exotic gases, un-isolatable viruses or common vaccines got paid for their work and then didn’t get any more. Texas Southwestern on the other hand found all sorts of causal links. As a result its researchers have gotten tens of millions of dollars in research money and they continue to churn out junk science that can’t be reproduced elsewhere. Uncle Sam is the number one purchaser, and thus producer of, science for sale – and much, if not most, of it is junk. The worst of it is deadly – see e.g. saturated fat research from the 70s and 80s. In my experience, the more trendy the research the more likely it is to be crap.

    2) Fred vom Saal is a nut.

    3) The premiss is “BPA was invented by someone looking for a hormone, therefore BPA must a hormone, all hormones are endocrine disruptors, therefore BPA is an endocrine disruptor”. It may even be true – if you are a reading this and you are a certain strain of mouse (but not a rat, guinea pig, dog, monkey or human).

    4) The most bizarre aspect of the whole silliness is the belief that the dose response relationship that is the basis of all pharmacology and toxicology is wrong so that the lower the dose, the greater the response. If you like homeopathy you’ll love this scare. PS One funny aspect of their “science” is that if you ban BPA, levels will drop; since lower levels are more toxic (indeed infinitely diluted BPA would produce 100% mortality if you extrapolate from their results) things would only get worse.

    5) The Europeans, who are no chemophiles, refused to ban BPA a few days ago.

  • Jeff, “insulating” researchers from sponsors may reduce some biases, but it will increase others. Do you know of data on the overall average effect?

    Michael, I’m not hoping for complex hedging instincts, but for direct subsidies of the markets of interest.

    Thanatos, thanks for the review.

  • is there any use for prediction markets that use play money, and the players get rewards in terms of reputation? for example, a scholar makes accurate predictions–he earns more play money than the average person. he can use his stats as a resume item used to help him secure jobs where his talents could make him (and his employer) a lot of money. could creating and popularizing play-money predictions markets be a way around laws against prediction markets, and a means of creating institutions that allow us to make better decisions?

    maybe establishing more such play-money systems would at least get people used to the idea and pave the way for the real thing.

  • I think evaluation institutions have a role, but they have to be allowed to develop. Allowing them to develop means that the government should not pre-empt their role. It is likely that, in the absence of a government role, and the presence of a real need, some such institutions will be founded.

    It is also likely that when such institutions are available, some will figure out a way to work around them, and some such institutions will be corruptly in league with those they evaluate. By such errors, the marketplace will learn what is important. By such errors, managers of future evaluation-institutions will learn what makes for a good business model.

  • Pingback: The Inconvenient Skeptic » The Big Flaw in Research.()