From a Post article while I was traveling: Wal-Mart and Toys R Us … will stop selling plastic baby bottles, food containers and other products that contain [BPA]. … One of the eyebrow-raising statistics about the BPA studies is the stark divergence in results, depending on who funded them. More than 90 percent of the 100-plus government-funded studies performed by independent scientists found health effects from low doses of BPA, while none of the fewer than two dozen chemical-industry-funded studies did. This striking difference in studies isn’t unique to BPA. When a scientist is hired by a firm with a financial interest in the outcome, the likelihood that the result of that study will be favorable to that firm is dramatically increased. …
I think evaluation institutions have a role, but they have to be allowed to develop. Allowing them to develop means that the government should not pre-empt their role. It is likely that, in the absence of a government role, and the presence of a real need, some such institutions will be founded.
It is also likely that when such institutions are available, some will figure out a way to work around them, and some such institutions will be corruptly in league with those they evaluate. By such errors, the marketplace will learn what is important. By such errors, managers of future evaluation-institutions will learn what makes for a good business model.
is there any use for prediction markets that use play money, and the players get rewards in terms of reputation? for example, a scholar makes accurate predictions--he earns more play money than the average person. he can use his stats as a resume item used to help him secure jobs where his talents could make him (and his employer) a lot of money. could creating and popularizing play-money predictions markets be a way around laws against prediction markets, and a means of creating institutions that allow us to make better decisions?
maybe establishing more such play-money systems would at least get people used to the idea and pave the way for the real thing.
Having been involved with this and similar issues I offer the following:
1) Government funded academic research is highly prone to produce positives. My favorite (or rather least favorite) example is the funding of so-called Gulf War Syndrome research. Those institutions which didn't find a link to heretofore unknown exotic gases, un-isolatable viruses or common vaccines got paid for their work and then didn't get any more. Texas Southwestern on the other hand found all sorts of causal links. As a result its researchers have gotten tens of millions of dollars in research money and they continue to churn out junk science that can't be reproduced elsewhere. Uncle Sam is the number one purchaser, and thus producer of, science for sale - and much, if not most, of it is junk. The worst of it is deadly - see e.g. saturated fat research from the 70s and 80s. In my experience, the more trendy the research the more likely it is to be crap.
2) Fred vom Saal is a nut.
3) The premiss is "BPA was invented by someone looking for a hormone, therefore BPA must a hormone, all hormones are endocrine disruptors, therefore BPA is an endocrine disruptor". It may even be true - if you are a reading this and you are a certain strain of mouse (but not a rat, guinea pig, dog, monkey or human).
4) The most bizarre aspect of the whole silliness is the belief that the dose response relationship that is the basis of all pharmacology and toxicology is wrong so that the lower the dose, the greater the response. If you like homeopathy you'll love this scare. PS One funny aspect of their "science" is that if you ban BPA, levels will drop; since lower levels are more toxic (indeed infinitely diluted BPA would produce 100% mortality if you extrapolate from their results) things would only get worse.
5) The Europeans, who are no chemophiles, refused to ban BPA a few days ago.
99.9% of government studies support more government intervention.
I'd like to hear somone name one government funded study in which the result was something other than a "politician" pontificating to us on how said study proves that the government needs to reduce our liberties so that the government can protect us from ourselves.
This looks to me like a good example of "perfect is the enemy of good." Any research will have some bias, but we should draw a distinction between imperfect results and those that are nearly worthless. Government-sponsored research has had its ups and downs, but there isn't a single overwhelming source of bias, and in practice it's worked reasonably well.
Industry-sponsored research is a whole different story. When a drug company pays for research, they always want to find out that their new drug is safe and effective. No matter how honorable and careful their scientists might be, the fact is that 100% of the sponsorship is hoping for a particular result; for smaller companies, if the drug isn't found safe and effective, it means there won't be any money to finish the research. It's going to be damned hard to keep that from influencing the reported results.
(Note: I'm not arguing against prediction markets or the HEI model; just pointing out that lumping together government- and industry-sponsored research as equally biased is just ludicrous.)
Robin: When you talk about forecasts conditional on other forecasts, as you often do with prediction markets, I think of the discussion where you asserted, correctly, that very few people actually hedge. It seems psychologically unrealistic to me to expect complex hedging to be widespread in prediction markets when even simple hedging is uncommon in existing markets (though many things are economically justified in the name of hedging).
Mr. Crossman: The article doesn't seem to be arguing for government funded research, but rather says that the solution is a funding model that "insulates" researchers from sponsors.
Firstly, my bad in attributing the model choice to Dr. Hanson, rather than to the article, although his classification of it as "wishful thinking" suggests that he agrees with the concept, were it possible.
Secondly, it is naive to think that paid researchers can be insulated from the influence of those who are the source of their pay. This concept has virtually eliminated the possibility of getting accurate medical information from the "thought leaders" in my field, erectile dysfunction. Those professors have gone from a relatively low fixed-salary arrangement (making them among the lowest paid urologic surgeons) provided solely by their universities, to the current "consulting" arrangements for the pharmaceutical and device industry, making them the highest paid of all urologists, some into the millions of dollars annually, while still maintaining their professorships. I am acquainted with all these "thought leaders"; not one thinks he has been influenced by industry. My own professional certifying board cautions all journal readers and all conference attendees: decide for yourself what is true. This is worse than knowing for certain that all of it is false.
Allan, sounds like you're making a good point, but not addressing the core of Robin's argument, and I think Robin's on the side of the angels on this one. Prediction markets can help "insulate" researchers (and us, the audience of the researchers) from the researchers own biases (or more broadly from many types of biases, beyond sponsor-derived bias).
sophiesdad: How strange that you should choose the Health Effects Institute as a "model" for unbiased research.' Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors [...]
I have trouble understanding this comment. Not only does the article claim that the funding is set up in such a way that this sponsorship is not a problem, but Robin didn't endorse the article, and even argued against it:
Robin Hanson: Alas this solution is mostly wishful thinking. "Government funded" does not mean "unbiased"
I have a little trouble understanding this, as well. :) The article doesn't seem to be arguing for government funded research, but rather says that the solution is a funding model that "insulates" researchers from sponsors.
How strange that you should choose the Health Effects Institute as a "model" for unbiased research. Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors:
American Suzuki Motor CorporationBMW of North America, Inc.Caterpillar, Inc.Cummins Engine CompanyDaimlerChryslerDetroit Diesel CorporationFord Motor CompanyGeneral Motors CorporationHino Motors, Ltd.Honda Motor Company, Ltd.Hyundai America Technical CenterInternational Truck and Engine CorporationIsuzu Motors, Ltd.Jaguar Cars, Inc.John Deere and CompanyKIA Motors America, Inc.Mack Trucks, Inc.Mazda Motor CorporationMercedes Benz of North America, Inc.Mitsubishi Motors CorporationNissan Motor Company, Ltd.Range Rover of North AmericaSubaru of America, Inc.Toyota Motor CorporationVolkswagen of America, Inc.Volvo North America Corporation
Other Sponsors:
American Petroleum InstituteAmerican Iron and Steel InstituteArvinMeritor A&ET S.p.A.California Air Resources BoardCorning, Inc.ExxonMobilWilliam and Flora Hewlett FoundationU.S. Department of Energy, National Energy Technology LaboratoryU.S. Department of Transportation, Federal Highway Administration
Does the internal combusiton engine have any recognized influence on air pollution?
Thomas Kuhn did some great research on the history of science, and found out that these so-called paradigms of science often come out as 'scientific truth', while they're not much more than social convention in the name of a given normality. Science takes too much for granted and believes too deeply in it's ability to 'see' and understand' when they merely assume and believe.
Eliezer, you lost me on that one. I think you're conflating/mixing up a bunch of things rather badly. In an attempt to reply to your comments the "we" in Robin's statement "we need evaluation institutions" are those of us trying to get the most accurate models of reality. What the contractors trust and why we should trust the information we get from the prediction markets would be separate questions. For the first question, I think the answer is the reputation for the prediction market for paying out predictions fairly according to the rules agreed by the participant contactors. For the second question, we look at what makes a successful prediction market empirically. I think the consensus of experts matters more here than governments or industries as the ultimate evaluator, alternatively, it's the consensus of us observers discussing using these prediction markets, most narrowly it's Robin (or me).
A prediction market requires a gold standard to determine the outcome and pay out its bets. If you can't trust government or manufacturer studies, what do the contracts trust?
I think evaluation institutions have a role, but they have to be allowed to develop. Allowing them to develop means that the government should not pre-empt their role. It is likely that, in the absence of a government role, and the presence of a real need, some such institutions will be founded.
It is also likely that when such institutions are available, some will figure out a way to work around them, and some such institutions will be corruptly in league with those they evaluate. By such errors, the marketplace will learn what is important. By such errors, managers of future evaluation-institutions will learn what makes for a good business model.
is there any use for prediction markets that use play money, and the players get rewards in terms of reputation? for example, a scholar makes accurate predictions--he earns more play money than the average person. he can use his stats as a resume item used to help him secure jobs where his talents could make him (and his employer) a lot of money. could creating and popularizing play-money predictions markets be a way around laws against prediction markets, and a means of creating institutions that allow us to make better decisions?
maybe establishing more such play-money systems would at least get people used to the idea and pave the way for the real thing.
Jeff, "insulating" researchers from sponsors may reduce some biases, but it will increase others. Do you know of data on the overall average effect?
Michael, I'm not hoping for complex hedging instincts, but for direct subsidies of the markets of interest.
Thanatos, thanks for the review.
Having been involved with this and similar issues I offer the following:
1) Government funded academic research is highly prone to produce positives. My favorite (or rather least favorite) example is the funding of so-called Gulf War Syndrome research. Those institutions which didn't find a link to heretofore unknown exotic gases, un-isolatable viruses or common vaccines got paid for their work and then didn't get any more. Texas Southwestern on the other hand found all sorts of causal links. As a result its researchers have gotten tens of millions of dollars in research money and they continue to churn out junk science that can't be reproduced elsewhere. Uncle Sam is the number one purchaser, and thus producer of, science for sale - and much, if not most, of it is junk. The worst of it is deadly - see e.g. saturated fat research from the 70s and 80s. In my experience, the more trendy the research the more likely it is to be crap.
2) Fred vom Saal is a nut.
3) The premiss is "BPA was invented by someone looking for a hormone, therefore BPA must a hormone, all hormones are endocrine disruptors, therefore BPA is an endocrine disruptor". It may even be true - if you are a reading this and you are a certain strain of mouse (but not a rat, guinea pig, dog, monkey or human).
4) The most bizarre aspect of the whole silliness is the belief that the dose response relationship that is the basis of all pharmacology and toxicology is wrong so that the lower the dose, the greater the response. If you like homeopathy you'll love this scare. PS One funny aspect of their "science" is that if you ban BPA, levels will drop; since lower levels are more toxic (indeed infinitely diluted BPA would produce 100% mortality if you extrapolate from their results) things would only get worse.
5) The Europeans, who are no chemophiles, refused to ban BPA a few days ago.
99.9% of government studies support more government intervention.
I'd like to hear somone name one government funded study in which the result was something other than a "politician" pontificating to us on how said study proves that the government needs to reduce our liberties so that the government can protect us from ourselves.
This looks to me like a good example of "perfect is the enemy of good." Any research will have some bias, but we should draw a distinction between imperfect results and those that are nearly worthless. Government-sponsored research has had its ups and downs, but there isn't a single overwhelming source of bias, and in practice it's worked reasonably well.
Industry-sponsored research is a whole different story. When a drug company pays for research, they always want to find out that their new drug is safe and effective. No matter how honorable and careful their scientists might be, the fact is that 100% of the sponsorship is hoping for a particular result; for smaller companies, if the drug isn't found safe and effective, it means there won't be any money to finish the research. It's going to be damned hard to keep that from influencing the reported results.
(Note: I'm not arguing against prediction markets or the HEI model; just pointing out that lumping together government- and industry-sponsored research as equally biased is just ludicrous.)
Robin: When you talk about forecasts conditional on other forecasts, as you often do with prediction markets, I think of the discussion where you asserted, correctly, that very few people actually hedge. It seems psychologically unrealistic to me to expect complex hedging to be widespread in prediction markets when even simple hedging is uncommon in existing markets (though many things are economically justified in the name of hedging).
Mr. Crossman: The article doesn't seem to be arguing for government funded research, but rather says that the solution is a funding model that "insulates" researchers from sponsors.
Firstly, my bad in attributing the model choice to Dr. Hanson, rather than to the article, although his classification of it as "wishful thinking" suggests that he agrees with the concept, were it possible.
Secondly, it is naive to think that paid researchers can be insulated from the influence of those who are the source of their pay. This concept has virtually eliminated the possibility of getting accurate medical information from the "thought leaders" in my field, erectile dysfunction. Those professors have gone from a relatively low fixed-salary arrangement (making them among the lowest paid urologic surgeons) provided solely by their universities, to the current "consulting" arrangements for the pharmaceutical and device industry, making them the highest paid of all urologists, some into the millions of dollars annually, while still maintaining their professorships. I am acquainted with all these "thought leaders"; not one thinks he has been influenced by industry. My own professional certifying board cautions all journal readers and all conference attendees: decide for yourself what is true. This is worse than knowing for certain that all of it is false.
Yeah, I wasn't arguing against prediction markets. They sound like a good idea to me. It was just a nitpick I suppose.
Allan, sounds like you're making a good point, but not addressing the core of Robin's argument, and I think Robin's on the side of the angels on this one. Prediction markets can help "insulate" researchers (and us, the audience of the researchers) from the researchers own biases (or more broadly from many types of biases, beyond sponsor-derived bias).
sophiesdad: How strange that you should choose the Health Effects Institute as a "model" for unbiased research.' Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors [...]
I have trouble understanding this comment. Not only does the article claim that the funding is set up in such a way that this sponsorship is not a problem, but Robin didn't endorse the article, and even argued against it:
Robin Hanson: Alas this solution is mostly wishful thinking. "Government funded" does not mean "unbiased"
I have a little trouble understanding this, as well. :) The article doesn't seem to be arguing for government funded research, but rather says that the solution is a funding model that "insulates" researchers from sponsors.
Dr. Hanson:
How strange that you should choose the Health Effects Institute as a "model" for unbiased research. Their topic of research is air pollution. Half their funding comes from the EPA, and the other half comes from these sponsors:
American Suzuki Motor CorporationBMW of North America, Inc.Caterpillar, Inc.Cummins Engine CompanyDaimlerChryslerDetroit Diesel CorporationFord Motor CompanyGeneral Motors CorporationHino Motors, Ltd.Honda Motor Company, Ltd.Hyundai America Technical CenterInternational Truck and Engine CorporationIsuzu Motors, Ltd.Jaguar Cars, Inc.John Deere and CompanyKIA Motors America, Inc.Mack Trucks, Inc.Mazda Motor CorporationMercedes Benz of North America, Inc.Mitsubishi Motors CorporationNissan Motor Company, Ltd.Range Rover of North AmericaSubaru of America, Inc.Toyota Motor CorporationVolkswagen of America, Inc.Volvo North America Corporation
Other Sponsors:
American Petroleum InstituteAmerican Iron and Steel InstituteArvinMeritor A&ET S.p.A.California Air Resources BoardCorning, Inc.ExxonMobilWilliam and Flora Hewlett FoundationU.S. Department of Energy, National Energy Technology LaboratoryU.S. Department of Transportation, Federal Highway Administration
Does the internal combusiton engine have any recognized influence on air pollution?
Thomas Kuhn did some great research on the history of science, and found out that these so-called paradigms of science often come out as 'scientific truth', while they're not much more than social convention in the name of a given normality. Science takes too much for granted and believes too deeply in it's ability to 'see' and understand' when they merely assume and believe.
Eliezer, you lost me on that one. I think you're conflating/mixing up a bunch of things rather badly. In an attempt to reply to your comments the "we" in Robin's statement "we need evaluation institutions" are those of us trying to get the most accurate models of reality. What the contractors trust and why we should trust the information we get from the prediction markets would be separate questions. For the first question, I think the answer is the reputation for the prediction market for paying out predictions fairly according to the rules agreed by the participant contactors. For the second question, we look at what makes a successful prediction market empirically. I think the consensus of experts matters more here than governments or industries as the ultimate evaluator, alternatively, it's the consensus of us observers discussing using these prediction markets, most narrowly it's Robin (or me).
I have in mind things like forecasts of future mortality rates conditional on whether BPA is banned.
A prediction market requires a gold standard to determine the outcome and pay out its bets. If you can't trust government or manufacturer studies, what do the contracts trust?