Tag Archives: Prediction Markets

Who Wants Unbiased Journals?

By Robin Hanson · April 27, 2012 11:00 pm · Comments (17)

Five years ago I proposed result-blind peer review, and I revised it later. Brendan Nyham just posted a nice long review of many such proposals, including a recent test at the journal Archives of Internal Medicine:

The … alternate review process was applied to the editorial review that occurred prior to outside peer review. … Of the 46 articles examined, 28 were positive, and 18 were negative. … Ultimately, 36 of the 46 articles (>77%) were rejected. … Editors were consistent in their assessment of a manuscript in both steps of the review process in over 77% of cases. … Over 7% of positive articles benefited from editors changing their minds between steps 1 and 2 of the alternate review process, deciding to push forward with peer review after reading the results. By contrast, … this never occurred with the negative studies. Indeed, 1 negative study, which was originally queued for peer review after an editor’s examination of the introduction and “Methods” section, was removed from such consideration after the results were made available. (more)

So even with two stage review, journal editors are tempted to publish papers with weak methods but positive results. And why not – unless important customers insisted, why would a journal handicap itself by committing itself to not publish such papers, which bring more fame and prestige to the journal.

Journal customers include universities who tenure professors who publish in prestigious journals, and grant givers who prefer grantees who publish similarly. But why should these customers handicap themselves – they also win by affiliating with those who publish papers with weak methods but positive results.

I’ve suggested that academia functions primarily to credential people as impressive and interesting in certain ways, so outsiders, like students and patron, can gain prestige by affiliating with them. If so, and if those who publish weak-method positive-results are in fact more impressive and interesting than those who publish stronger-method negative-results, there is little prospect to get rid of this publication bias.

What is possible is to augment publications with betting market prices estimating the chance each result will be upheld by future research. This would let readers get unbiased estimates on the reliability of research results. Alas, it seems there is no customer willing to pay extra to get such reliability estimates. Most everyone involved in the process mainly cares about signals of impressiveness; few care much about which research results are actually true.

GD Star Rating
loading...
Tagged as: ,

Ban Election Arguments?

By Robin Hanson · April 8, 2012 1:45 pm · Comments (28)

While Intrade has betting markets on the US presidential election, they are unregulated and of questionable US legality. Nadex went through the expensive legal hoops to apply for permission to run a regulated market. Last week:

The CFTC determined that the contracts involve gaming and are contrary to the public interest. (more)

Why?

It could unduly influence election results. … the contracts could run afoul of the election process if traders had financial incentives to vote for particular candidates. (more)

They still allow election betting at the Iowa Electronic Markets, where stakes are limited to $500. They still let people work for campaigns and administrations, which gives them financial incentives to vote for certain candidates. And they let candidates take positions favoring some industries, occupations, and locations, over others, which gives people financial incentives to vote for and against candidates.

We also let people tell other people which candidates they favor, which gives people non-financial incentives to vote for those candidates later. And since every bet for a candidate is matched with a bet against that candidate, whenever a betting market gives anyone a financial incentive to vote for a candidate, it at the same time gives someone else a financial incentive to vote against that candidate. Why are all the rest of these “due” influences, while bets are “undue” influences?

Paula Dwyer argues:

Naked credit default swaps on Greek sovereign debt (buying a CDS without owning the underlying debt) are no more than a bet on a Greek default. Will the CFTC be barring them, too? (more)

Law and Economics professors Eric Posner and Glen Weyl support the CFTC:

Financial instruments that serve primarily as a means of speculation rather than hedging should be banned … Suppose that two individuals, neither of whom uses or produces oil, harbor different opinions about the future price of oil and decide to wager on it. Both parties willingly participate, because they think they’re each getting the best of their confused counterparty. Clearly, both of them cannot gain from this transaction, and the wager itself creates rather than reduces risk. While each party thinks it is getting the better of the other, both agree that on average both of them will be worse off because on average they will win and lose on the same number of bets, and both of their incomes will be less smooth and predictable on account of their wagering. As a consequence, this sort of speculation is socially harmful. …

In controlled and appropriate contexts, [gambling] can be a source of entertainment for people who are aware of and willing to accept the potential losses. But participants in financial markets are usually seeking financial security rather than entertainment, and they typically have little sense of the risks they are taking on. … A second potential benefit of allowing trading in derivatives is the information that they provide to market participants. The knowledge of the likely outcome of the presidential election provided by the wisdom of the crowds is useful for planning by businesses, individuals, and governments. But that information is only valuable to the extent that it enables real economic decisions to be made more effectively.

Consider: why should we let people argue on elections? Similar to the above, one could say:

People mainly argue in the hope of winning arguments, thinking that they are taking advantage of confused opponents. While each side hopes that further events and discussions will reveal them to have been more in the right, both sides understand that this can’t happen for both of them. Yes, people might argue just to have fun, but election pundits seem serious – wanting more to prove the other side wrong. And most people who argue politics seem to have little understanding of what they are talking about. Yes, arguments can produce useful info for others, but the value of the info produced in election arguments is small compared to the time lost arguing. Thus we should ban arguments on elections.

Election arguers and bettors both seem motivated by a similar mix of enjoying the process and hoping to win. But the info produced by bettors is far more persuasive, reliable, and useful – you have far better reasons to believe betting market odds than whatever the apparent winner of a political argument has claimed.

You might counter that people sometimes argue about who should win an election, rather than who will win. But betting markets can collect info on that topic as well – we can bet on outcomes after the election conditional on who wins the election. These sort of markets would be enormously helpful to tell voters about which candidate will best promote health, peace, or prosperity. Yet such markets are now banned because they might “unduly” influence elections, or let people “waste” their time “arguing” about elections. Heaven forbid we should waste time figuring out which candidate would actually help us more.

GD Star Rating
loading...
Tagged as: , , ,

Seeking Firm Info Hypocrisy Examples

By Robin Hanson · March 6, 2012 1:40 pm · Comments (35)

Arnold Kling:

[James] Manzi is a fan of randomized controlled experiments in business and public policy (in the latter, examples include the Rand health care study and the Wisconsin income-maintenance studies). I believe that decision-makers will resist this approach, for the same reason that they resist Robin Hanson’s suggestion to use prediction markets. That is, decisions are not necessarily about achieving results. They are often about establishing the status of the decision-maker. For a decision-maker to conduct experiments or to employ prediction markets is to admit ignorance and doubt, which lowers the decision-maker’s status. (more)

The examples of disinterest in random trials and prediction markets both help convince me that management is often less interested in information that it pretends to be. But since I’m giving a talk on the subject soon, I’d like other examples. So I ask you, dear readers: what common patterns of manager behavior suggests they are less (or more) interested in info than they let on?

GD Star Rating
loading...
Tagged as: ,

Info Market Failure

By Robin Hanson · February 12, 2012 9:35 pm · Comments (8)

Unless project gains can be very clearly proven to analysts, or perhaps so small and numerous to allow averaging over them, public firms are basically incapable of taking a loss on earnings this quarter in order to make gains several years later. … CEOs are strongly tempted to instead please analysts by grabbing higher short-term quarterly earnings. …

Private firms are 3.5 times more responsive to changes in investment opportunities than are public firms. … IPO firms are significantly more sensitive to investment opportunities in the five years before they go public than after. (more)

A month ago I said that these results imply that we need wealth inequality, to ensure we make the discretionary investments on which all our future wealth depends.

Today I want to admit that these results also imply that even thick speculative markets, full of lots of people trading lots of money, often have big info failures. While I am a big fan of using speculative markets to aggregate info, I must admit that they quite often fail to aggregate all relevant info, even when a lot of money can be won there.

CEOs at private firms choose investments based on private info on likely rates of return. If the same firm were to be made public, however, the above evidence suggests that CEOs would make less than 25% of those investments. In the other 75+% of cases, the CEO would estimate that market speculators would not credit the stock price for the value of those promising investments, but would instead punish the firm for lower short term earnings. It seems that market speculators cannot distinguish these investments from other less promising ones that CEOs would undertake if speculators were to credit these. CEOs typically know crucial investment details not available to speculators.

Now I can see ways to improve existing stock markets, so that they could aggregate more investment info. We could allow and even encourage “insider” stock trading by firm insiders like the CEO. And we could create decision markets, trading the stock value conditional on specific investment decisions. But while these changes should raise that <25% figure, i.e., the fraction of investments by private firms that would also be made by a public firm, they might not raise it by much.

Speculative markets can work info aggregation wonders, at least compared to common methods like surveys or committee meetings. But if you really want as much info as possible on big investments, we still know of nothing better than rich private investors with a lot on the line.

GD Star Rating
loading...
Tagged as: ,

Heads In The Sand

By Robin Hanson · December 31, 2011 9:05 pm · Comments (12)

The end of a Boston Globe article on The future of prediction:

But the real question, when it comes to predicting the future of forecasting, may not be whether we can or can’t forecast accurately — it’s whether we want to. Robin Hanson, an economist at George Mason University and a pioneer of prediction market design, thinks that what’s holding back our ability to predict is not technology or a lack of ingenuity. He believes companies and governments already have much of what they need to be a lot better at predicting the future, and that the reason they’re not taking more advantage of it is that in many cases, having accurate predictions in hand makes managers, CEOs, and government officials accountable in a way that lots of them don’t want to be.

That’s because knowing the future can be a scary thing: It means genuinely answering for the costs of our decisions, confronting the likelihood of failure, seeing that arrows point down as often as they point up. When we’re offered a look into the crystal ball, it may in fact be human nature to turn away.

“We’re two-faced,” Hanson said. “We like to talk as though we wanted better forecasts, but often we have other agendas. When the opportunity to know the future presents itself — as, increasingly, it will — we may end up discovering that we’d rather stay in the dark.”

When projects fails, project managers like to say “No one could have foreseen that. We did the best we could.” This strategy doesn’t work so well when prediction markets or other credible methods create clear public track records showing consensus estimates of a high chance of failure, and perhaps also what could have been done to reduce that chance.

GD Star Rating
loading...
Tagged as: ,

Fixing Election Markets

By Robin Hanson · November 4, 2011 10:00 am · Comments (11)

One year from now the US will elect a new president, almost surely either a Republican R or a Democrat D. If there are US voters for whom politics is about policy, such voters should want to estimate post-election outcomes y like GDP, unemployment, or war deaths, conditional on the winning party w = R or D. With reliable conditional estimates E[y|w] in hand, such voters could then support the party expected to produce the best outcomes.

Sufficiently active conditional prediction markets can produce conditional estimates E[y|w] that are well-informed and resistent to biases and manipulation. One option is to make bets on y that are called off if w is not true. Another is to trade assets like  ”Pays $y if w” for assets like “Pays $1 if w.” A basic problem this whole approach, however, is that simple estimates E[y|w] may reflect correlation instead of causation.

For example, imagine that voters prefer to elect Republicans when they see a war looming. In this case if y = war deaths then E[y|R] might be greater than E[y|D], even if Republicans actually cause fewer war deaths when they run a war. Wolfers and Zitzewitz discuss a similar problem in markets on which party nominees would win the election:

It is tempting to draw a causal interpretation from these results: that nominating John Edwards would have produced the highest Democratic vote share. …The decision market tells us that in the state of the world in which Edwards wins the nomination, he will also probably do well in the general election. This is not the same as saying that he will do well if, based on the decision market, Democrats nominate Edwards. (more)

However, this problem has a solution: conditional close-election markets — markets that estimate post-election outcomes conditional not only on which party wins, but also on the election being close. This variation not only allows a closer comparison between candidates’ causal effects on outcomes, but it is also more relevant to an outcome-oriented voter’s decision. After all, an election must be close in order for your vote to influence the election winner.

To show that conditional close markets estimate causality well, I’ll need to get technical. And use probability math. Which I do now; beware.

Continue reading "Fixing Election Markets" »

GD Star Rating
loading...
Tagged as: ,

Official Optimism

By Robin Hanson · November 2, 2011 8:30 am · Comments (16)

Governments consistently overestimate their future budgets:

Analyzing data for 33 countries, Frankel finds that the average upward bias in the official forecast of the budget balance, relative to the realized balance, is 0.2 percent of GDP at the one-year horizon, 0.8 percent at the two-year horizon, and 1.5 percent at the three-year horizon. The longer the horizon, and the more genuine uncertainty there is, the more scope there is for wishful thinking. The bias is not larger for the commodity producers, … or for the developing countries, than for others. …

Over-optimism in predicting growth appears linked to over-optimism in predicting budget balances. On average, the upward bias in growth forecasts is 0.4 percent when looking one year ahead, 1.1 percent at the two-year horizon, and 1.8 percent at three years. The bias in growth forecasting appears in the United States and most other industrialized countries, but not among the commodity producing countries in the sample. …

Over-optimism is more prominent, for both budget balances and for economic growth, during economic booms. …. Countries subject to a budget rule … make official forecasts of growth and budget deficits that are even more biased and more correlated with booms than do other countries. Evidently when such governments exceed the deficit limits set by the rules, they respond by adjusting their forecasts rather than by adjusting their policies …

As a result of budget institutions created in 2000, Chile’s official forecasts of growth and of budget balance have not been overly optimistic, even in booms. (more)

The key institutional innovation [in Chile] is that there are two panels of experts whose job it is each mid-year to make the judgments, respectively, what is the output gap and what is the medium term equilibrium price of copper, rather than leaving the job to government officials. …. A reinforcement of the Chilean idea would be to give the panels legal independence. There could be laws protecting them from being fired, as there are for governors of independent central banks. (more)

Prediction markets forecasting budget balances and growth rates would be easy, and they’d reliably resist political pressure for overly optimistic estimates. So why even bother with trying to figure out how to design expert panels that can remain both expert and independent? Either Frankel naively thinks this easy, he is ignorant of the market solution, or doesn’t really want to promote accurate budget estimates.

GD Star Rating
loading...
Tagged as: , ,

Predict Yourself

By Robin Hanson · September 4, 2011 7:00 pm · Comments (9)

To act more on far ideals, predict what you will do:

Asking participants to predict their future vaccination behavior … substantially increased vaccination rates among patients with high short-term vaccination barriers (who, in the absence of this intervention, have low vaccination acceptance rates). These findings are consistent with past research on temporal construal, which suggests that people asked to think about a future behavior tend to focus its abstract benefits, and disregard concrete barriers that might impede it. (more)

Consider personal prediction markets, which predict what you will do in the future, such as whether you will lose weight, get married, get an A, get promoted, etc. By allowing your associates to participate in such markets, you could let them (anonymously) tell you what they really think about what you will do. Looking often at the predictions of such markets, and asking yourself if those predictions are wrong, could help you to live up to your far ideals about what you should and will do with your life.

GD Star Rating
loading...
Tagged as: , ,

Join The DAGGRE Team

By Robin Hanson · August 31, 2011 11:55 pm · Comments (18)

A few weeks back Tyler Cowen posted an appeal from Philip Tetlock:

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict [government-chosen] major trends and events around the world over the next four years. … [We] will be one of the five teams competing – and we’d like you to consider joining our team as a forecaster.

You may have seen other teams’ appeals as well. Today I can announce that GMU hosts one of the five teams, please join us! Active participants will earn $50 a month, for about two hours of forecasting work. You can sign up here, and start forecasting as soon as you are accepted.

The government sponsor is IARPA (Intelligence Advanced Research Projects Activity), under the ACE (Aggregative Contingent Estimation) program, and our team is DAGGRE (Decomposition-Based Elicitation & Aggregation).

Our approach has three distinctive features, all visible to participants:

  1. We use an edit-based interface – a current consensus on all questions is visible to all participants, and any user may change any part. Each edit is scored on whether it moves the consensus closer to or further from the truth.  (This is equivalent to a market-maker-based prediction market).
  2. For each question IARPA assigns, we “decompose” it by adding related questions, and letting participants forecast both related questions and how they relate to the assigned questions. For example, users can assume answers to some questions, and then forecast other questions conditional on their assumptions. (This is equivalent to a combinatorial prediction market.)
  3. We will sometimes walk users through a special elicitation process that has been shown in field and lab experiments to produce more accurate estimates.

(Items #2,3 might not show for a week or two.) We are eager to see how our approach compares to the other approaches. Come get paid to help us find out!

GD Star Rating
loading...
Tagged as: ,

New Scientist Contest

By Robin Hanson · August 16, 2011 5:30 pm · Comments (17)

New Scientist magazine set up a contest between new prediction techniques, including prediction markets:

We decided to see how the latest techniques would stand up to the task of predicting what people will buy. … Over the past four months, we have set four teams the task of trying to predict the sales of each issue of New Scientist, using some of the most promising and innovative approaches available. … Continue reading "New Scientist Contest" »

GD Star Rating
loading...
Tagged as: ,