Once More, With Feeling

Justin Wolfers:

Judging by the pre-vote polls and prediction markets, the Democratic primary in New Hampshire created one of the most surprising upsets in U.S. political history.

Many take this to show prediction markets have failed. Paul Krugman:

Prediction markets – which you see, again and again, touted as having some mystical power to aggregate information, know no more than the conventional wisdom. … There’s no hint that the market saw either Iowa or New Hampshire coming, or knew anything beyond the bloviations of the talking heads.

Daniel Gross:

So, I’ve been watching the action in one of the political futures markets this evening, Intrade. And the action in this prediction market has reinforced my opinion that these are less futures markets than immediate-past markets.  The price movement tends to respond to conventional wisdom and polling data; it doesn’t lead them.

Again we see the usual media formula: when prediction markets give a >50% chance to what happened, they are hailed as oracles, but when they give a <50% chance, they are unmasked as charlatans.  But this is just the wrong way to evaluate such things. Justin Wolfers:

Prices suggest a probabilistic statement that the ultimate outcome was about a 7% chance. And as any horseplayer can tell you, sometimes the long shots do win.

We do not claim that prediction markets will always be more accurate that other methods.  Often other institutions do pretty much the best they can.  Rather, compared to other long-lived institutions which make frequently-updated probabilistic forecasts, we claim that well-traded prediction markets will in most situations rarely be much less accurate, and in some situations be much more accurate.  Often other institutions are seriously broken, and do much worse than is possible.   

GD Star Rating
Tagged as:
Trackback URL:
  • Sounds a lot like ‘science makes mistakes, is never certain, must be broken.’ If anyone genuinely believed they could do better, they would do, and cash in while they were at it.

  • I suspect that there is one element that might have been
    involved in this particular prediction failure, gender bias
    in terms of participation in these markets. I do not know,
    but I would lay odds that men way outnumber women in them.
    The comeback by Hillary was substantially a matter of women
    surging in response to her “tearing up” and the attacks on
    her. Insensitive men betting in the market did not pick up
    on this. But then, nobody else did either…

  • Constant

    There is reason for disappointment, or could be, depending on what one’s hopes were. The real disappointment is not that the winner got less than 50% (or whatever it was that happened) but that the markets did very little to reduce the level of uncertainty. As an analogy, suppose that it rains on average about 1 day out of every 10. Suppose, then, that a prediction market in weather predicts, every day, a 1/10 chance of rain. The market is “not wrong” in the sense that, indeed, the events that it assigns a 1/10 probability happen 1/10 of the time. However, it is disappointing to someone who was hoping that the weather prediction market could reduce his level of uncertainty. The political markets seem disappointing in this sense. So far, anyway.

  • Silas

    Robin_Hanson: Wouldn’t the relevant metric be: “Do x% of events happen that were assigned x% probability, over a large number of events, by method Y?” ? That is, the test of a prediction method is whether e.g. 7% of predictions assigned 7% probability come true.

  • Twilight of My Liberal Idol

    I’ve been singing Krugman’s praises for years. Anyone who could write The Accidental Theorist has a lot of credit in…

  • ryan

    Dr. Hanson,
    I don’t expect the prediction markets to be terribly prescient, but I do expect them to be internally consistent. In this particular instance, how plausible is it that the wide swings in the prices of Obama & Clinton futures represent rational responses to new information? Prior to Iowa, Intrade put Obama’s chance of winning the Iowa and the nomination at around 70% and 25% respectively. Suppose the markets assumed that a loss in Iowa would absolutely doom the Obama campaign, then the conditional probability of his nomination given a victory in Iowa is around 35%. But his numbers jumped to around 70%. Why wasn’t this shift irrational?

  • Nick Tarleton

    Ryan, you assume the results of the different primaries are independent. But Iowa could have sent signals affecting traders’ judgment of other primaries (e.g., Edwards’s second place finish could cause him to be taken more seriously); or voters could be subject to an irrational bandwagon effect (look how Obama moved up in the NH polls after Iowa). Still, 25% to 70% seems much too large of an increase, and it does look like Intrade overreacts to recent events.

  • Unknown

    If Intrade overreacts on a regular basis, and not just on occasion, then this suggests there’s an easy way to make some money, as well as starting to correct this at the same time.

  • ryan

    I’m not assuming that so much as assuming that whatever the markets believed two days after Iowa about the independence (or lack thereof) of the primaries, they had more or less the same beliefs two days prior to Iowa. For all I know, it might be perfectly reasonable for someone to take the results of Iowa as a sufficient statistic for the results of all primaries and caucuses. But suppose you thought that, and you also thought Obama had a 70% chance of winning Iowa. Why would you have simultaneously thought that he only had a 25% chance of winning the nomination? Like I said, I’m just wondering how it is that the markets are agreeing with their own implied beliefs.

  • Nick Tarleton

    As I meant to suggest by mentioning Edwards, the signal could consist of something besides who won, something the market didn’t appear to anticipate because there was no market for second place. Still, I agree that most of the apparent overreaction was real irrationality.

  • Unknown

    Ryan, presumably Obama could have won Iowa, but by a small margin, and in this case his chances of winning the nomination probably would not have went up to 70%.

    Again, presumably the main reason it was thought that he would not win the nomination was it Clinton would win it; her third place finish in Iowa was evidence against Clinton’s winning the nomination, much stronger evidence than if Obama had won and Clinton had finished a close second, and therefore more evidence that Obama would win the nomination.

    Whether these arguments are valid or not, they seem to indicate that there was nothing particularly inconsistent about Intrade’s beliefs.

  • Caledonian

    We do not claim that prediction markets will always be more accurate that other methods.

    This is irrelevant. You cannot compute the accuracy of a method with a single datapoint. The only thing one failure shows is that the method isn’t guaranteed to be correct.

  • David J. Balan

    The most reliable guide to life that I have found so far is that Paul Krugman is always right about everything.

  • Mike

    Yes; I think the Intrade odds were just about right. Strange/unexpected things sometimes happen, and it’s as simple as that.

  • ryan

    That’s a good point, but it still seems to me that the Iowa results were something the market said ex ante was fairly likely. So why the massive post-Iowa correction? Is 38% to 30% that impressive a point spread when the market was expecting an Obama victory?

  • Eric Hansen

    I wonder if the Intrade markets are attracting the right people. As a layman I could bet $50 on one side or another, that wouldn’t add much information. If the amount to gain were high enough to persuade Wall Street traders to quit their jobs and really investigate what is most probable, now that would add information. Anyone who hasn’t done significant investigation into the matter is just adding noise.

  • Robin,

    Yes, I agree. As Justin put it, the prediction markets are an aggregator of the conventional wisdom. When a probabilistic forecast is “wrong,” that’s part of the prediction error. With enough experience, perhaps we can assign reasonable uncertainty distributions to prediction-market forecasts.

    P.S. Regarding Ben Jones’s comment above: Not everyone enjoys betting or wants to put the time/effort into it.

  • Jor

    Unknown — I was thinking the same thing, but these markets are too rare. You would need to catch it reacting irrationally many times to make money.

    By the way, in ’04, when the mid-day exit polls were released, in-trade also wildly over-reacted and swung Kerry to 70%+. That made the day even more depressing for me when the result finally came out.

  • Andrew, granted. My point is that if the financial markets were consistently significantly off compared to other predictors, surely this would be quickly spotted and exploited? In the same way, anyone discovering they were consistently better at deciding football results than the bookies would break the system very quickly. Or receive a number of job offers….

  • Stuart Armstrong

    My point is that if the financial markets were consistently significantly off compared to other predictors, surely this would be quickly spotted and exploited? In the same way, anyone discovering they were consistently better at deciding football results than the bookies would break the system very quickly. Or receive a number of job offers….

    From my understanding, bookies decide their odds by arbitrage, not by estimating the true odds. Similarly, banks spend huge amounts of effort chasing down arbitrage opportunities; according to my friends in banking, actually playing the market by predicting the future values of stocks is a mugs game.

    So there may be a limit to how accurate a betting market can become. There may be a method that is consistently more accurate than the market, as long as it’s hard to implement, and the returns are relatively lower than the arbitrage opportunities.

    A quick impression here: since elections (like sport events) are filled with emotional and irrational beliefs, the arbitrage opportunities may be higher than in the cold stock market. If this is true, we’d expect the accuracy of the election betting markets to be lower than standard financial markets, as the smart bettors go for arbitrage rather than accuracy. Any data here?

    But your general point stands: there will be no easy method that is consistently better than betting markets.

  • Andrew, calibrated probabilistic forecasts already tell us their error rate.

  • Unknown

    I would expect going for arbitrage rather than accuracy would be one of the best ways to improve the accuracy of the market.

  • The uses of the word arbitrage here suggest that people are confused about the meaning of that word. The standard meaning refers to exploiting inconsistencies between two financial instruments.
    Market makers / bookies may occasionally use arbitrage, but mostly use other strategies such as reading supply and demand from the behavior of other traders.

  • Unknown

    Peter, that was exactly what I meant by arbitrage. This will improve the market because it will tend to penalize more the one with the more inaccurate price. And it is one of the best ways to improve accuracy because it doesn’t depend on a prior knowledge of the true probabilities.

  • tcpkac

    I don’t believe arbitrage has anything to do with accuracy. In fact, the word ‘accuracy’ in the context of prices is about as useful as the word ‘fairness’ in the context of taxes (how’s that for a troll ?).
    Arbitrage just implies a momentary difference in assessment.

  • J Thomas

    “If Intrade overreacts on a regular basis, and not just on occasion, then this suggests there’s an easy way to make some money, as well as starting to correct this at the same time.”

    This argument gets used a lot. However, the people who know better may not have enough money or want to risk enough money to actually correct the odds.

    How much risk is it? Suppose that the market is agreed that a candidate has a 10% chance but I have looked at all the available information very carefully and I believe it’s an 80% chance based on that information. How much money should I put on it? Say I invest $50,000 in this candidate and tomorrow an old girlfriend shows up who talks about his sexual fetishes? i couldn’t have predicted that, and I’m probably out $50,000.

    This might be a reliable way to make money if you can average it out over hundreds of thousands of bets, but I think the stock market is less risky. If the stock market overreacts on a regular basis, then I don’t depend on an unknown event. I win when they overreact provided I could get in place to profit by it.