At Loooooong Last

In 2001, DARPA started funding my Policy Analysis Market:

We planned to cover eight nations. For each nation in each quarter of a year, we planned to have traders predict its military activity, political instability, economic growth, US military activity, and US financial involvement. In addition traders would predict US GDP, world trade, … and a few to-be-determined miscellaneous items. This would require a hundred or so base markets. Most important, we wanted to let our traders predict combinations of these, such has how moving US troops out of Saudi Arabia would affect political stability there, how that would affect stability in neighboring nations, and how all that might change oil prices. …

[We] prepared for and ran lab experiments comparing two new combinatorial trading mechanisms with traditional mechanism. These experiments, where six traders set 255 independent prices in five minutes, found that a combinatorial market maker was the most accurate. Phase II was mostly being spent implementing a scaleable production version of this market maker.

Alas, disaster hit a month before we were to start live testing, and five months before we were to start public trading:

The media storm hit on July 28, 2003, when two senators (falsely) complained that we were planning to let people bet on individual terrorist attacks. The next morning the secretary of defense announced that FutureMAP was cancelled.

While the press on that event did help jump-start today’s prediction market industry, I have always regretted that the storm didn’t wait until we had a demo to show, of combinatorial markets on Mideast geopolitical events. This is why if felt so satisfying to announce Friday:

We are live! If you register at DAGGRE.org, you can join hundreds of others who browse and edit estimates on over 100 questions intended to be of interest to the US intelligence community. … You can also make assumptions, and then browse and edit as before.

Over nine years later, you can finally see the demo I wanted everyone to see in ’03! Of course this is only a play money market, and it isn’t open to everyone. We don’t allow foreigners, you can’t lose any money in it, and we only pay for activity, not accuracy. So there’s less reason for you to believe these prices as event estimates. But still, you can see combinatorial prediction markets in action. At long looooong last!

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • Romeo Stevens

    Have there been any attempts to look for systemic bias in play money markets vs real?

    • Jwbatey

      I was going to echo Romeo’s question. It would seem that the play money versions of this system would end up with a few lucky guessers holding the cash. In real markets others could then put in more cash if they were confident they could out-educated-guess… but that doesn’t seem to be an option here.

    • http://overcomingbias.com RobinHanson

      To my knowledge, there haven’t been controlled experiments where everything was held constant except play vs. real money.

      • Simon Lambert

        Having a parallel real money market might also reduce the incentives to distort the play money market.

  • Pingback: Assorted links

  • VV

    Aren’t prediction markets subject to the paradox that in order do be profitable they must be inaccurate?

    Since prediction markets are strictly zero sum, in order to profit predictors must be able to consistently beat the market by a margin that compensates prediction effort, transaction costs and risk aversion.

    If there is somebody who consistently does better than than average, there
    must be somebody else who consistently does worse, so hopefully these bad predictors will be driven away, increasing the accuracy of the average.

    But there is a catch: the more accurate the market average is, the more difficult it is to beat. If an event “true” probability is p, and the market estimates it at p+epsilon, then even if you are able to perfectly estimate p, your expected payoff is approx. epsilon * your bet.

    If epsilon is small (the market is accurate), then unless you can bet a very large amount of money, and the market has enough liquidity to take your bet, your payoff will not cover the prediction costs and transaction costs.

    Even if you and the market have enough liquidity, there is the issue of risk aversion: most interesting questions in a prediction market will be quite uncertain (after all, if a question had a probability > 0.9 or < 0.1, there would be little reason to put it on a prediction market, except for weird "Pascal's mugging" questions). By betting a large sum of money on a question with 0.4 < p < 0.6 you are taking a huge risk: even if your estimation of p was better than the market average you may just lose all your money by chance, and you certainly don't want to risk that unless your expected gain is very high, which can't be if the market is accurate.

    Therefore it seems to me that even when assuming unbounded liquidity, the accuracy of prediction markets is bounded.

    • http://juridicalcoherence.blogspot.com/ srdiamond

      Prediction markets, like gambling in general, are for people who either aren’t risk averse (that is, are desperate) or people who crave excitement (thus aren’t risk averse regarding some transactions).

    • http://juridicalcoherence.blogspot.com/ srdiamond

      The questions you raise are interesting in their broader practical application. As you pointed out in our previous discussion, once you’ve entered the stock market, trading there is a zero-sum game. So, why does so much trading go on? It turns out this is a very good question. Kahneman argues in Thinking, Fast and Slow that the whole trading industry is built on cognitive illusion and self-serving hype. He analyzed the data to find that traders lost money for their clients. The professional traders are beautiful examples of self-deceiving hypocrisy (with the exception of a very few who can deliver).

      But I don’t agree with your conclusion regarding the inherent limits of prediction markets (or by implication stock markets). If prediction is perfect, there’s no incentive to trade. So what? The prediction is perfect already.

    • rrb

      “Since prediction markets are strictly zero sum…”

      Many markets are like this, but Robin has written that he has higher expectations for markets that are subsidized by someone interested in the info:

      http://www.overcomingbias.com/2012/07/prediction-markets-fail-to-mooch.html

      Then there’s a net flow of money to the traders and it’s not zero-sum anymore. And you can expect to make some money even when the market’s predictions are very good.

      • http://juridicalcoherence.blogspot.com/ srdiamond

        Robin sensibly maintains that if you’re prepared to spend enough money, you can get enough entrants.

        But what’s enough? Here, Robin is purely algebraic, but theoretically, you can’t count on it being enough at least until the subsidy exceeds the transactions fees plus the interest on the funds traders venture.

      • VV

         But how do you hand out these subsides?

        If you subsidize only those whose predictions come true, then yes, you increase their expected payoff, but the issues I pointed out remain, particularly risk aversion: subsides or not, rational predictors are not going to bet lots of money on events that can turn out one way or the other.

        If you also subsidize those whose predictions come false, then you reduce the incentive to make accurate predictions. DAGGRE.org is an extreme example of this, since it rewards users for merely partecipating in the ‘market’, regardless of the accuracy of their predictions.

      • http://juridicalcoherence.blogspot.com/ srdiamond

        But how do you hand out these subsides?

        They go to winners. Risk aversion could be overcome to make the investment perfectly rational if and only if you make the subsidy great enough. I gather that the hope is something between rational investment and outright zero-sum-game gambling. Maybe someone who would otherwise bet on the horses could be induced to bet on prediction markets because his expected loss on the prediction market, due to the subsidy, is lower than on the horses.

        (Note to Drewfus: Thick markets with lots of rubes are definitely advantageous in the prediction markets, as Robin points out in his entry on subsidized prediction markets. The prediction markets want horse-track betters; yes, even slot players. You need thick and noisy markets.)

      • http://www.facebook.com/profile.php?id=599840205 Christian Kleineidam

        You subsidize by having transaction gains instead of transaction costs.
        On the stockmarket rational predictors are betting massive amount of money on events that can turn out one way or the other.

      • VV

         

        You subsidize by having transaction gains instead of transaction costs.

        So you reward activity, not accuracy.

        On the stockmarket rational predictors are betting massive amount of money on events that can turn out one way or the other.

        Rational predictors on the stock market?

        In the stock market it’s an empirically truth that you can make money using very simple and predicable investment strategies such as [index tracking](http://en.wikipedia.org/wiki/Index_fund). Such strategies would not work if everybody was using them, furthermore since they are completely predictable, hence they should be trivial to beat, yet they make money, which means that people who don’t use them manage to do worse on average.

        In fact, the stock average prices on the stock market fluctuate widely, and nobody (except perhaps some economists) really think these prices are accurate estimates of the holding values of the stocks.

      • http://juridicalcoherence.blogspot.com/ srdiamond

        You subsidize by having transaction gains instead of transaction costs.

        Well, yes, but there are various ways to subsidize transaction gains. The method of choice is to enlarge the pot.

      • http://juridicalcoherence.blogspot.com/ srdiamond

        Rational predictors on the stock market?

        But they fall short of complete rationality for reasons other than lack of incentive. Adequate incentives are what I take this discussion to be about. I think prediction marketers would be satisfied if they had sufficiently numerous traders as rational as stock market investors. It’s the potential number of traders, absent huge subsidies, that’s most questionable. The main problem is that, as gambling events, prediction markets fail the excitement test.

    • Drewfus

      “Since prediction markets are strictly zero sum, in order to profit predictors must be able to consistently beat the market by a margin that compensates prediction effort, transaction costs and risk aversion.”

      In a prediction market, the commission rate – denoted r in Parimutuel betting – can be negative.

      Presumably the organization wanting the odds data would pay a commission (perhaps call this the B2B commission, for clarity) to the betting agency, equalling r plus the agencies operating costs + profit per betting pool. The value of the data to the organization should obviously exceed the (B2B) commission paid to the agency.

      With a negative commission rate (aka ‘house-take’), even bettors of average accuracy can break-even, better than average bettors can make consistent profits, and only poor bettors will lose in the longer-term. Therefore, the negative commission makes the prediction markets attractive betting propositions in general, but still punishes poorer judges. This suggests an optimum trade-off between betting appeal and ‘punishing failure’.

      An interesting aspect is the effect on the status and authority of organization leaders who partly defer to the data of prediction markets as the basis for decision making. Perhaps the commission paid to the betting agency should come partly out of their salaries?

  • Rafał Rzepecki

    I’m disappointed by the restriction to US citizens. I’d very much like to see how it works.

  • Filipe

    Congratulations!

  • http://juridicalcoherence.blogspot.com/ srdiamond

    Don’t think this gets you off the hook on writing a book!

  • http://twitter.com/TruthHive TruthHive

    Whew, such a crazy Idea I must participate

  • Jayson Virissimo

    Robin Hanson, congratulations.

  • Pingback: Overcoming Bias : Announcing: SciCast