Open Thread

By Robin Hanson · July 1, 2016 7:00 am · Comments (20) · ·

This is a place to discuss relevant topics that have not appeared in recent posts. (Used to do this monthly; let’s see if worth reviving.)

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • free_agent

    I’d like to see an analysis of the failure of the betting markets to predict Brexit.

    • http://overcomingbias.com RobinHanson

      I don’t see why that is more interesting than asking why ANY forecast isn’t 100% certain.

      • Adrian Cable

        Robin,
        What I would like to see (and apologies if this has already been done and I missed it) is a meta-study comparing implied probabilities from prediction markets before events, against post hoc distributions of outcomes.

        So, something like: look at all events where the prediction market implied a probability of 0-10%. What proportion of those events then happened? Do the same for 20-30% etc. so we have a map of deciles of a priori to post hoc.

        We really want this plot to be linear, but my guess is that the curve will instead be S-shaped: that prediction markets will underprice “unlikely” events and overprice “likely” events. The exact shape might have some interesting implications.

        -Adrian

      • http://overcomingbias.com RobinHanson

        That is called a “calibration curve”, and prediction markets tend to be well calibrated, and more so the lower are transaction costs.

      • sflicht

        But Adrian is right that the to extent PM’s are not well-calibrated, the curve is S-shaped (for some value of S). The exact shape is sort of interesting, AFAICT not completely consistent across studies, and apparently combines several effects. Among these are: 1. Favorite-Long Shot bias (a 100-1 underdog will actually be priced at more like 2-3%; I think market design, i.e. transaction costs and automatic liquidity provision mechanisms if any, have a lot to do with this). 2. Asymmetry with respect to the p 1-p transformation on probability space; there might be a fundamental feature of human psychology that causes us to think about 99% chances differently from 1% chances. 3. There may be elements of prospect theory and other behavioral phenomena (mental accounting/disposition effects) that come into play, since the types of bettors on either side of a skewed binary event may be subject to such biases to different degrees.

        I haven’t seen the ultimate meta-study about this question, although I think that combining all the data from GJP and other IARPA ACE markets (e.g. DAGGRE, Scicast), together with other isolated experiments (IEM, foresight exchange, etc.), and perhaps non-PM predictive surveys (e.g. GJP’s, or the results from websites like Predictionbook) there’s probably enough data to establish even fairly subtle effects (say a couple % points of systematic miscalibration) with a relatively high degree of statistical significance. I’m not sure how subtle the effects actually are when averaged across all the available data. The only data I spent any amount of time looking at was from Scicast, where you can see very distinct S-shaped calibration curves.

      • Adrian Cable

        Robin,
        It is not at all clear to me why prediction markets should in general be well-calibrated. The prediction market only amalgamates information available to individual market participants, and sometimes it is impossible to judge how representative of the whole that amalgamation is. If we are running a prediction market on the length of the emperor’s nose, but the emperor has never appeared in public, why should we expect the market to be any better than 100% noise?

        In the Brexit case (and similar political events), the information amalgamated was presumably exclusively from publically available sentiment data (i.e. polls), and participants’ individual viewpoints. The polls have “known unknown” (e.g. difference in demographic between telephone and Internet voters) and “unknown unknown” biases. Participants in prediction market trading are highly self-selecting and may not at all be representative of the views of the UK population at large. I would not expect prediction markets for these kinds of events to be at all well calibrated.

        So my question was really on whether the calibration curve for (e.g.) political event markets has a shape that remains stable across events and time (signifying some kind of behavioural invariant that we can measure), and whether there then really is an opportunity to use this to do better.

        -Adrian

      • http://overcomingbias.com RobinHanson

        Traders have incentives to find and correct mis-calibrations. So the strong their incentives via high liquidity and low transaction costs, the closer these markets get to well-calibrated.

      • Jared

        Because straight up polling was more accurate than betting markets. This isn’t just betting markets failing. This was them being beaten by another method and that’s very interesting.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Well, you would expect betting markets to lose occasionally in comparison tests. But prediction markets should in theory do better than polls because prediction markets are supposed to be good at combining information, and one supposes polls are a subset of that information. So, the appropriate null hypothesis isn’t no difference but a better performance by prediction markets. Still, being outdone by polls in a particular instance could be insufficient to reject even the revised null hypothesis.

        [One thing I’m struck by is the stability of the market values in the general presidential election. Orlando happens, for example, but the odds remain the same. The odds are different on different markets, but on each they stay the same. Yet these events have an effect on the polls, and it seems they should affect the odds.]

    • Vítor Margato

      The Economist seems to share free_agent’s concern. They recently wrote: “The list of losers from Britain’s vote to leave the European Union is long indeed, but very far down on it are evangelisers for the accuracy of prediction markets”. I don’t think that’s the Bayesian (and by Bayesian I mean logical) way of interpreting it. Prediction markets assigned high probabilities to e.g. both The Revenant winning the Oscar and the Golden State Warrios winning the 2015-16 NBA championship. Neither happened. But, well, that’s why we assign probabilities to things. You *can*, and in fact you should, use those three events to update (lower, in this case) your confidence on the relative accuracy of prediciton markets — but please take into account all the times where they “got it right”.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Supporting the opposite conclusion: that the probabilities do not justify the recent errors (also supporting my observation about the Trump market not varying properly with new information):

        http://www.slate.com/articles/news_and_politics/moneybox/2016/07/why_political_betting_markets_are_failing.html

      • Vítor Margato

        Oh I think you’re right! I mean, the inertia Gelman and Rotschild talk about in that article has in fact been observed in all kinds of markets — e.g. a study of the Brazilian stock market found out it takes up to 50 minutes so that prices fully incorporate new information. That inertia is due not only to the default perception that the market got it right already, but also to the fact that the research needed to correct prices is costly. Hypermind is dramatically more illiquid than a stock market, so it should take not minutes but days, perhaps weeks (!). So, as I said, one *should* use information on recent mistakes to revise their confidence in the precision of prediction markets. Still, their track record remains impressive.

    • Jared

      Arent there restrictions on how much money you’re allowed to put in betting markets? I bet if there were millions of dollars left on the table, many would take that. But if there is only a limited amount to be made then ideology trumps money.

  • Durthain

    If possible, I’d love to hear your thoughts on two things:

    1. Scott Adams’ take on Trump, Clinton, and persuasion (Do you think there’s something to what he’s saying? Do you think that the prediction markets are inadequately accounting for Trump’s persuasiveness? How much should Trump’s exceeding expectations in the primaries lead us to think he’ll do so again in November?)
    2. The apparent recent rise in nationalism in the West (How does it map onto forager/farmer values? Is it just a blip that will fade out in the next few years, or is there any reason to think it could be a long-term trend even as the world gets wealthier?)

  • blink

    Thinking of the Age of Em, what advice do you have for current humans? I have in mind practical suggestions, given the expectation that ems will arrive in a not-to-distant future and that our actions will be largely unable to alter the society which ultimately prevails (i.e., your regulation-free baseline).

    For example, should we/our children de-value investments in economic skills ems are likely to assume? Should we maximize our earnings in the near future before our skills are made obsolete, perhaps even dropping out of school or delaying leisure to the future? (If this is not optimal for those presently living, will it be so for soon-to-be born generations?) Should we invest more is skills/signals that other humans will continue to admire even once ems arrive? If so, which might these be (presumably activities more like art than chess, perhaps some athletic skills…)? How should one invest if the goal is to maximize the probability of (a) being extremely wealthy or (living comfortably after ems arrive?

    • http://overcomingbias.com RobinHanson

      The book does have sections on these topics.

      • blink

        Fair enough; I’ve only skimmed the book as yet. It is not obvious from the TOC where to look, though — can you point to some passages or chapters?

      • http://overcomingbias.com RobinHanson

        See the Success section of the Choices chapter.

    • Dave Lindbergh

      Accumulate capital.

  • Dave Lindbergh

    Opinions on Ralph Merkle’s “DAOs, Democracy and Governance”?

    http://merkle.com/papers/DAOdemocracyDraft.pdf