Oh I think you're right! I mean, the inertia Gelman and Rotschild talk about in that article has in fact been observed in all kinds of markets -- e.g. a study of the Brazilian stock market found out it takes up to 50 minutes so that prices fully incorporate new information. That inertia is due not only to the default perception that the market got it right already, but also to the fact that the research needed to correct prices is costly. Hypermind is dramatically more illiquid than a stock market, so it should take not minutes but days, perhaps weeks (!). So, as I said, one *should* use information on recent mistakes to revise their confidence in the precision of prediction markets. Still, their track record remains impressive.
Supporting the opposite conclusion: that the probabilities do not justify the recent errors (also supporting my observation about the Trump market not varying properly with new information):
Traders have incentives to find and correct mis-calibrations. So the strong their incentives via high liquidity and low transaction costs, the closer these markets get to well-calibrated.
Robin,It is not at all clear to me why prediction markets should in general be well-calibrated. The prediction market only amalgamates information available to individual market participants, and sometimes it is impossible to judge how representative of the whole that amalgamation is. If we are running a prediction market on the length of the emperor's nose, but the emperor has never appeared in public, why should we expect the market to be any better than 100% noise?
In the Brexit case (and similar political events), the information amalgamated was presumably exclusively from publically available sentiment data (i.e. polls), and participants' individual viewpoints. The polls have "known unknown" (e.g. difference in demographic between telephone and Internet voters) and "unknown unknown" biases. Participants in prediction market trading are highly self-selecting and may not at all be representative of the views of the UK population at large. I would not expect prediction markets for these kinds of events to be at all well calibrated.
So my question was really on whether the calibration curve for (e.g.) political event markets has a shape that remains stable across events and time (signifying some kind of behavioural invariant that we can measure), and whether there then really is an opportunity to use this to do better.
But Adrian is right that the to extent PM's are not well-calibrated, the curve is S-shaped (for some value of S). The exact shape is sort of interesting, AFAICT not completely consistent across studies, and apparently combines several effects. Among these are: 1. Favorite-Long Shot bias (a 100-1 underdog will actually be priced at more like 2-3%; I think market design, i.e. transaction costs and automatic liquidity provision mechanisms if any, have a lot to do with this). 2. Asymmetry with respect to the p <-> 1-p transformation on probability space; there might be a fundamental feature of human psychology that causes us to think about 99% chances differently from 1% chances. 3. There may be elements of prospect theory and other behavioral phenomena (mental accounting/disposition effects) that come into play, since the types of bettors on either side of a skewed binary event may be subject to such biases to different degrees.
I haven't seen the ultimate meta-study about this question, although I think that combining all the data from GJP and other IARPA ACE markets (e.g. DAGGRE, Scicast), together with other isolated experiments (IEM, foresight exchange, etc.), and perhaps non-PM predictive surveys (e.g. GJP's, or the results from websites like Predictionbook) there's probably enough data to establish even fairly subtle effects (say a couple % points of systematic miscalibration) with a relatively high degree of statistical significance. I'm not sure how subtle the effects actually are when averaged across all the available data. The only data I spent any amount of time looking at was from Scicast, where you can see very distinct S-shaped calibration curves.
Well, you would expect betting markets to lose occasionally in comparison tests. But prediction markets should in theory do better than polls because prediction markets are supposed to be good at combining information, and one supposes polls are a subset of that information. So, the appropriate null hypothesis isn't no difference but a better performance by prediction markets. Still, being outdone by polls in a particular instance could be insufficient to reject even the revised null hypothesis.
[One thing I'm struck by is the stability of the market values in the general presidential election. Orlando happens, for example, but the odds remain the same. The odds are different on different markets, but on each they stay the same. Yet these events have an effect on the polls, and it seems they should affect the odds.]
Thinking of the Age of Em, what advice do you have for current humans? I have in mind practical suggestions, given the expectation that ems will arrive in a not-to-distant future and that our actions will be largely unable to alter the society which ultimately prevails (i.e., your regulation-free baseline).
For example, should we/our children de-value investments in economic skills ems are likely to assume? Should we maximize our earnings in the near future before our skills are made obsolete, perhaps even dropping out of school or delaying leisure to the future? (If this is not optimal for those presently living, will it be so for soon-to-be born generations?) Should we invest more is skills/signals that other humans will continue to admire even once ems arrive? If so, which might these be (presumably activities more like art than chess, perhaps some athletic skills…)? How should one invest if the goal is to maximize the probability of (a) being extremely wealthy or (living comfortably after ems arrive?
Arent there restrictions on how much money you're allowed to put in betting markets? I bet if there were millions of dollars left on the table, many would take that. But if there is only a limited amount to be made then ideology trumps money.
Because straight up polling was more accurate than betting markets. This isn't just betting markets failing. This was them being beaten by another method and that's very interesting.
Robin,What I would like to see (and apologies if this has already been done and I missed it) is a meta-study comparing implied probabilities from prediction markets before events, against post hoc distributions of outcomes.
So, something like: look at all events where the prediction market implied a probability of 0-10%. What proportion of those events then happened? Do the same for 20-30% etc. so we have a map of deciles of a priori to post hoc.
We really want this plot to be linear, but my guess is that the curve will instead be S-shaped: that prediction markets will underprice "unlikely" events and overprice "likely" events. The exact shape might have some interesting implications.
Oh I think you're right! I mean, the inertia Gelman and Rotschild talk about in that article has in fact been observed in all kinds of markets -- e.g. a study of the Brazilian stock market found out it takes up to 50 minutes so that prices fully incorporate new information. That inertia is due not only to the default perception that the market got it right already, but also to the fact that the research needed to correct prices is costly. Hypermind is dramatically more illiquid than a stock market, so it should take not minutes but days, perhaps weeks (!). So, as I said, one *should* use information on recent mistakes to revise their confidence in the precision of prediction markets. Still, their track record remains impressive.
Supporting the opposite conclusion: that the probabilities do not justify the recent errors (also supporting my observation about the Trump market not varying properly with new information):
http://www.slate.com/articl...
Traders have incentives to find and correct mis-calibrations. So the strong their incentives via high liquidity and low transaction costs, the closer these markets get to well-calibrated.
Robin,It is not at all clear to me why prediction markets should in general be well-calibrated. The prediction market only amalgamates information available to individual market participants, and sometimes it is impossible to judge how representative of the whole that amalgamation is. If we are running a prediction market on the length of the emperor's nose, but the emperor has never appeared in public, why should we expect the market to be any better than 100% noise?
In the Brexit case (and similar political events), the information amalgamated was presumably exclusively from publically available sentiment data (i.e. polls), and participants' individual viewpoints. The polls have "known unknown" (e.g. difference in demographic between telephone and Internet voters) and "unknown unknown" biases. Participants in prediction market trading are highly self-selecting and may not at all be representative of the views of the UK population at large. I would not expect prediction markets for these kinds of events to be at all well calibrated.
So my question was really on whether the calibration curve for (e.g.) political event markets has a shape that remains stable across events and time (signifying some kind of behavioural invariant that we can measure), and whether there then really is an opportunity to use this to do better.
-Adrian
But Adrian is right that the to extent PM's are not well-calibrated, the curve is S-shaped (for some value of S). The exact shape is sort of interesting, AFAICT not completely consistent across studies, and apparently combines several effects. Among these are: 1. Favorite-Long Shot bias (a 100-1 underdog will actually be priced at more like 2-3%; I think market design, i.e. transaction costs and automatic liquidity provision mechanisms if any, have a lot to do with this). 2. Asymmetry with respect to the p <-> 1-p transformation on probability space; there might be a fundamental feature of human psychology that causes us to think about 99% chances differently from 1% chances. 3. There may be elements of prospect theory and other behavioral phenomena (mental accounting/disposition effects) that come into play, since the types of bettors on either side of a skewed binary event may be subject to such biases to different degrees.
I haven't seen the ultimate meta-study about this question, although I think that combining all the data from GJP and other IARPA ACE markets (e.g. DAGGRE, Scicast), together with other isolated experiments (IEM, foresight exchange, etc.), and perhaps non-PM predictive surveys (e.g. GJP's, or the results from websites like Predictionbook) there's probably enough data to establish even fairly subtle effects (say a couple % points of systematic miscalibration) with a relatively high degree of statistical significance. I'm not sure how subtle the effects actually are when averaged across all the available data. The only data I spent any amount of time looking at was from Scicast, where you can see very distinct S-shaped calibration curves.
See the Success section of the Choices chapter.
Fair enough; I've only skimmed the book as yet. It is not obvious from the TOC where to look, though -- can you point to some passages or chapters?
Accumulate capital.
Opinions on Ralph Merkle's "DAOs, Democracy and Governance"?
http://merkle.com/papers/DA...
The book does have sections on these topics.
Well, you would expect betting markets to lose occasionally in comparison tests. But prediction markets should in theory do better than polls because prediction markets are supposed to be good at combining information, and one supposes polls are a subset of that information. So, the appropriate null hypothesis isn't no difference but a better performance by prediction markets. Still, being outdone by polls in a particular instance could be insufficient to reject even the revised null hypothesis.
[One thing I'm struck by is the stability of the market values in the general presidential election. Orlando happens, for example, but the odds remain the same. The odds are different on different markets, but on each they stay the same. Yet these events have an effect on the polls, and it seems they should affect the odds.]
Thinking of the Age of Em, what advice do you have for current humans? I have in mind practical suggestions, given the expectation that ems will arrive in a not-to-distant future and that our actions will be largely unable to alter the society which ultimately prevails (i.e., your regulation-free baseline).
For example, should we/our children de-value investments in economic skills ems are likely to assume? Should we maximize our earnings in the near future before our skills are made obsolete, perhaps even dropping out of school or delaying leisure to the future? (If this is not optimal for those presently living, will it be so for soon-to-be born generations?) Should we invest more is skills/signals that other humans will continue to admire even once ems arrive? If so, which might these be (presumably activities more like art than chess, perhaps some athletic skills…)? How should one invest if the goal is to maximize the probability of (a) being extremely wealthy or (living comfortably after ems arrive?
That is called a "calibration curve", and prediction markets tend to be well calibrated, and more so the lower are transaction costs.
Arent there restrictions on how much money you're allowed to put in betting markets? I bet if there were millions of dollars left on the table, many would take that. But if there is only a limited amount to be made then ideology trumps money.
Because straight up polling was more accurate than betting markets. This isn't just betting markets failing. This was them being beaten by another method and that's very interesting.
Robin,What I would like to see (and apologies if this has already been done and I missed it) is a meta-study comparing implied probabilities from prediction markets before events, against post hoc distributions of outcomes.
So, something like: look at all events where the prediction market implied a probability of 0-10%. What proportion of those events then happened? Do the same for 20-30% etc. so we have a map of deciles of a priori to post hoc.
We really want this plot to be linear, but my guess is that the curve will instead be S-shaped: that prediction markets will underprice "unlikely" events and overprice "likely" events. The exact shape might have some interesting implications.
-Adrian