Leonhardt Blows It

Imagine someone said:

Of course I believe in science – I’m no nut job. I’m a modern guy. But scientists sometimes get it wrong, so we can’t just believe everything they say – we have to use our judgement. For example, my judgement tells me that astrology just makes sense. Well not today – today’s horoscope suggests I drink less, while I know I can handle my benders. But usually my horoscope feels right. And usually I feel no objection to what scientists say. Which is what I mean when I say that I believe in science.

Yes, every source errs sometimes, making it seem oh so sophisticated to say you don’t take sides, you just use your judgement in each case. But that is often just an excuse to believe whatever you feel like. On prediction markets, David Leonhardt sounds similar:

The odds at Intrade … continued to show about a 75 percent chance that the law’s so-called mandate would be ruled unconstitutional, right up until the morning it was ruled constitutional. … Today, mocking Intrade, ideally on Twitter, is a sign of sophistication. …

The early successes of prediction markets were notable. … But the crowd was not everywhere wise. For one thing, many of the betting pools on Intrade and Betfair attract relatively few traders, in part because using them legally is cumbersome. … The thinness of these markets can cause them to adjust too slowly to new information.

And there is this: If the circle of people who possess information is small enough — as with the selection of a vice president or pope or, arguably, a decision by the Supreme Court — the crowds may not have much wisdom to impart. “There is a class of markets that I think are basically pointless,” says Justin Wolfers. …

But such schadenfreude raises a question: once you accept that prediction markets are flawed, do you turn back to the inside experts? ALAS, the experts’ overall record remains as poor as the behavioral economists maintained — and often worse than the markets’ record. …

The answer, I think, is to take the best of what both experts and markets have to offer, realizing that the combination of the two offers a better window onto the future than either alone. Markets are at their best when they can synthesize large amounts of disparate information, as on an election night. Experts are most useful when a system exists to identify the most truly knowledgeable. …

Nate Silver … has found that a simple average of well-known economic forecasts is substantially more accurate than individual forecasts. Other times, the approach might involve as much art as science — and, again, the Internet allows for strategies that once would have been impossible.

Think for a moment about what a Twitter feed is: it’s a personalized market of experts (and friends), in which you can build your own focus group and listen to its collective analysis about the past, present and future. An RSS feed, in which you choose blogs to read, works similarly. You make decisions about which experts are worthy of your attention, based both on your own judgments about them and on other experts’ judgments. (more)

No, the vast majority of folks should not trust the vague impression on a subject they glean from a Twitter or RSS feed, over active prediction market prices. They shouldn’t even average the two. I’m confident that an empirical test of forecasts based on such impressions, or averages, would find them less accurate. Alas, folks like Leonhardt might then say “And that’s just why you must use your judgement about when when to use your judgement.”

Yes, in a sense, you always use your judgement – if you take my advice to rely on prediction market prices instead of Twitter feed impressions,  your judgement will have to approve of that. But using your judgement isn’t the same as accepting your case-specific intuitions – you usually can and should judge them unreliable.

Also, prediction markets just are not “crowds” in contrast to “experts” – the whole point of prediction markets is to get participants to self-select as the true experts. Don’t participate unless you think you know more than other participants, and those who actually do know less lose on average and get slowly pushed out.

Yes, you should be wary of “prediction markets” where no one trades, or limited to the kids from Mrs. Calloway’s seventh grade civics class, or using a play money no one cares about. Not everything called a “prediction market” is one. But the Intrade market on the Obamacare court case was an active valid market, on an appropriate subject. When it assigned a 75% chance to an event it was saying real loud that it would be wrong 1/4 of the time. And studies have consistently found such markets are well-calibrated in this way. What more do you want?

Yes, Intrade markets on court cases are unlikely to extract inside court info, and would be less accurate than sources with access to such info. But do you really think your Twitter feed has better access? Intrade traders watch Twitter, and incorporate what info they find into prices as best they can. Skeptics who tweet their disagreements but aren’t willing to bet can’t be very confident.

Yes, prediction markets can’t be reliable sources unless some people at some times think they are unreliable, and bet on that opinion. It is those with enough confidence in their disagreements to bet that make such markets accurate. If you are such a person, more power to you. But if you are not such a person, you will almost always get more accurate estimates by just trusting the current prices of an active prediction market, relative to forming a vague impression based on your Twitter feed.

Of course, on subjects like major court cases, most people care about others things besides accuracy. Twitter feeds can connect you to people, helping you to form and show your allegiances. For such social purposes, prediction markets are worse. But since people can’t usually admit such priorities, they have to make up excuses, such as about listening to Twitter feeds to aggregate a vague ineffable wisdom of expert crowds.

GD Star Rating
Tagged as:
Trackback URL:
  • daedalus2u

    An interpretation of this particular case could also be that the people who have so much money that they are willing to bet it on the SCOTUS ruling are also the people who are most out of touch with reality.  In other words, people with more money than sense.  

    Averaging a number of estimates only works to reduce common mode error when those estimates are independent and so don’t have common mode error.  That is their errors are non-correlated and so will tend to average out.  If the estimates are not independent (for example if the estimators are just playing follow-the-leader, or if they all live in the same echo-chamber), then their errors are not independent and won’t average out.  

    The worst thing to do is to base your estimate on tribal beliefs.  Tribal beliefs are known to be acquired through tribal interactions which are always not independent of others in the tribe.  Hence tribal beliefs are beliefs most likely to be biased and in error, and members of the tribe are those least able to tell that they are in error. 

    • Ari Timonen

      Your interpretation? Or scientific study of prediction markets?

      Easy to say when you have nothing on the line…

      • daedalus2u

         Nothing but my intellectual integrity, which I place an extremely high value on.  

        Virtually everyone places higher value on the status obtained by subscribing to certain tribal beliefs, which is the only reason there are tribal beliefs that contradict reality.  

        That is one of the major points of Robin’s writings, that people put more value on status than they do on having beliefs that match reality.  Of course people have a hard time admitting that, even to themselves, but sometimes it happens.  

        To those who believe they should adopt what ever beliefs offer them the greatest monetary return, then if tribal beliefs do that, then they will adopt what ever tribal beliefs do that. 

      • Ari Timonen

        Intellectual integrity != confidence interval. I can say “I believe X will be next president”, and be totally honest about it but once you bring in accountability, then we will see just how much. There are very few opinions I’d put a lot of money on, because I know just how easy it is to be wrong.

        If you are going to talk about these tribal beliefs all day, then maybe you could post some peer-reviewed research, possibly yours? It is easy to say that “oh, only people with more money than sense bet there” out of Stetson. Maybe instead post some research made by people who study prediction markets and know the dynamics of them.I never said you should adopt beliefs that offer greatest monetary return. People just respond to incentives, and making them (monetarily) accountable usually makes them a lot more rational.

    • Jayson Virissimo

      With your superior knowledge of the effect of tribal beliefs on prediction markets, shouldn’t you be able to make a killing betting in such markets (and therefore, improve the accuracy of their prices)?

    • Ilya Shpitser

      Speaking of tribal beliefs, do you actually disagree with a “typical American progressive” on any testable question with political overtones?

      • AnthonyC

        “Growing enough food to feed people requires industrial efficiency and industrial scale” seems like a start.

    • daedalus2u

       The marginal benefit to society that I could do by spending my resources making prediction markets more accurate is tiny.  My time and effort generates more benefits doing other things.  People with tribal beliefs are not going to change them simply because they are shown to be wrong.  That is the whole point of tribal beliefs.  They hold the tribe together in the face of adversity and even reality. 

      How many believe that Obama was born in Kenya?  Or that Obama is a Muslim?  Or that the Earth is less than 10,000 years old?  Or that there were WMD in Iraq?  Or that AGW isn’t happening?  Or that allowing gays to marry will destroy heterosexual marriages?  Or that Obamacare has death panels in it?  Or that Obama has expanded the size of government?  Or that golden tablets were dug up on a farm in NY in 1823 containing the religious history of one of Israel’s Lost Tribes?  None of these beliefs have any data to support them, they were arrived at in the absence of data.  Data may have been cherry-picked to try and support these beliefs, but that is simply lying by another name.  What is interesting is that members of the tribe rarely appreciate that they have been lied to, and if they do, it doesn’t change their beliefs, and it certainly doesn’t turn them against the tribal leaders that did the lying and who promote and foster those false beliefs (for their own benefit).   

      I don’t really pay attention to what other progressives are trying to do, so I am not so much caught up in progressive tribal beliefs.  I do think the anti-genetic modification stuff is nonsense, as is the pro-organic stuff.  I think growing organic food is a waste of farmland.  People can do it as a hobby, but growing enough food to feed people requires industrial efficiency and industrial scale.  Some pesticides are ok, but the EPA does a lousy job of regulating out the ones that are not ok, such as the neonicotinoids that are causing colony collapse disorder in bees and the triazine herbicides. 

      I don’t consider myself to be a member of any tribe.  I am extremely liberal and I only know of one person who is as liberal as I am, and none that are more liberal.  I feel that is because reality has a liberal bias and I want to go with reality, not any kind of tribal belief. 

      I actively seek out other sources of information.  This blog is not exactly a hotbed of progressive activism promoting progressive tribal beliefs. 

  • Doug

    I agree with almost everything you’re saying. Prediction markets (especially liquid and deep ones) I am sure are orders of magnitude more informative than even the average expert.

    The question that you don’t seem to directly address is do experts offer any marginal information to in addition to prediction markets. On this I’m inclined to say yes. You wouldn’t want to equal weight the two, but at some non-zero weight you might want to use experts.

    The experience with many prediction systems, e.g. Netflix, shows that even  crappy, but diverse, predictors have small amounts of marginal information to add.

    To take this back to prediction markets, look at the most liquid, robust and deep prediction markets we have today: equity markets. Over long periods of time this prediction market has been systematically wrong with regard to a few simple rules. Low P/E stocks out perform high P/E stocks on average, momentum stocks out-perform, and small cap outperforms.

    Not that you want to ignore the prediction market in favor of these things, but you want to tilt what the prediction market is telling you (a market-cap weighted index) in favor of value, growth and small-cap. This indicates to me that there’s additional predictors that can still generate marginal improvement in addition to prediction markets.

    • lightreadingguide

      We now know one winning bet would have been “the “individual mandate will not be found constitutional under the commerce clause.”  We also know that a losing bet would have been “one of the judges subject to changing his mind will not change his mind based on his stated desire to find the individual mandate, slightly rewritten,  consitutional under some clause besides the commerce clause”. 
      To take your small-cap analogy to heart, that second bet could only have been placed in a third tier small-cap market, i.e. a market not only smaller than the big market “constitutional or not”, but also smaller than the less-big market “not struck down on some basis widely publicly discussed” (which would add in jurisdiction but would not add in the rarely discussed taxing power issue); a prediction markets winner in this case would be most like someone who bets on the “field” at a horse race.
      If I were in charge of going back in time and setting up a more  optimal prediction market on this, I would phrase the bet as “which argument set forth in one or more law review articles or briefs most closely approximates the winning result”,  but that would have taken a lot of work to set forth the multiple distinguishable possibilities and would have discouraged everyone from betting unless they had done possibly excessive research  (according to one source I have read = at Eugene Volokhs group blog = there is exactly one law review article, out of dozens or more,  which set forth an argument closely tracking the winning argument) 

      • Paul

        I took the intrade market as more evidence for the claim that Roberts changed his mind at the last minute.

    • Arch1

      I’d be very interested in seeing Robin’s or others’ responses to this.

    • http://overcomingbias.com RobinHanson

      Claims about tiny improvements aren’t very interesting, or testable.

    • http://entitledtoanopinion.wordpress.com TGGP

      Somebody should start such a weighted index so I can buy it. Or if it already exists, who offers it and just what is its track-record compared to the market?

  • Student T

    There’s a special problem with using prediction markets to predict judicial opinions, particularly the ACA decision. FArrovian cycling can occur in Supreme Court opinions. To some extent, the potential for cycling is constrained by precedent, but the ACA decision was a matter of first impression. So there’s cycling, which is inherently unpredictable.

    A related problem is that there might be logrolling, but, absent insider info, there’s no way to know. 

    This doesn’t mean that prediction markets aren’t the best publicly available predictors of Supreme Court court opinions. They probably are.

  • Jsh

    Learn to spell judgment.  J-u-d-g-m-e-n-t.

    • http://www.mccaughan.org.uk/g/ gjm

      Both the Oxford English Dictionary (for BrE) and the online Merriam-Webster (for AmE) permit “judgement” as well. M-W (and, I think, most US usage) prefers “judgment”; OED (and, I think, most UK usage) prefers “judgement”; neither is *wrong* in either variety of English and there is no reason to think that Robin needs to “learn to spell” the word.