Leonhardt Blows It

Imagine someone said:

Of course I believe in science – I’m no nut job. I’m a modern guy. But scientists sometimes get it wrong, so we can’t just believe everything they say – we have to use our judgement. For example, my judgement tells me that astrology just makes sense. Well not today – today’s horoscope suggests I drink less, while I know I can handle my benders. But usually my horoscope feels right. And usually I feel no objection to what scientists say. Which is what I mean when I say that I believe in science.

Yes, every source errs sometimes, making it seem oh so sophisticated to say you don’t take sides, you just use your judgement in each case. But that is often just an excuse to believe whatever you feel like. On prediction markets, David Leonhardt sounds similar:

The odds at Intrade … continued to show about a 75 percent chance that the law’s so-called mandate would be ruled unconstitutional, right up until the morning it was ruled constitutional. … Today, mocking Intrade, ideally on Twitter, is a sign of sophistication. …

The early successes of prediction markets were notable. … But the crowd was not everywhere wise. For one thing, many of the betting pools on Intrade and Betfair attract relatively few traders, in part because using them legally is cumbersome. … The thinness of these markets can cause them to adjust too slowly to new information.

And there is this: If the circle of people who possess information is small enough — as with the selection of a vice president or pope or, arguably, a decision by the Supreme Court — the crowds may not have much wisdom to impart. “There is a class of markets that I think are basically pointless,” says Justin Wolfers. …

But such schadenfreude raises a question: once you accept that prediction markets are flawed, do you turn back to the inside experts? ALAS, the experts’ overall record remains as poor as the behavioral economists maintained — and often worse than the markets’ record. …

The answer, I think, is to take the best of what both experts and markets have to offer, realizing that the combination of the two offers a better window onto the future than either alone. Markets are at their best when they can synthesize large amounts of disparate information, as on an election night. Experts are most useful when a system exists to identify the most truly knowledgeable. …

Nate Silver … has found that a simple average of well-known economic forecasts is substantially more accurate than individual forecasts. Other times, the approach might involve as much art as science — and, again, the Internet allows for strategies that once would have been impossible.

Think for a moment about what a Twitter feed is: it’s a personalized market of experts (and friends), in which you can build your own focus group and listen to its collective analysis about the past, present and future. An RSS feed, in which you choose blogs to read, works similarly. You make decisions about which experts are worthy of your attention, based both on your own judgments about them and on other experts’ judgments. (more)

No, the vast majority of folks should not trust the vague impression on a subject they glean from a Twitter or RSS feed, over active prediction market prices. They shouldn’t even average the two. I’m confident that an empirical test of forecasts based on such impressions, or averages, would find them less accurate. Alas, folks like Leonhardt might then say “And that’s just why you must use your judgement about when when to use your judgement.”

Yes, in a sense, you always use your judgement – if you take my advice to rely on prediction market prices instead of Twitter feed impressions,  your judgement will have to approve of that. But using your judgement isn’t the same as accepting your case-specific intuitions – you usually can and should judge them unreliable.

Also, prediction markets just are not “crowds” in contrast to “experts” – the whole point of prediction markets is to get participants to self-select as the true experts. Don’t participate unless you think you know more than other participants, and those who actually do know less lose on average and get slowly pushed out.

Yes, you should be wary of “prediction markets” where no one trades, or limited to the kids from Mrs. Calloway’s seventh grade civics class, or using a play money no one cares about. Not everything called a “prediction market” is one. But the Intrade market on the Obamacare court case was an active valid market, on an appropriate subject. When it assigned a 75% chance to an event it was saying real loud that it would be wrong 1/4 of the time. And studies have consistently found such markets are well-calibrated in this way. What more do you want?

Yes, Intrade markets on court cases are unlikely to extract inside court info, and would be less accurate than sources with access to such info. But do you really think your Twitter feed has better access? Intrade traders watch Twitter, and incorporate what info they find into prices as best they can. Skeptics who tweet their disagreements but aren’t willing to bet can’t be very confident.

Yes, prediction markets can’t be reliable sources unless some people at some times think they are unreliable, and bet on that opinion. It is those with enough confidence in their disagreements to bet that make such markets accurate. If you are such a person, more power to you. But if you are not such a person, you will almost always get more accurate estimates by just trusting the current prices of an active prediction market, relative to forming a vague impression based on your Twitter feed.

Of course, on subjects like major court cases, most people care about others things besides accuracy. Twitter feeds can connect you to people, helping you to form and show your allegiances. For such social purposes, prediction markets are worse. But since people can’t usually admit such priorities, they have to make up excuses, such as about listening to Twitter feeds to aggregate a vague ineffable wisdom of expert crowds.