But the real question, when it comes to predicting the future of forecasting, may not be whether we can or can’t forecast accurately — it’s whether we want to. Robin Hanson, an economist at George Mason University and a pioneer of prediction market design, thinks that what’s holding back our ability to predict is not technology or a lack of ingenuity. He believes companies and governments already have much of what they need to be a lot better at predicting the future, and that the reason they’re not taking more advantage of it is that in many cases, having accurate predictions in hand makes managers, CEOs, and government officials accountable in a way that lots of them don’t want to be.

That’s because knowing the future can be a scary thing: It means genuinely answering for the costs of our decisions, confronting the likelihood of failure, seeing that arrows point down as often as they point up. When we’re offered a look into the crystal ball, it may in fact be human nature to turn away.

“We’re two-faced,” Hanson said. “We like to talk as though we wanted better forecasts, but often we have other agendas. When the opportunity to know the future presents itself — as, increasingly, it will — we may end up discovering that we’d rather stay in the dark.”

When projects fails, project managers like to say “No one could have foreseen that. We did the best we could.” This strategy doesn’t work so well when prediction markets or other credible methods create clear public track records showing consensus estimates of a high chance of failure, and perhaps also what could have been done to reduce that chance.

Sufficiently active conditional prediction markets can produce conditional estimates E[y|w] that are well-informed and resistent to biases and manipulation. One option is to make bets on y that are called off if w is not true. Another is to trade assets like  ”Pays $y if w” for assets like “Pays $1 if w.” A basic problem this whole approach, however, is that simple estimates E[y|w] may reflect correlation instead of causation.

For example, imagine that voters prefer to elect Republicans when they see a war looming. In this case if y = war deaths then E[y|R] might be greater than E[y|D], even if Republicans actually cause fewer war deaths when they run a war. Wolfers and Zitzewitz discuss a similar problem in markets on which party nominees would win the election:

It is tempting to draw a causal interpretation from these results: that nominating John Edwards would have produced the highest Democratic vote share. …The decision market tells us that in the state of the world in which Edwards wins the nomination, he will also probably do well in the general election. This is not the same as saying that he will do well if, based on the decision market, Democrats nominate Edwards. (more)

However, this problem has a solution: conditional close-election markets — markets that estimate post-election outcomes conditional not only on which party wins, but also on the election being close. This variation not only allows a closer comparison between candidates’ causal effects on outcomes, but it is also more relevant to an outcome-oriented voter’s decision. After all, an election must be close in order for your vote to influence the election winner.

To show that conditional close markets estimate causality well, I’ll need to get technical. And use probability math. Which I do now; beware.

First let me introduce some notation. Here are some relevant variables:

x = context before the election
v = sum of votes, each +1 or -1,  in election
w = R if v>0, D if v<0, the election winner
y = an outcome influenced by election winner

Assume ties v=0 are decided by a coin flip. Let the estimates of a consistent market reflect consensus beliefs given by a joint probability distribution p(y,w,v,x). Assume traders know that this joint must satisfy a causality relation:

p(y,w,v,x) = [R(y|vx)*(1[v>0]+½*1[v=0]) +

D(y|vx)*(1[v<0]+½*1[v=0])]*q(vx)

where R(y|v,x), D(y|v,x) describe expected causal results of parties R,D on outcome y, which may depend on context v,x, and where q(v,x) describes expectations for that context. (The form 1[claim] is 1 if claim is true, else 0.)

Let us approximate v as being distributed continuously.  If so, here are integral expressions for naive conditional estimates, the ones that simple conditional prediction markets would give:

E[y|R] = ∫_{v>0} y R(y|vx) q(vx) dydvdx /

∫_{v>0} R(y|vx) q(vx) dydvdx

E[y|D] = ∫_{v<0} y D(y|vx) q(vx) dydvdx /

∫_{v<0} D(y|vx) q(vx) dydvdx

Note that while the difference between E[y|R] and E[y|D] does reflect differences between causal effects R(y|vx) and D(y|vx), it can also give a misleading comparison as these expressions integrate over quite different ranges of v.

Consider in contrast outcome estimates conditional on an exactly tied election, where everyone’s vote matters:

E[y|R,v=0] = ∫ y R(y|0x) q(0x) dydx /

∫ R(y|0x) q(0x) dydx

E[y|D,v=0] = ∫ y D(y|0x) q(0x) dydx /

∫ D(y|0x) q(0x) dydx

Since both these expressions integrate over exactly the same range for all parameters, a comparison between these estimates gives a direct comparison between the causal effects R(y|vx) and D(y|vx) of the different parties.

Of course prediction markets may not give meaningful for very unlikely conditions like a tie v=0. A reasonable compromise with practicality would be to condition on close elections, won by e or fewer votes:

E[y|R,|v|<e] = ∫_{v in [0,+e]} y R(y|vx) q(vx) dydvdx /

∫_{v in [0,+e]} R(y|vx) q(vx) dydvdx

E[y|D,|v|<e] = ∫_{v in [-e,0]} y D(y|vx) q(vx) dydvdx /

∫_{v in [-e,0]} D(y|vx) q(vx) dydvdx

If we assume that averages of R(y|vx), D(y|vx) over yx are continuous in v, then these close election estimates must approach the ideal tied election estimates in the limit as the allowed vote margin e goes to zero.

In six of the 57 US presidential elections where the public voted, the election was won by less than 1% of the vote. (They were: Bush II, Nixon, Kennedy, Harrison, Clevenland, Garfield.) So prediction markets on post election outcomes that are conditional both on a particular party winning, and on a 1%-close election, should have a roughly 5% chance of paying off. That seems feasible, at least given sufficient market subsidies.

Analyzing data for 33 countries, Frankel finds that the average upward bias in the official forecast of the budget balance, relative to the realized balance, is 0.2 percent of GDP at the one-year horizon, 0.8 percent at the two-year horizon, and 1.5 percent at the three-year horizon. The longer the horizon, and the more genuine uncertainty there is, the more scope there is for wishful thinking. The bias is not larger for the commodity producers, … or for the developing countries, than for others. …

Over-optimism in predicting growth appears linked to over-optimism in predicting budget balances. On average, the upward bias in growth forecasts is 0.4 percent when looking one year ahead, 1.1 percent at the two-year horizon, and 1.8 percent at three years. The bias in growth forecasting appears in the United States and most other industrialized countries, but not among the commodity producing countries in the sample. …

Over-optimism is more prominent, for both budget balances and for economic growth, during economic booms. …. Countries subject to a budget rule … make official forecasts of growth and budget deficits that are even more biased and more correlated with booms than do other countries. Evidently when such governments exceed the deficit limits set by the rules, they respond by adjusting their forecasts rather than by adjusting their policies …

As a result of budget institutions created in 2000, Chile’s official forecasts of growth and of budget balance have not been overly optimistic, even in booms. (more)

The key institutional innovation [in Chile] is that there are two panels of experts whose job it is each mid-year to make the judgments, respectively, what is the output gap and what is the medium term equilibrium price of copper, rather than leaving the job to government officials. …. A reinforcement of the Chilean idea would be to give the panels legal independence. There could be laws protecting them from being fired, as there are for governors of independent central banks. (more)

Prediction markets forecasting budget balances and growth rates would be easy, and they’d reliably resist political pressure for overly optimistic estimates. So why even bother with trying to figure out how to design expert panels that can remain both expert and independent? Either Frankel naively thinks this easy, he is ignorant of the market solution, or doesn’t really want to promote accurate budget estimates.

Asking participants to predict their future vaccination behavior … substantially increased vaccination rates among patients with high short-term vaccination barriers (who, in the absence of this intervention, have low vaccination acceptance rates). These findings are consistent with past research on temporal construal, which suggests that people asked to think about a future behavior tend to focus its abstract benefits, and disregard concrete barriers that might impede it. (more)

Consider personal prediction markets, which predict what you will do in the future, such as whether you will lose weight, get married, get an A, get promoted, etc. By allowing your associates to participate in such markets, you could let them (anonymously) tell you what they really think about what you will do. Looking often at the predictions of such markets, and asking yourself if those predictions are wrong, could help you to live up to your far ideals about what you should and will do with your life.

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict [government-chosen] major trends and events around the world over the next four years. … [We] will be one of the five teams competing – and we’d like you to consider joining our team as a forecaster.

You may have seen other teams’ appeals as well. Today I can announce that GMU hosts one of the five teams, please join us! Active participants will earn $50 a month, for about two hours of forecasting work. You can sign up here, and start forecasting as soon as you are accepted.

The government sponsor is IARPA (Intelligence Advanced Research Projects Activity), under the ACE (Aggregative Contingent Estimation) program, and our team is DAGGRE (Decomposition-Based Elicitation & Aggregation).

Our approach has three distinctive features, all visible to participants:

1. We use an edit-based interface – a current consensus on all questions is visible to all participants, and any user may change any part. Each edit is scored on whether it moves the consensus closer to or further from the truth.  (This is equivalent to a market-maker-based prediction market).
2. For each question IARPA assigns, we “decompose” it by adding related questions, and letting participants forecast both related questions and how they relate to the assigned questions. For example, users can assume answers to some questions, and then forecast other questions conditional on their assumptions. (This is equivalent to a combinatorial prediction market.)
3. We will sometimes walk users through a special elicitation process that has been shown in field and lab experiments to produce more accurate estimates.

(Items #2,3 might not show for a week or two.) We are eager to see how our approach compares to the other approaches. Come get paid to help us find out!

We decided to see how the latest techniques would stand up to the task of predicting what people will buy. … Over the past four months, we have set four teams the task of trying to predict the sales of each issue of New Scientist, using some of the most promising and innovative approaches available. …

We had each hone their techniques on historic data – sales of New Scientist between 2006 and 2010 in UK stores. We also provided images of all the magazine covers. … The forecasters were free to study any other data they deemed useful. …

Data scientists … looked at the numbers and scratched their heads. … Bollen … wanted to examine the connection between tweets about New Scientist and the magazine’s sales. … Yet none emerged. … [Others] started by identifying and extrapolating long-term trends in our sales. …

Our second entrant – a “prediction market” – didn’t fare much better. These markets date back to work in the 1990s by Robin Hanson … Hanson realised that this “wisdom of the crowd” could be used to forecast other events. … Consensus Point … set up a prediction market involving New Scientist staff. Around 25 of us used an online interface to express how much confidence we had in each edition of the magazine. If we thought a cover had big potential to drive sales, for example, we would buy shares in it. … For this task, as a crowd we did not prove wise. …

A different crowd turned out to have more smarts. … CrowdFlower intern Diyang Tang started by asking workers to rate old covers. … She asked if they would pay $10 – almost twice the actual price – to buy the corresponding issue. The fraction of workers that said yes correlated with historic sales, so she applied this approach in the contest. …

In the last days of the contest, the “Turkers” were battling it out for first place with our final contestant, Sebastian Wernicke, a former bioinformatics statistician, … [who] applied a statistical algorithm to the task. … He ran a pixel-by-pixel analysis of each cover that revealed the distribution of different colours. He also considered the topics, wording and image type. Details of public holidays were thrown into the mix on the assumption that time off may affect reading habits. (more)

I have two points to make here. First, while the article gives no statistics, I’ve been able to see all the forecasts for four of the contestants, and can say that at the 10% level there are no statistically significant accuracy differences between any pair of contestants. They averaged 8.5% to 10.7% error over 17 forecasts.

Second, and more important, this was a test of prediction markets as methods, not as forums:

Methods are ways to do things; forums are ways to pick who decides what to do. … Good forums induce people to find good methods. … To me, prediction markets are mostly interesting as forums, not methods. … Averaging popular opinion may be an interesting method, as is statistical analysis, but comparing these does not evaluate prediction markets as forums. (more)

While the New Scientist article mentions me, this contest seems mostly inspired by James Surowiecki’s “wisdom of crowds” concept. To many, this concept says that an almost mystical wisdom on most any topic can be found merely by averaging the opinions of random folks who’ve hardly considered the subject. Had New Scientist asked me (which they didn’t), I’d have expressed little confidence that averaging top-of-the-head opinions by 25 random distracted staffers would beat the concentrated attention of professional collectors and analysts of data.

In this forecasting situation, the forum question is: what is our best forecast of next week’s sales, given access to the forecasts of the various available experts and their methods? Just because one method barely beat others in a particular contest doesn’t mean we should give it all of our weight in future forecasts. Judgment must be used to weigh the different track records of methods in different past contexts. A prediction market would be a fine forum in which to make such judgements, but it would work best if its participants had access to the forecasts made by those different methods.

The idea that you can gain subtle wisdom just by asking your question to a few dozen random isolated people is not remotely as interesting or useful an idea. Even if you use a prediction market to do the asking.

Over at Cato Unbound, two famous hedgehogs recently replied to Tetlock. John Cochrane argued that no one can do well at the unconditional forecasts that Tetlock studied, but that hedgehogs shine at conditional forecasts, such as GDP change given a big stimulus. Bruce Bueno De Mesquita noted that his hedgehoggy use of game theory is liked by the CIA and by peer review.

Today at Cato Unbound, I note that since Tetlock’s data is hardly universal, that leaves room for counter-claims that he missed important ways in which hedgehogs are more accurate. But I find it disappointing, and also a bit suspicious, that neither Cochrane nor De Mesquita express interest in helping to design better studies, much less in participating in such studies. I note that “it is certainly possible to collect and score accuracy on conditional forecasts”, and conclude:

Research patrons eager to fund hedgehoggy research by folks like Cochrane and De Mesquita show little interest in funding forecasting competitions at the scale required to get public participation by such prestigious folks. So hedgehogs like Cochrane and De Mesquita can continue to claim superior accuracy, with little fear of being proven wrong anytime soon. All of which brings us back to our puzzling disinterest in forecast accuracy, which was the subject of my response.

Corporations and governments spend staggering amounts of money on forecasting, and one might think they would be keenly interested in determining the worth of their purchases and ensuring they are the very best available. But most aren’t. They spend little or nothing analyzing the accuracy of forecasts and not much more on research to develop and compare forecasting methods. Some even persist in using forecasts that are manifestly unreliable. … This widespread lack of curiosity … is a phenomenon worthy of investigation.

My response essay considers this puzzle. The editor summarizes:

Robin Hanson argues that most people aren’t interested in the accuracy of predictions because predictions often aren’t about knowing the future. They are about affiliating with an ideology or signaling one’s authority. … He suggests that one way to make predictions more accurate might be to lift both the social stigma and legal prohibitions against gambling.

Key quotes:

Even if disinterest in forecast accuracy is explained by forecasting being only a minor role for pundits, academics, and managers, might we still hope for reforms to encourage more accuracy? …

Hope … mainly comes from the fact that we pretend to care more about forecast accuracy than we actually seem to care. We don’t need new forecasting methods so much as a new social equilibrium, one that makes forecast hypocrisy more visible to a wider audience, and so shames people into avoiding such hypocrisy. …

It isn’t enough to devise ways to record forecast accuracy—we also need a new matching social respect for such records. Might governments encourage a switch to more respect for forecast accuracy? Yes: by not explicitly discouraging it!

Dubner: So that sounds very logical, very appealing; how realistic is it?

Hanson: Well it depends on there being a set of customers who want this product. So, you know, if prediction markets have an Achilles heel it is certainly the possibility that people don’t really want accurate forecasts.

The gold standard of modern social science is … a “random controlled trial.” … And yet… real-world policy-makers continue to neglect, evade, and actively oppose experimental tests of efficacy. … Tim Harford explains why:

Politicians resist pilot schemes with objective measures of success. … politically inconvenient is the fact that half of the pilot schemes will fail… so the pilot will simply produce stark evidence of that failure. …

This is all a nice example of a theme I’ve been pushing for a while ….

Political agency problems are often a byproduct of voter irrationality. The principals give their agents grossly suboptimal incentives, then complain that the agents fail to carry out their assignments. … Pay-for-performance is a good idea, but the public is too irrational to accept it.

Note that private CEOs are also quite reluctant to run randomized trials of their business ideas. Yes some trials happens in marketing, but firms overall still display a puzzling neglect of randomized trials, and of prediction markets. Both mechanisms offer more accurate info, but at the cost of a high rate of clear public embarrassments – clear evidence showing that you endorsed crap.

Yes firms do implement incentive pay more often, but firms still remain puzzlingly reluctant to correct such incentives for overall trends in the economy or the local industry. Maybe voters are more reluctant than stockholders to discipline their agents, making the private sector more efficient at managing many forms of activity. But in both cases there remains a puzzling reluctance to force leaders to prove their value.

My hypothesis: leaders have status, with which voters and stockholders want to affiliate. While people talk about being offended by leader dominance, they are actually quite eager to submit, and reluctant to risk leader wrath by questioning leader quality. The people’s romance with the state makes them even more reluctant to hold political leaders accountable, so this effect is even worse in politics.