Planning Fallacy

The Denver International Airport opened 16 months late, at a cost overrun of $2 billion (I've also seen $3.1 billion asserted).  The Eurofighter Typhoon, a joint defense project of several European countries, was delivered 54 months late at a cost of £19 billion instead of £7 billion.  The Sydney Opera House may be the most legendary construction overrun of all time, originally estimated to be completed in 1963 for $7 million, and finally completed in 1973 for $102 million.

Are these isolated disasters brought to our attention by selective availability?  Are they symptoms of bureaucracy or government incentive failures?  Yes, very probably.  But there's also a corresponding cognitive bias, replicated in experiments with individual planners.

Buehler et. al. (1995) asked their students for estimates of when they (the students) thought they would complete their personal academic projects.  Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done.  Would you care to guess how many students finished on or before their estimated 50%, 75%, and 99% probability levels?

• 13% of subjects finished their project by the time they had assigned a 50% probability level;
• 19% finished by the time assigned a 75% probability level;
• and only 45% (less than half!) finished by the time of their 99% probability level.

As Buehler et. al. (2002) wrote, "The results for the 99% probability level are especially striking:  Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students' confidence in their time estimates far exceeded their accomplishments."

More generally, this phenomenon is known as the "planning fallacy".  The planning fallacy is that people think they can plan, ha ha.

A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et. al. (2000), who found that:

• Asking subjects for their predictions based on realistic "best guess" scenarios; or
• Asking subjects for their hoped-for "best case" scenarios...

...produced indistinguishable results.

When people are asked for a "realistic" scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes - the same vision as their "best case".

Reality, it turns out, usually delivers results somewhat worse than the "worst case".

Unlike most cognitive biases, we know a good debiasing heuristic for the planning fallacy.  It won't work for messes on the scale of the Denver International Airport, but it'll work for a lot of personal planning, and even some small-scale organizational stuff.  Just use an "outside view" instead of an "inside view".

People tend to generate their predictions by thinking about the particular, unique features of the task at hand, and constructing a scenario for how they intend to complete the task - which is just what we usually think of as planning.  When you want to get something done, you have to plan out where, when, how; figure out how much time and how much resource is required; visualize the steps from beginning to successful conclusion.  All this is the "inside view", and it doesn't take into account unexpected delays and unforeseen catastrophes.  As we saw before, asking people to visualize the "worst case" still isn't enough to counteract their optimism - they don't visualize enough Murphyness.

The outside view is when you deliberately avoid thinking about the special, unique features of this project, and just ask how long it took to finish broadly similar projects in the past.  This is counterintuitive, since the inside view has so much more detail - there's a temptation to think that a carefully tailored prediction, taking into account all available data, will give better results. 

But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become.  Buehler et. al. (2002) asked an experimental group of subjects to describe highly specific plans for their Christmas shopping - where, when, and how.  On average, this group expected to finish shopping more than a week before Christmas.  Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days.  Both groups finished an average of 3 days before Christmas.

Likewise, Buehler et. al. (2002), reporting on a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline.  They actually finished 1 day before deadline.  Asked when they had previously completed similar tasks, they responded, "1 day before deadline."  This is the power of the outside view over the inside view.

A similar finding is that experienced outsiders, who know less of the details, but who have relevant memory to draw upon, are often much less optimistic and much more accurate than the actual planners and implementers.

So there is a fairly reliable way to fix the planning fallacy, if you're doing something broadly similar to a reference class of previous projects. Just ask how long similar projects have taken in the past, without considering any of the special properties of this project.  Better yet, ask an experienced outsider how long similar projects have taken.

You'll get back an answer that sounds hideously long, and clearly reflects no understanding of the special reasons why this particular task will take less time. This answer is true.  Deal with it.

---

Buehler, R., Griffin, D. and Ross, M. 1994. Exploring the "planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67: 366-381.

Buehler, R., Griffin, D. and Ross, M. 1995. It's about time: Optimistic predictions in work and love. Pp. 1-32 in European Review of Social Psychology, Volume 6, eds. W. Stroebe and M. Hewstone. Chichester: John Wiley & Sons.

Buehler, R., Griffin, D. and Ross, M. 2002. Inside the planning fallacy: The causes and consequences of optimistic time predictions.  Pp. 250-270 in Gilovich, T., Griffin, D. and Kahneman, D. (eds.) Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge, U.K.: Cambridge University Press.

Newby-Clark, I. R., Ross, M., Buehler, R., Koehler, D. J. and Griffin, D. 2000. People focus on optimistic and disregard pessimistic scenarios while predicting their task completion times. Journal of Experimental Psychology: Applied, 6: 171-182.

Comments

[chris cottee]

This reminds me of the "extreme programming" planning technique: you estimate work by chopping it up into tasks and figuring out how long they would take in "ideal days" (that is if everything goes perfectly) . Then you see how many "ideal days" worth of tasks you managed to deliver last month (for instance) and multiply your estimate by the ratio actual days/ideal days.

This method works reasonably well but people can be quite resistant to facing up to the consequences of how long things actually take. It strikes me that this is a pretty good template for de-biasing in general: use frequent feedback of reality to adjust your priors.

[Robin Hanson]

Don't forget that this bias is reinforced by incentives, as managers tend to prefer overconfident planning forecasters to accurate ones.

[Stuart Armstrong]

Very interesting idea, the inside-outside contrast. Is this connected in any way to the fact that outside consultants are often better a spotting problems than those inside the company?

Both groups finished an average of 3 days before Christmas.

Is this true in general? Is there really no difference between time taken on an optimistic schedule and a realistic one? I've found that projects with long dealines tend to "slip" a bit, no matter how long the deadline is.

[Shakespeare's Fool]

In Boston we have heard many reporters claim that
the delays in the Big Dig (the highways tunneled
under the city and the harbor) were increased by
contractors stretching out the work to increase
their incomes. This suggests that there are strong
incentives, particularly in projects paid for
through government, to over promise (underbid)
and under deliver (negotiate higher pay when
work is under way).

Not so much a planning bias as a pocket book bias.

John

[James]

I really like your posts, but with most of them I'm already a believer. But estimating how long it takes me to finish papers is something I'm *really* bad at, by a factor of about 10, yet I never learn. So this post could be a big help.

[Eliezer Yudkowsky]

> Both groups finished an average of 3 days before Christmas.

Is this true in general? Is there really no difference between time taken on an optimistic schedule and a realistic one? I've found that projects with long dealines tend to "slip" a bit, no matter how long the deadline is.

My impression from reading the literature so far is that project estimates are highly informative, that is, students which estimated earlier completion times did tend to finish earlier. However, manipulations which resulted in earlier estimates, such as asking students to visualize their plans in detail, did not affect completion time; nor did manipulations which led to more realistic estimates.

That's my impression from reading so far, but I also have a dim recollection of having heard different results from elsewhere (a case where manipulating estimates did affect completion times).

There may also be a difference between individual and bureaucratic projects, or manipulating an external deadline versus manipulating the estimate.

[James Blair]

I notice in the 1994 study, the students were directly asked for their forecasts. Do any of the studies try to get students to write down their forecasts on an envelope to be opened after they have finished their project, to try to avoid any possible social pressure?

[Laura]

The article fails to take into account actual time spent on the project. Why is it that I can write just as good a paper with a one week or 10 week deadline? Is the problem underestimation of time or lack of motivation to finish before the predicted deadline? I don't think student reports are a very good model for this kind of cognitive bias.

[Jannia]

Interesting. That corresponds remarkably well with the best advice I've ever gotten for estimating software schedules: take your best guess as to when you'll be done, then double it.

[savagehenry]

Having worked on several large software projects I have experienced first hand overly optimistic goals. Just because you think you have a clue about what it is you're supposed to do doesn't mean that when you actually sit down and start working that you know even 1/10th what you need to know. For me personally that is the reason why the projects I've worked took longer than I (and everyone else involved) thought at first. I simply didn't know what I didn't know. It wasn't long before half my work days were spent simply reading and learning about what it was I was trying to do before I actually got around to doing it.

[Venkman]

I was reading this inside view / outside view and before I got to the explanation of what it means, i had a completely different meaning forming in my head. So... here's my idea.

How about estimating based on how others perform? What I mean is, instead of thinking how long you'd take to finish that paper (or whatever), try to thinking how long would it take your classmate, your friend, someone else.

Do you think this could give better estimates?

[Sudeep D'Souza]

Estimates get derailed by "unexpected delays or unforeseen catastrophes". These are nothing but risks. If one can manage the project by managing the risks in the project and keeping it in control I think most tasks can be accomplished within a reasonable estimate.

I have written more on this on my blog http://sudeepdsouza.blogspot.com/2008/03/managing-risk-for-better-estimates.html.