Instead of watching fireworks on July 4, I did 1500 piece jigsaw puzzle of fireworks, my first jigsaw in at least ten years. Several times I had the strong impression that I had carefully eliminated every possible place a piece could go, or every possible piece that could go in a place. I was very tempted to conclude that many pieces were missing, or that the box had extra pieces from another puzzle. This wasn’t impossible – the puzzle was an open box a relative had done before. And the alternative seemed humiliating.
But I allowed a very different part of my mind, using different considerations, to overrule this judgment; so many extra or missing pieces seemed unlikely. And in the end there was only one missing and no extra pieces. I recall a similar experience when I was learning to program. I would carefully check my program and find no errors, and then when my program wouldn’t run I was tempted to suspect compiler or hardware errors. Of course the problem was almost always my fault.
Most, perhaps all, ways to overcome bias seem like this. In the language of Kahneman and Lovallo’s classic ’93 paper, we allow an outside view to overrule an inside view. From their paper:
Two distinct modes of forecasting were applied to the same problem in this incident. The inside view of the problem is the one that all participants adopted. An inside view forecast is generated by focusing on the case at hand, by considering the plan and the obstacles to its completion, by constructing scenarios of future progress, and by extrapolating current trends. The outside view is the one that the curriculum expert was encouraged to adopt. It essentially ignores the details of the case at hand, and involves no attempt at detailed forecasting of the future history of he project. Instead, it focuses on the statistics of a class of cases chosen to be similar in relevant respects to the present one. The case at hand is also compared to other members of the class, in an attempt to assess its position in the distribution of outcomes for the class. …
The inside and outside views draw on different sources of information, and apply different rules to its use. … It should be obvious that when both methods are applied with equal intelligence and skill, the outside view is much more likely to yield a realistic estimate. … it is a serious error to assume the outcomes of the most likely scenarios are also the most likely, and that the outcomes for which no plausible scenarios come to mind are impossible. … It is a conservative approach, which will fail to predict extreme and exceptional events, but will do well with common ones. …
Our main observation, which is psychological: the inside view is overwhelmingly preferred in intuitive forecasting. The natural way to think about a problem is to bring to bear all one knows about it, with special attention to its unique features. The intellectual detour into the statistics of related cases is seldom chosen spontaneously. Indeed, the relevance of the outside view is sometimes explicitly denied: physicians and lawyers often argue against the application of statistical reasoning to particular cases. In these instances, the preference for the inside view almost bears a moral character. The insider view is valued as a serious attempt to come to grips with the complexities of the unique case at hand, and the outside view is rejected for relying on crude analogy from superficially similar instances. …
The contrast between the inside and outside views has been confirmed in systematic research. . … A typical result is that respondents are only correct on about 80% of cases when they describe themselves as "99% sure." People are overconfident in evaluating the accuracy of their beliefs one at a time. It is interesting, however, that there is no evidence of overconfidence bias when respondents are asked after the session to estimate the number of questions for which they picked the correct answer.
If overcoming bias comes down to having an outside view overrule an inside view, then our questions become: what are valid outside views, and what will motivate us to apply them?