Beware the Inside View

Instead of watching fireworks on July 4, I did 1500 piece jigsaw puzzle of fireworks, my first jigsaw in at least ten years.  Several times I had the strong impression that I had carefully eliminated every possible place a piece could go, or every possible piece that could go in a place.  I was very tempted to conclude that many pieces were missing, or that the box had extra pieces from another puzzle.  This wasn’t impossible – the puzzle was an open box a relative had done before.  And the alternative seemed humiliating. 

But I allowed a very different part of my mind, using different considerations, to overrule this judgment; so many extra or missing pieces seemed unlikely.  And in the end there was only one missing and no extra pieces.  I recall a similar experience when I was learning to program.  I would carefully check my program and find no errors, and then when my program wouldn’t run I was tempted to suspect compiler or hardware errors.  Of course the problem was almost always my fault.   

Most, perhaps all, ways to overcome bias seem like this.  In the language of Kahneman and Lovallo’s classic ’93 paper, we allow an outside view to overrule an inside view.  From their paper:


Two distinct modes of forecasting were applied to the same problem in this incident.  The inside view of the problem is the one that all participants adopted.  An inside view forecast is generated by focusing on the case at hand, by considering the plan and the obstacles to its completion, by constructing scenarios of future progress, and by extrapolating current trends.  The outside view is the one that the curriculum expert was encouraged to adopt.   It essentially ignores the details of the case at hand, and involves no attempt at detailed forecasting of the future history of he project.  Instead, it focuses on the statistics of a class of cases chosen to be similar in relevant respects to the present one.  The case at hand is also compared to other members of the class, in an attempt to assess its position in the distribution of outcomes for the class.  …

The inside and outside views draw on different sources of information, and apply different rules to its use.  … It should be obvious that when both methods are applied with equal intelligence and skill, the outside view is much more likely to yield a realistic estimate. … it is a serious error to assume the outcomes of the most likely scenarios are also the most likely, and that the outcomes for which no plausible scenarios come to mind are impossible.  … It is a conservative approach, which will fail to predict extreme and exceptional events, but will do well with common ones.  …

Our main observation, which is psychological: the inside view is overwhelmingly preferred in intuitive forecasting.  The natural way to think about a problem is to bring to bear all one knows about it, with special attention to its unique features.  The intellectual detour into the statistics of related cases is seldom chosen spontaneously. Indeed, the relevance of the outside view is sometimes explicitly denied: physicians and lawyers often argue against the application of statistical reasoning to particular cases.   In these instances, the preference for the inside view almost bears a moral character.  The insider view is valued as a serious attempt to come to grips with the complexities of the unique case at hand, and the outside view is rejected for relying on crude analogy from superficially similar instances.  …

The contrast between the inside and outside views has been confirmed in systematic research.  .  … A typical result is that respondents are only correct on about 80% of cases when they describe themselves as "99% sure."  People are overconfident in evaluating the accuracy of their beliefs one at a time.  It is interesting, however, that there is no evidence of overconfidence bias when respondents are asked after the session to estimate the number of questions for which they picked the correct answer.

If overcoming bias comes down to having an outside view overrule an inside view, then our questions become: what are valid outside views, and what will motivate us to apply them? 

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://profile.typekey.com/andrewgelman/ Andrew

    This reminds me of the “backseat driver principle” (see also here).

  • Floccina

    Sorry to post this hear but…

    Robin I think that you need to edit the wikipedia RAND Health Insurance Experiment entry.
    http://en.wikipedia.org/wiki/RAND_Health_Insurance_Experiment

  • Luke G.

    This post is a little off-topic and lengthy, so bear with me…

    One of the things that has fascinated me about this blog (which I only discovered very recently) is that there seems to be the tacit assumption that bias *can* be overcome. I’m working towards my PhD in English Literature at a large, midlevel state university, and if there is one thing I can firmly say my professors all agree on it is that all we have are biases: there is no inherent, objective truth. My fellow graduate students also affirm this belief—in fact, it is taken as so obvious, so irrefutable that “the search for Truth” has become a standard joke.

    This blanket assumption doesn’t really sit well with me: it always seemed to me that while there could be truths obscured and refracted by prejudice and perception, it didn’t mean that there wasn’t a truth at all. But why, I’ve wondered, are my views so much different than my peers?

    I think perhaps it is my background. Most of my peers leapt straight from undergraduate college to the graduate school, whereas I spent many years in the workforce. I’ve had much the same experience as Robin with computer programming, where I’d strongly suspect the problem would lie with the faulty coding language only to find that 98 percent of the time it was a problem with my input. I also spent some years as journalist, seeking to uncover a truth mired in a swamp of protestation, obfuscations, and denials. I didn’t always get it right, but I never doubted there was a truth to be obtained.

    So, a long preamble to get to this point: I wonder if my kind experiences are what convinces me that truth can exist independently of perception, mindset, or worldview. In other words, does work in fields where results are reproduceable and quantifiable, where a correct answer is indeed attainable therefore pre-dispose people to believing that biases can be overcome—or alternatively, that they can’t?

    And further: is the belief that biases can be overcome itself a bias that should be overcome, or is the belief that biases can’t be overcome the bias that must be overcome? How do you decide?

  • http://gtziralis.googlepages.com George Tziralis

    The same phenomenon in statistics and machine learning, overfitting (http://en.wikipedia.org/wiki/Overfitting), is usually eliminated by ‘cross-validation’. An extension of this remedy could be straightforward.
    However, I think that what the inside-outside view mainly offers is that it highlights the need for an absolute frame of reference, under which we all agree of its outside character. And, like in physics, we unfortunately have to agree on such a frame, before we start describing any system and eliminating its biases.

  • michael vassar

    Luke G: I think that a lot of people, and not just here, would consider your assertion regarding the agreement of your professors to be a terrible indictment of your English Literature PhD program, and possibly of English Literature programs at (at least) large mid-level state universities in general.

    My general take on the subject is that with respect to technical education and with respect to the educational content of elite universities there is at least room for doubt, but with respect to the non-technical content of non-elite university curricula one is safe assuming that it’s simply fraud and priesthood.

    OTOH, I am a bit concerned that I may have a bias towards excess skepticism of accepted beliefs in general and towards the value of educational institutions (and the information value of academic credentials) in particular.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Luke, it is easier for me to believe you misunderstood your teachers than to believe they say such silly things. Can you find that view stated in writing, preferably from a source those people would accept as canonical?

  • billswift

    A similar distinction was made in the “brownlash” literature, especially by Julian Simon. What you are calling an inside view he referred to as “technological forecasting”, and the outside view as “economic forecasting” (if I remember correctly, its been some time since I’ve read them and don;t have the books handy to refer to.)

  • Norman Siebrasse

    Robin, Thanks for raising this distinction between inside / outside views. While K&L focus on the business setting, I think exactly the same problem is rampant in law. Their reference to the aversion to statistical reasoning in law appears to relate to the use of statistics in evidentiary questions, and while they are right on that point, I think the problem is more widespread. In particular, even though law is nominally about rules of general application, there is very strong tendency to want to decide cases on the perceived justice of the facts. The common law is of course premised on this approach, and even when a rule is developed, there is a strong tendency to distinguish it when it doesn’t fit. And it’s not just judges – how many academic articles are titled ‘A Plea for Context…” or “A Contextual Analysis of…” It might be suggested that the inside view is appropriate in the legal context because the legal analysis is ex post, with all the facts at hand, while the business decisions K&L focus on are prospective and so involve risk. But the overconfidence bias K&L cite is equally applicable in ex post decision, as we have excessive confidence in our own ability to correctly come to grips with the all of the details of the particular scenario, as is required by the inside view. Since each case is normally considered in isolation, the inside view offers the possibility of perfect ex post decisions in each case, even though if we consider decisions in the aggregate we know that errors will be common. On the other hand, even though a sound rule (the outside view) may generate the correct result in a large proportion of cases, we know that it ignores potentially relevant features of the case at hand, so the probability of error is salient.

    Whether anything can be done about this bias is another question, K&L conclude by noting “The adoption of an outside view in such cases violates strong intuitions about the relevance of information. Indeed, the deliberate neglect of the features that make the current problem unique can appear irresponsible.” This is certainly true in law, both in judicial decisions and academic commentary.

  • http://www.godofthemachine.com Aaron Haspel

    I also used to blame the hardware and the compiler, until I finally realized that they were never at fault. Had they been at fault at least once in a while, I might still be blaming them today, and I know other programmers who still do.

    An interesting example of inside and outside views comes from fantasy baseball. I was extremely successful at the game, which I attribute largely to watching as little actual baseball as possible, paying attention only to the box scores and the injury reports. My fellow team owners, by contrast, would watch constantly and tinker with their lineups obsessively, benching players who “looked bad” or were “cold” or whom the manager had decided to give a day or two off. Sometimes they would ask me for advice, and I would tell them to stop watching baseball. They thought I was kidding.

    I succeeded, in essence, by refusing to construct an inside view, so I would not be tempted to consult it. Unfortunately I don’t think this strategy generalizes very well.

  • David

    I also blamed the compiler. It’s funny that we all blame the tools first for the blemish. Invariably, I futzed the pointers.

    Re Luke G: It’s very interesting to see the questions that come from someone whose experience hasn’t been reaped from quantitative fields. I wonder what he is looking for when he is looking for truth. I imagine he could agree that it is true that the US has a president, a vice president, a Congress… Moreover, he could agree that (100/538) of the Congress is Senators, (3/538) of the Congress are non-voting Representatives, and the rest are simply Representatives. The point I’m trying to make is the he is using an overloaded term, ‘truth’, when he really wants to talk about the existential question of whether or not there are externalities. ‘Truth’ would be his way of verifying that there are.

    My inside view is that he should take a statistics course or a design of experiments course, to round out is education.

  • savagehenry

    Hah, I don’t think I ever blamed the compiler or hardware. I always blamed the people who made the compiler for not making the error messages clear enough. But that’s probably because my first computer science teacher drilled it into my head that typos are the number one cause of errors followed closely by inconsistency between what you want the program to do and what you actually told it to do.

  • http://profile.typekey.com/nickbostrom/ Nick Bostrom

    Robin: Congratulations on solving the jigsaw puzzle!

    Luke: I’ve found a way to reinterpret the claims you report in a way that makes them less crazy. The idea is that these claims about “truth” are not really about the truth of specific propositions. Instead they are about what we could call the aptness of representations. Consider a group of artists creating drawings of the same model. Suppose that by “truth” we mean “apt and accurate representation”. Then we could say that that there is no one Objective Truth, but instead there may be many different drawings that are all equally “true”. Some depict the model from the side, others from the front; some emphasize certain features, some other features. Some drawings might be worse than others, and have various distortions, yet it would be sophomoric to think that at most one of the artists’ drawings could capture the “real truth” about the subject.

    Now, to make up a link so that this is not completely off-topic: if something like the above interpretation of what is meant by “truth” in English departments is correct, then your example might perhaps be viewed as an illustration of what K&L wrote about. From the inside it looks like what the English professors claim just cannot make any sense; yet the outside view suggests that most of the times when it appears to a student as if a consensus among the professors is completely crazy, it is because the student has a misconception rather than the professors.

    If we could generalize from this one case, we might also suggest that after one has considered the outside view, there is additional insight to be gained by returning to the inside view and try to understand — from the inside — how the conclusion one has reached from the outside could be true and how one must revise one’s inside perceptions so that they conform better to the outside view. The goal, then, would be not simply to give up on the inside view in favor of the outside view, but to use the outside view to test, calibrate, and improve one’s inside perceptions.

  • Nick Tarleton

    From the inside it looks like what the English professors claim just cannot make any sense; yet the outside view suggests that most of the times when it appears to a student as if a consensus among the professors is completely crazy, it is because the student has a misconception rather than the professors.

    Does this imply that apparently ridiculous postmodernist writing should be taken more seriously, rather than shrugged off with a vague reference to the Sokal affair? (I’m sorry if this question sounds loaded; it wasn’t meant that way.)

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Nick B, your hypothesis is plausible, and yes, if one can resist the urge to over-rely on the inside view, it should add info to the outside view.

    Nick T, I have found valuable insights in postmodernist writing.

  • Pingback: Overcoming Bias : AI Progress Estimate

  • Pingback: An Impact-Oriented Knowledge Distribution | Asymmetrical Comparisons

  • Pingback: Having Opinions and Getting them Right | Francisco Contreras - Unabashedly Skeptical

  • Pingback: The Age of Em | Bayesian Investor Blog

  • Pingback: Cuidado com a visão interior | IERFH

  • Pingback: Effective Altruism is not a no-brainer | Compass Rose