Absence of Evidence Is Evidence of Absence

From Robyn Dawes’s Rational Choice in an Uncertain World:

Post-hoc fitting of evidence to hypothesis was involved in a most grievous chapter in United States history: the internment of Japanese-Americans at the beginning of the Second World War.  When California governor Earl Warren testified before a congressional hearing in San Francisco on February 21, 1942, a questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time.  Warren responded, "I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed… I believe we are just being lulled into a false sense of security."

Consider Warren’s argument from a Bayesian perspective.  When we see evidence, hypotheses that assigned a higher likelihood to that evidence, gain probability at the expense of hypotheses that assigned a lower likelihood to the evidence.  This is a phenomenon of relative likelihoods and relative probabilities.  You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.

Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists.  You could argue that a Fifth Column might delay its sabotage.  But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.

Let E stand for the observation of sabotage, H1 for the hypothesis of a Japanese-American Fifth Column, and H2 for the hypothesis that no Fifth Column exists.  Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(E|H1), it cannot be as large as the likelihood that no Fifth Column does no sabotage, the probability P(E|H2).  So observing a lack of sabotage increases the probability that no Fifth Column exists.

A lack of sabotage doesn’t prove that no Fifth Column exists.  Absence of proof is not proof of absence.  In logic, A->B, "A implies B", is not equivalent to ~A->~B, "not-A implies not-B".

But in probability theory, absence of evidence is always evidence of absence.   If E is a binary event and P(H|E) > P(H), "seeing E increases the probability of H"; then P(H|~E) < P(H), "failure to observe E decreases the probability of H".  P(H) is a weighted mix of P(H|E) and P(H|~E), and necessarily lies between the two.  If any of this sounds at all confusing, see An Intuitive Explanation of Bayesian Reasoning.

Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely to produce the signs.  The absence of an observation may be strong evidence of absence or very weak evidence of absence, depending on how likely the cause is to produce the observation.  The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all), is very weak evidence of absence (though it is evidence nonetheless).  This is the fallacy of "gaps in the fossil record" – fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded.  But if there are no positive observations at all, it is time to worry; hence the Fermi Paradox.

Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge.  The strength of a model is not what it can explain, but what it can’t, for only prohibitions constrain anticipation.  If you don’t notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.

GD Star Rating
Trackback URL:
  • Frank Ch. Eigler

    Perhaps this criticism of the California governor assumes an over-naive probabilistic modelling, with only two events (“no acts of espionage” => “fifth column exists [or not]”). In reality, there existed some non-public information about an existing japanese spy network (MAGIC decodes; informants) that is unlikely to have been mentioned in a public hearing.

    Perhaps the reasoning was more like this: “We know that they are already here. We know that some fraction of the population sympathizes with the mother nation. If the fifth column did not exist in an organized form, we might have seen some sabotage already. Since there hasn’t been any, maybe they are holding back for a major strike.”

  • The Bayes Who Wasn’t There

    From an early age, I’ve furrowed my brow at the claim that “Absence of evidence is not evidence of absence.”…

  • Frank: It is impossible for A and ~A to both be evidence for B. If a lack of sabotage is evidence for a fifth column, then an actual sabotage event must be evidence *against* a fifth column. Obviously, had there been an actual instance of sabotage, nobody would have thought that way- they would have used the sabotage as more “evidence” for keeping the Japanese locked up. It’s the Salem witch trials, only in a more modern form- if the woman/Japanese has committed crimes, this is obviously evidence for “guilty”; if they are innocent of any wrongdoing, this too is a proof, for criminals like to appear especially virtuous to gain sympathy.

  • Frank Ch. Eigler

    A and ~A are not each evidence for B, if B is “there is a fifth column active”. In some ways, as I said, they already knew B – it was true. There were questions of degree – how organized? how ready? how many? – for which A and ~A each provide some hints at.

  • Keith Elis

    Earl Warren tumbled headlong into the standard conspiracy theory attractor with, I might add, no deleterious effect on his career. This man was later the 14th Chief Justice of the US Supreme Court and has probably had more lasting effect on US society than any single figure of the 20th century. Thanks for the post.

  • Not sure if this reasoning applies to human factors. People can intentionally deceive. Therefore, reasoning that can be applied to natural phenomenon cannot be used to analyze social or political interactions. Game theory is probably a better theory for social or political interactions.

  • http://en.wikipedia.org/wiki/Japanese_American_internment#Was_the_internment_justified_by_military_necessity.3F

    But that’s not the point. The point is that Earl Warren’s reasoning was invalid. It didn’t matter what other evidence he had (Warren certainly did not know about the ultra-classified MAGIC decodes). The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.

  • Roy Haddad

    I suspect a part of the appeal of this saying comes from a mental unease with conflicting evidence. It is easier to think of the absence of evidence as not evidence at all, rather than as evidence against where the evidence in favor just happens to be much stronger. Perhaps it is a specific case of a general distaste for very small distinctions, especially those close to 0?

    Ad hominem argumentation is another example of evidence which is usually weak, but is still evidence.

  • Michael Sullivan

    The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.

    You are assuming that there are only two types of evidence, sabotage v. no sabotage, but there can be much more differentiation in the actual facts.

    Given Frank’s claim, there is a reasoning model for which your claim is inaccurate. Whether this is the model Earl Warren had in his head is an entirely different question, but here it is:

    We have some weak independent evidence that some fifth column exists giving us a prior probability of >50%. We have good evidence that some japanese americans are disaffected with a prior of 90%+. We believe that a fifth column which is organized will attempt to make a *significant* coordinated sabotage event, possibly holding off on any/all sabotage until said event. We also believe that the disaffected who are here, if there is *no* fifth column would engage is small acts of sabotage on their own with a high probability.

    Therefore, if there are small acts of sabotage that show no large scale organization, this is weak evidence of a lack of a fifth column. If there is a significant sabotage event, this is *strong* evidence of a fifth column. If there is no sabotage at all, this is weak evidence of a fifth column. Not all sabotage is alike, it’s not a binary question.

    Now, this is a nice rationalization after the fact. The question is, if there had been rare small acts of sabotage, what is the likelihood that this would have been taken by Warren and others in power as evidence that there was no fifth column. I submit that it is very unlikely, and your criticism of their actual logic would thus be correct. But we can’t know for certain since they were never presented with that particular problem. And in fact, I wish that you, or someone like you, had been on hand at the hearing to ask the key question: “Precisely what would you consider to be evidence that the fifth column does *not* exist?”

    Of course, whether widespread internment was a reasonable policy, even if the logic they were using were not flawed, is a completely separate question, on which I’d argue that *very* strong evidence should be required to adopt such a severe policy (if we are willing to consider it at all), not merely of a fifth column, but of widespread support for it. It is hard to come up with a plausible set of priors where “no sabotage” could possibly imply a high probability of that situation.

  • Absence of Evidence Is Evidence of Absence.

    Here is a good little article, highlighting an important difference between logical reasoning and collecting evidence. You are probably familiar with the claim that absence…

  • I would agree that the lack of sabotage cannot be argued as support for accepting an increase in the probability of the existence of a fifth column. But it may not be sufficient to lower the probability that there is a fifth column, and certainly may not be sufficient to lower a prior of greater than 50% to below 50%, even assuming that one is a Bayesian.

  • Nick Tarleton

    If sabotage increases the probability, lack of sabotage necessarily decreases the probability.

    What’s special about 50%?

  • Absence ofEvidence

    Eliezer Yudkowsky challenges the notion that the absence of evidence is not the evidence of absence:

    Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely…

  • When you hear someone say “X is not evidence …”, remember that the Bayesian concept of evidence is not the only concept attached to that word. I know my understanding of the word evidence changed as I adopted the Bayesian worldview. My recollection of my prior use of the word is a bit hazy, but it was probably influenced a good deal by beliefs about what a court would admit as evidence.(This is a comment on the title of the post, not on Earl Warren’s rationalization).

  • Eric Falkentein

    I think what is useful about this tendentious bit of logic is that if your prior is sufficiently strong, no evidence necessarily implies you are wrong. Brilliant!

  • Michael Sullivan

    If sabotage increases the probability, lack of sabotage necessarily decreases the probability.

    That’s true in the averages, but different types of sabotage evidence may have different effects on the probability, some negative, some positive. It’s conceivable, though unlikely, for sabotage to on average decrease the probability.

  • DM

    This is all fine and good, but it does not address what “evidence” is. I cannot gather evidence of extra solar planets (either evidence for or against existence) with my naked eyes. So in this experiment, even though I see no “evidence” of extra solar planets by looking up into the sky, I still do not have evidence of absense, because in fact I have no evidence at all.

    Evidence, from the aspect of probability theory, is only meaningful when the experiment is able to differential between existence and absence.

    Then the real question becomes: do we have evidence that our experiment is able to yield evidence? And the only way to prove this to the affirmative, is to find something. You cannot *know* you experiment is designed correctly.