Comments on: Absence of Evidence Is Evidence of Absence http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sun, 12 Feb 2012 04:20:14 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: DM http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416936 DM Mon, 13 Apr 2009 18:37:03 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416936 This is all fine and good, but it does not address what "evidence" is. I cannot gather evidence of extra solar planets (either evidence for or against existence) with my naked eyes. So in this experiment, even though I see no "evidence" of extra solar planets by looking up into the sky, I still do not have evidence of absense, because in fact I have no evidence at all. Evidence, from the aspect of probability theory, is only meaningful when the experiment is able to differential between existence and absence. Then the real question becomes: do we have evidence that our experiment is able to yield evidence? And the only way to prove this to the affirmative, is to find something. You cannot *know* you experiment is designed correctly. This is all fine and good, but it does not address what “evidence” is. I cannot gather evidence of extra solar planets (either evidence for or against existence) with my naked eyes. So in this experiment, even though I see no “evidence” of extra solar planets by looking up into the sky, I still do not have evidence of absense, because in fact I have no evidence at all.

Evidence, from the aspect of probability theory, is only meaningful when the experiment is able to differential between existence and absence.

Then the real question becomes: do we have evidence that our experiment is able to yield evidence? And the only way to prove this to the affirmative, is to find something. You cannot *know* you experiment is designed correctly.

]]> By: Michael Sullivan http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416935 Michael Sullivan Tue, 14 Aug 2007 18:52:10 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416935 <i>If sabotage increases the probability, lack of sabotage necessarily decreases the probability.</i> That's true in the averages, but different types of sabotage evidence may have different effects on the probability, some negative, some positive. It's conceivable, though unlikely, for sabotage to on average decrease the probability. If sabotage increases the probability, lack of sabotage necessarily decreases the probability.

That’s true in the averages, but different types of sabotage evidence may have different effects on the probability, some negative, some positive. It’s conceivable, though unlikely, for sabotage to on average decrease the probability.

]]> By: Eric Falkentein http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416934 Eric Falkentein Tue, 14 Aug 2007 17:52:00 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416934 I think what is useful about this tendentious bit of logic is that if your prior is sufficiently strong, no evidence necessarily implies you are wrong. Brilliant! I think what is useful about this tendentious bit of logic is that if your prior is sufficiently strong, no evidence necessarily implies you are wrong. Brilliant!

]]> By: Peter McCluskey http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416933 Peter McCluskey Tue, 14 Aug 2007 16:15:03 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416933 When you hear someone say "X is not evidence ...", remember that the Bayesian concept of evidence is not the only concept attached to that word. I know my understanding of the word evidence changed as I adopted the Bayesian worldview. My recollection of my prior use of the word is a bit hazy, but it was probably influenced a good deal by beliefs about what a court would admit as evidence.(This is a comment on the title of the post, not on Earl Warren's rationalization). When you hear someone say “X is not evidence …”, remember that the Bayesian concept of evidence is not the only concept attached to that word. I know my understanding of the word evidence changed as I adopted the Bayesian worldview. My recollection of my prior use of the word is a bit hazy, but it was probably influenced a good deal by beliefs about what a court would admit as evidence.(This is a comment on the title of the post, not on Earl Warren’s rationalization).

]]> By: The Everyday Economist http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416937 The Everyday Economist Tue, 14 Aug 2007 14:17:18 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416937 <strong>Absence ofEvidence</strong> Eliezer Yudkowsky challenges the notion that the absence of evidence is not the evidence of absence: Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely... Absence ofEvidence

Eliezer Yudkowsky challenges the notion that the absence of evidence is not the evidence of absence:

Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely…

]]> By: Nick Tarleton http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416932 Nick Tarleton Mon, 13 Aug 2007 23:56:26 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416932 If sabotage increases the probability, lack of sabotage necessarily decreases the probability. What's special about 50%? If sabotage increases the probability, lack of sabotage necessarily decreases the probability.

What’s special about 50%?

]]> By: Barkley Rosser http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416931 Barkley Rosser Mon, 13 Aug 2007 22:06:38 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416931 I would agree that the lack of sabotage cannot be argued as support for accepting an increase in the probability of the existence of a fifth column. But it may not be sufficient to lower the probability that there is a fifth column, and certainly may not be sufficient to lower a prior of greater than 50% to below 50%, even assuming that one is a Bayesian. I would agree that the lack of sabotage cannot be argued as support for accepting an increase in the probability of the existence of a fifth column. But it may not be sufficient to lower the probability that there is a fifth column, and certainly may not be sufficient to lower a prior of greater than 50% to below 50%, even assuming that one is a Bayesian.

]]> By: THE BIZOP NEWS http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416938 THE BIZOP NEWS Mon, 13 Aug 2007 19:20:00 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416938 <strong>Absence of Evidence Is Evidence of Absence.</strong> Here is a good little article, highlighting an important difference between logical reasoning and collecting evidence. You are probably familiar with the claim that absence... Absence of Evidence Is Evidence of Absence.

Here is a good little article, highlighting an important difference between logical reasoning and collecting evidence. You are probably familiar with the claim that absence…

]]> By: Michael Sullivan http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416930 Michael Sullivan Mon, 13 Aug 2007 14:32:09 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416930 <i>The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.</i> You are assuming that there are only two types of evidence, sabotage v. no sabotage, but there can be much more differentiation in the actual facts. Given Frank's claim, there is a reasoning model for which your claim is inaccurate. Whether this is the model Earl Warren had in his head is an entirely different question, but here it is: We have some weak independent evidence that some fifth column exists giving us a prior probability of >50%. We have good evidence that some japanese americans are disaffected with a prior of 90%+. We believe that a fifth column which is organized will attempt to make a *significant* coordinated sabotage event, possibly holding off on any/all sabotage until said event. We also believe that the disaffected who are here, if there is *no* fifth column would engage is small acts of sabotage on their own with a high probability. Therefore, if there are small acts of sabotage that show no large scale organization, this is weak evidence of a lack of a fifth column. If there is a significant sabotage event, this is *strong* evidence of a fifth column. If there is no sabotage at all, this is weak evidence of a fifth column. Not all sabotage is alike, it's not a binary question. Now, this is a nice rationalization after the fact. The question is, if there had been rare small acts of sabotage, what is the likelihood that this would have been taken by Warren and others in power as evidence that there was no fifth column. I submit that it is very unlikely, and your criticism of their actual logic would thus be correct. But we can't know for certain since they were never presented with that particular problem. And in fact, I wish that you, or someone like you, had been on hand at the hearing to ask the key question: "Precisely what would you consider to be evidence that the fifth column does *not* exist?" Of course, whether widespread internment was a reasonable policy, even if the logic they were using were not flawed, is a completely separate question, on which I'd argue that *very* strong evidence should be required to adopt such a severe policy (if we are willing to consider it at all), not merely of a fifth column, but of widespread support for it. It is hard to come up with a plausible set of priors where "no sabotage" could possibly imply a high probability of that situation. The particular observation of no sabotage was evidence against, and could not legitimately be worked into evidence for.

You are assuming that there are only two types of evidence, sabotage v. no sabotage, but there can be much more differentiation in the actual facts.

Given Frank’s claim, there is a reasoning model for which your claim is inaccurate. Whether this is the model Earl Warren had in his head is an entirely different question, but here it is:

We have some weak independent evidence that some fifth column exists giving us a prior probability of >50%. We have good evidence that some japanese americans are disaffected with a prior of 90%+. We believe that a fifth column which is organized will attempt to make a *significant* coordinated sabotage event, possibly holding off on any/all sabotage until said event. We also believe that the disaffected who are here, if there is *no* fifth column would engage is small acts of sabotage on their own with a high probability.

Therefore, if there are small acts of sabotage that show no large scale organization, this is weak evidence of a lack of a fifth column. If there is a significant sabotage event, this is *strong* evidence of a fifth column. If there is no sabotage at all, this is weak evidence of a fifth column. Not all sabotage is alike, it’s not a binary question.

Now, this is a nice rationalization after the fact. The question is, if there had been rare small acts of sabotage, what is the likelihood that this would have been taken by Warren and others in power as evidence that there was no fifth column. I submit that it is very unlikely, and your criticism of their actual logic would thus be correct. But we can’t know for certain since they were never presented with that particular problem. And in fact, I wish that you, or someone like you, had been on hand at the hearing to ask the key question: “Precisely what would you consider to be evidence that the fifth column does *not* exist?”

Of course, whether widespread internment was a reasonable policy, even if the logic they were using were not flawed, is a completely separate question, on which I’d argue that *very* strong evidence should be required to adopt such a severe policy (if we are willing to consider it at all), not merely of a fifth column, but of widespread support for it. It is hard to come up with a plausible set of priors where “no sabotage” could possibly imply a high probability of that situation.

]]> By: Roy Haddad http://www.overcomingbias.com/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416929 Roy Haddad Mon, 13 Aug 2007 06:25:34 +0000 http://prod.ob.trike.com.au/2007/08/absence-of-evidence-is-evidence-of-absence.html#comment-416929 I suspect a part of the appeal of this saying comes from a mental unease with conflicting evidence. It is easier to think of the absence of evidence as not evidence at all, rather than as evidence against where the evidence in favor just happens to be much stronger. Perhaps it is a specific case of a general distaste for very small distinctions, especially those close to 0? Ad hominem argumentation is another example of evidence which is usually weak, but is still evidence. I suspect a part of the appeal of this saying comes from a mental unease with conflicting evidence. It is easier to think of the absence of evidence as not evidence at all, rather than as evidence against where the evidence in favor just happens to be much stronger. Perhaps it is a specific case of a general distaste for very small distinctions, especially those close to 0?

Ad hominem argumentation is another example of evidence which is usually weak, but is still evidence.

]]>