Me last week: Paul Davies, chair of the group that decides what SETI scientists will do if evidence of aliens is ever found, thinks … until scientists can say something to the public with great (~99%) confidence, they should say nothing. … Most early low-probability signs … being false alarms is “damaging to the credibility of science.” So until scientists can confidently say that an asteroid will hit us or that we see aliens, they should just whisper to each other. … One might justify this confidence-or-silence policy by arguing … reporters are biased to present low probability news as if it were high probability.
As all real-money prediction markets suffer from serious problems (lack of interest, very high fees, dubious legality), I only use their play-money versions as an intellectual exercise.
And, finally, I am always skeptical of something that has a very high magnitude of harm and an infinitely small probability of occuring.
Some people try to justify this as well, saying, look at the expected value of loss.
Sort of like Pascals wager: if God exists even with a small probability, then I'd better be good; to which I reply: if there is a big hairy monster out there, and if you eat Tulips on Tuesday, then no harm will befall you from the hairy monster: that is the same argument too, and I do not eat Tulips on Tuesday.
My experience does not include big hairy monsters, but they may exist, and in fact, I have a program you should fund to find out if they do or do not exist.
OK, now I understand that this is limited to the field of space, particularly aliens and catastrophic asteroid crashes.
And, the question is: why treat this differently than other events of a similar nature.
The first question is: Is the premise true--Ask the question: Are there in fact things other than aliens and asteroid crashes where people seek more confirmatory evidence before they speak and excite the public. Yes. Potential nuclear reactor defects, potential underwater oil well leaks [oops, forget about that], tolerance levels for pesticides and combinations thereof. There are probably more, so fill in the blank here.....
Second, what are the consequences of the Fire Drill--will resources be displaced that have greater risk elsewhere, will greater risks be assumed than the expected value of potential risks avoided [what is the probability the risk can be avoided may also be a question]. In other words, is the magnitude of the harm we create in avoiding a low probablility event even worse.
Third, does the Fire Drill involve things that people have a difficult time conceptualizing or analyzing rationally, or, to put it another way, is it the kind of thing I will hear incessantly on Fox News and will devolve into something I will later see on South Park.
One could even take the Mars biology example as well.
Say the claim results in a big hunka funded project based on that rock study and the new study takes away funding for another project, making a cancer drug in a weightless atmosphere.
I think you do have to consider tradeoff, and you also have to consider how the public will amplify sloppy scienctific claims, like it or not, making it difficult to do that which has a higher probability of success.
Shouting Fire when you see what you think is a glimmer of light can cause a stampede.
> Most Research Findings Are False seems pretty clear evidence, as does the high rate of celebrated new medical treatments that are later repudiated, and the very low marginal health-effectiveness of medicine.
Isn't the very low marginal health-effectiveness of medicine explained simply by our willingness to generally spend almost any amount of money on emergent life-extending measures? (This willingness--which is hard to philosophically reconcile with the fact that we put an effective price on life in many other non-emergent situations--might then be explained by some sort of signaling if you wish.) This has always been true; it's just become a problem recently because we've only just acquired the technology which enable us to spend huge sums of money for tiny life extensions. (In the past, there wasn't anything you could do for someone with brain cancer.)
So can we really take the very low marginal health-effectiveness of medicine as evidence "that scientists do not usually insist on such high standards of confidence for publication"?
Here are several examples of adverse consequences of some academic yelling "Fire" in a crowded room.
Most stem from the problem of allocating resources.
Someone yells: I have this improbable cure for cancer. Everyone else drops what they're doing and pursues this avenue of research, depleting the funding of programs that have higher probabilities of success, including the one that is the cure for cancer.
The person yelling "Fire" usually has a conflict as well. They are, afterall, the "Authority", and so they will be hired to conduct the study.
Now, with the proper funding from GMU, and specifically your research budget, I would be happy to discuss this further.
Oh, a relevant difference between this and the medicine ones:
Scientists claiming that chemical X cures disease Y, or that it doesn't, is commonplace, and even if that particular claim is wrong, it won't have much of a status effect on anyone except for the few who published the claim (except in the case when it becomes widely accepted before being discredited).
On the other hand, claiming to find alien life is very much in the domain of crackpots, so false claims of alien life will make scientists look like a bunch of nutters far more than false claims about drug efficiacy.
Khoth, your argument would seem to apply equally well to tentative results in any field, yet the standards are set higher in the alien life field. The point is to explain variations in where the standard is set.
I still don't see it. If alien life is confirmed, the report will come out sooner or later and have all the status effects that a confirmed report of the existence of aliens would have. The case where the silencing makes a difference is when the evidence isn't good and it turns out to be false alarm, and the status effect of that isn't to elevate crackpots, it's to drag scientists down to their level.
Tomasz, intrade is real money. They don't pay interest on deposits, which penalizes long term questions.Khoth, SETI scientists feel pressure from other scientists.Bill, what adverse consequences?Rolf, it is still the sixties in many other fields.
On the subject of the certainty required for publication: In particle physics - which is quite high-status within science - there are rules for using the terms 'evidence' and 'observation'. If you see a signal that is three standard deviations above your background, you can say this is 'evidence for' whatever you're looking for. At five standard observations you can claim 'observation of'. If it's less than three sigma, you can only publish it as a 'search for'. The origin of this rule is in the sixties when people were publishing low-quality results right and left and claiming new particles, basically because they were looking for "Anything interesting in such-and-such a mass range" and when you split a given range into a hundred bins, it's fairly likely you'll see something at two or three sigma. These days it is required that you state beforehand what you are looking for and where you expect to find it.
Intrade has both real money (intrade.com) and play money (intrade.net) versions.
As all real-money prediction markets suffer from serious problems (lack of interest, very high fees, dubious legality), I only use their play-money versions as an intellectual exercise.
And, finally, I am always skeptical of something that has a very high magnitude of harm and an infinitely small probability of occuring.
Some people try to justify this as well, saying, look at the expected value of loss.
Sort of like Pascals wager: if God exists even with a small probability, then I'd better be good; to which I reply: if there is a big hairy monster out there, and if you eat Tulips on Tuesday, then no harm will befall you from the hairy monster: that is the same argument too, and I do not eat Tulips on Tuesday.
My experience does not include big hairy monsters, but they may exist, and in fact, I have a program you should fund to find out if they do or do not exist.
Given the current views on Illegal Immigration in the US, I seriously doubt that aliens and time travelers will be identifying themselves here.
OK, now I understand that this is limited to the field of space, particularly aliens and catastrophic asteroid crashes.
And, the question is: why treat this differently than other events of a similar nature.
The first question is: Is the premise true--Ask the question: Are there in fact things other than aliens and asteroid crashes where people seek more confirmatory evidence before they speak and excite the public. Yes. Potential nuclear reactor defects, potential underwater oil well leaks [oops, forget about that], tolerance levels for pesticides and combinations thereof. There are probably more, so fill in the blank here.....
Second, what are the consequences of the Fire Drill--will resources be displaced that have greater risk elsewhere, will greater risks be assumed than the expected value of potential risks avoided [what is the probability the risk can be avoided may also be a question]. In other words, is the magnitude of the harm we create in avoiding a low probablility event even worse.
Third, does the Fire Drill involve things that people have a difficult time conceptualizing or analyzing rationally, or, to put it another way, is it the kind of thing I will hear incessantly on Fox News and will devolve into something I will later see on South Park.
Bill the issue is why the consequences in this field are different, to justify different standards.
>Rolf, it is still the sixties in many other fields.
Yes; I was holding up my own field as the standard to which others should aspire, not as an exemplar of how all science is done. :)
One could even take the Mars biology example as well.
Say the claim results in a big hunka funded project based on that rock study and the new study takes away funding for another project, making a cancer drug in a weightless atmosphere.
I think you do have to consider tradeoff, and you also have to consider how the public will amplify sloppy scienctific claims, like it or not, making it difficult to do that which has a higher probability of success.
Shouting Fire when you see what you think is a glimmer of light can cause a stampede.
This is tangential, but....
> Most Research Findings Are False seems pretty clear evidence, as does the high rate of celebrated new medical treatments that are later repudiated, and the very low marginal health-effectiveness of medicine.
Isn't the very low marginal health-effectiveness of medicine explained simply by our willingness to generally spend almost any amount of money on emergent life-extending measures? (This willingness--which is hard to philosophically reconcile with the fact that we put an effective price on life in many other non-emergent situations--might then be explained by some sort of signaling if you wish.) This has always been true; it's just become a problem recently because we've only just acquired the technology which enable us to spend huge sums of money for tiny life extensions. (In the past, there wasn't anything you could do for someone with brain cancer.)
So can we really take the very low marginal health-effectiveness of medicine as evidence "that scientists do not usually insist on such high standards of confidence for publication"?
Here are several examples of adverse consequences of some academic yelling "Fire" in a crowded room.
Most stem from the problem of allocating resources.
Someone yells: I have this improbable cure for cancer. Everyone else drops what they're doing and pursues this avenue of research, depleting the funding of programs that have higher probabilities of success, including the one that is the cure for cancer.
The person yelling "Fire" usually has a conflict as well. They are, afterall, the "Authority", and so they will be hired to conduct the study.
Now, with the proper funding from GMU, and specifically your research budget, I would be happy to discuss this further.
Oh, a relevant difference between this and the medicine ones:
Scientists claiming that chemical X cures disease Y, or that it doesn't, is commonplace, and even if that particular claim is wrong, it won't have much of a status effect on anyone except for the few who published the claim (except in the case when it becomes widely accepted before being discredited).
On the other hand, claiming to find alien life is very much in the domain of crackpots, so false claims of alien life will make scientists look like a bunch of nutters far more than false claims about drug efficiacy.
Khoth, your argument would seem to apply equally well to tentative results in any field, yet the standards are set higher in the alien life field. The point is to explain variations in where the standard is set.
I still don't see it. If alien life is confirmed, the report will come out sooner or later and have all the status effects that a confirmed report of the existence of aliens would have. The case where the silencing makes a difference is when the evidence isn't good and it turns out to be false alarm, and the status effect of that isn't to elevate crackpots, it's to drag scientists down to their level.
Adding to my post, see tickers XLif (Extraterrestrial Life by 2050) 59--60 and XLif2 (Intelligent ETs by 2050) 21--22.
There are play-money markets about alien life on Foresite Exchange, but I can't tell whether that site is significantly active.
Tomasz, intrade is real money. They don't pay interest on deposits, which penalizes long term questions.Khoth, SETI scientists feel pressure from other scientists.Bill, what adverse consequences?Rolf, it is still the sixties in many other fields.
On the subject of the certainty required for publication: In particle physics - which is quite high-status within science - there are rules for using the terms 'evidence' and 'observation'. If you see a signal that is three standard deviations above your background, you can say this is 'evidence for' whatever you're looking for. At five standard observations you can claim 'observation of'. If it's less than three sigma, you can only publish it as a 'search for'. The origin of this rule is in the sixties when people were publishing low-quality results right and left and claiming new particles, basically because they were looking for "Anything interesting in such-and-such a mass range" and when you split a given range into a hundred bins, it's fairly likely you'll see something at two or three sigma. These days it is required that you state beforehand what you are looking for and where you expect to find it.