The only thing that I am 100% convinced of now, is that certain entities couldn’t care less about the planet, but this could prove to be an effective economic tool against the US and/or capitalism in general.

I suspect that I’m completely misunderstanding you. It seems to me that this kind of study can just as easily find underconfidence, if too many studies fall within the error bars.
It may be more useful to figure out which groups and menthods are overconfident, but it is cheap and I think quite reasonable to assume uniform overconfidence.

Robin Hanson:
Your narrow claim that all studies get published doesn’t eliminate selection bias. In “Cargo Cult Science,” Feynman blames the history of the charge of the electron by a type of selection bias following initial error. He also seems to say that physicists have learned from this mistake, but that seems like a tall order.

Hypothetically, each paper could contain correctly computed confidence intervals. But if certain studies get published by showing an aberrant point estimate calculation, then it will appear as if the published studies have confidence intervals that are too narrow.

(For a similar dynamic, see http://www.slate.com/id/2103486 “Ordinarily, studies with large sample sizes should be more convincing than studies with small sample sizes. Following the fates of 10,000 workers should tell you more than following the fates of 1,000 workers. But with the minimum-wage studies, that wasn’t happening…”)

Indeed, fat tails could explain these results. I remember reading somewhere about how fat-tailed normal distributions being much more common than people think (one family of distributions in particular). I don’t remember the name of this distribution for sure, but it was generalization of the Gaussian, and it was also the limit distribution for its own family (there was an analog of the Central Limit Theorem).

1) Calculating confidence intervals, by itself, throws away an insane amount of information compared to the raw data.

2) You could just as easily say: “Confidence interval XYZ was calculated using statistical model Q, which has a historical surprise index in physics of 7% (Henrion and Fischoff 1986).” Then the same information is transmitted as in the current case, but it comes with an appropriate caution.

Not all methods of measurement are identical, and different groups use different instruments, so the systematic errors made by different groups are different. That means that it is not necessarily the case that all groups are underestimating their errors — in fact, it is most likely that only some of them are underestimating error. Increasing your error based on the “gut feel” that it is not large enough is no more scientific than underestimating it.

You are, in effect, promoting a bias towards overestimation of error. Overestimation bias is also a bias. It is, however, a “silent bias” in that metastudies are less likely to find it because fewer measurements will lie outside each others error bars. Overestimation, however, means that it is harder for an outside observer to determine where the different experiments are clustering, and thus makes it harder to figure out where the most likely figure actually lies! The price for overestimation is loss of valuable information!

Generally speaking, I favor a different solution entirely, which is documenting your calculation, so third parties can see what your assumptions were. A silent assumption is possibly deceptive, but a documented assumption is not, especially since you have to make some assumptions in almost all scientific measurement.

In the old days, when reports had to be published entirely on paper and it was difficult to put all the information you collected into your publication, this was impossible. Now that scientific publications routinely accompany the main article with supplementary data published online, I think we should simply go all the way and provide the reader with as much raw data as possible along with the exact assumptions and methods used to reduce it. Third parties can then know what assumptions you made, and can make their own assessment of what your error bars really mean.