Comments on: Publication Bias and the Death Penalty http://www.overcomingbias.com/2007/11/publication-bia.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sat, 11 Feb 2012 23:23:58 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Anonymous http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412182 Anonymous Wed, 28 Nov 2007 04:10:58 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412182 Are you planning on publishing a response to Dezhbakhsh and Rubin's <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1018533" rel="nofollow">response</a> to your paper? Are you planning on publishing a response to Dezhbakhsh and Rubin’s response to your paper?

]]> By: Andrew http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412181 Andrew Mon, 26 Nov 2007 16:16:31 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412181 Just to add another signal: I liked Justin's death penalty paper a lot; <a href="http://www.stat.columbia.edu/~cook/movabletype/archives/2006/01/decision_analys_2.html" rel="nofollow">here</a>. Just to add another signal: I liked Justin’s death penalty paper a lot; here.

]]> By: ScentOfViolets http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412180 ScentOfViolets Tue, 20 Nov 2007 15:16:07 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412180 I'll counter your two words with one of my own: p-values. It's not a tongue-in-cheek argument to say that what seems to be the default value of 0.05 is chosen precisely so that papers can be published in the softer sciences. But by having a minimal discussion of statistical technique, a researcher can submit a fifteen-page article to a peer-reviewed outlet and appear to be generating significant results. What really needs to happen is that every paper should have a discussion about why a particular p-value is chosen (perhaps even a discussion of the counternull value), and more fundamentally, there should be a much more rigorous education in statistics for research. Way too many people interpret 'statistically significant at p=0.05' as 'there is a 95% chance the hypothesis has been confirmed'. I’ll counter your two words with one of my own: p-values. It’s not a tongue-in-cheek argument to say that what seems to be the default value of 0.05 is chosen precisely so that papers can be published in the softer sciences. But by having a minimal discussion of statistical technique, a researcher can submit a fifteen-page article to a peer-reviewed outlet and appear to be generating significant results.

What really needs to happen is that every paper should have a discussion about why a particular p-value is chosen (perhaps even a discussion of the counternull value), and more fundamentally, there should be a much more rigorous education in statistics for research. Way too many people interpret ‘statistically significant at p=0.05′ as ‘there is a 95% chance the hypothesis has been confirmed’.

]]> By: Ty Buller http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412179 Ty Buller Tue, 20 Nov 2007 02:19:32 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412179 Liptak's NY Times article contains the following quote: "The economics studies are, moreover, typically published in peer-reviewed journals, while critiques tend to appear in law reviews edited by students." Without any slight to yours and Professor Donohue's article, I wonder if this characterization isn't a part of the critique of the evidence against the death penalty? Liptak’s NY Times article contains the following quote:

“The economics studies are, moreover, typically published in peer-reviewed journals, while critiques tend to appear in law reviews edited by students.”

Without any slight to yours and Professor Donohue’s article, I wonder if this characterization isn’t a part of the critique of the evidence against the death penalty?

]]> By: g http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412178 g Mon, 19 Nov 2007 23:44:48 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412178 Department of Stating the Obvious: I'd naively have thought that, scientifically or politically speaking, a "negative" result on this question (say, a coefficient betwen -2 and +2 with 95% confidence) should be exactly as interesting as a "positive" one (say, a coefficient between 8 and 12 with 95% confidence). So, assuming that there isn't a consistent political bias in favour of the death penalty among researchers and publishers in this field (which there might be, I guess, but it's not obvious why there should be), it's clearly the magical words "statistically significant" that are biasing the results. Advice to researchers in the field: Exploit prior publication bias! If you get a "negative" result, write it up as "Our results differ significantly (p<0.05 in each case) from those of prior publications such as those of Dezbakhsh and Shepherd [1], Dezbakhsh, Rubin and Shepherd [2], and Mocan and Gittings [3]." (But alas, alas, for the Cult Of Statistical Significance. How much better the world would be if conclusions were expressed as "Our symmetrical 95% confidence interval for the coefficient is ..." or "The likelihood curve for the parameter is shown in Fig. 1". There'd still be publication bias of a sort, in favour of research yielding narrow intervals and sharply peaked curves -- and, I guess, in favour of research where those intervals and peaks are in unexpected places. But it would be much less serious, and somewhat self-correcting.) Department of Stating the Obvious:

I’d naively have thought that, scientifically or politically speaking, a “negative” result on this question (say, a coefficient betwen -2 and +2 with 95% confidence) should be exactly as interesting as a “positive” one (say, a coefficient between 8 and 12 with 95% confidence). So, assuming that there isn’t a consistent political bias in favour of the death penalty among researchers and publishers in this field (which there might be, I guess, but it’s not obvious why there should be), it’s clearly the magical words “statistically significant” that are biasing the results.

Advice to researchers in the field: Exploit prior publication bias! If you get a “negative” result, write it up as “Our results differ significantly (p<0.05 in each case) from those of prior publications such as those of Dezbakhsh and Shepherd [1], Dezbakhsh, Rubin and Shepherd [2], and Mocan and Gittings [3].”

(But alas, alas, for the Cult Of Statistical Significance. How much better the world would be if conclusions were expressed as “Our symmetrical 95% confidence interval for the coefficient is …” or “The likelihood curve for the parameter is shown in Fig. 1″. There’d still be publication bias of a sort, in favour of research yielding narrow intervals and sharply peaked curves — and, I guess, in favour of research where those intervals and peaks are in unexpected places. But it would be much less serious, and somewhat self-correcting.)

]]> By: Eliezer Yudkowsky http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412177 Eliezer Yudkowsky Mon, 19 Nov 2007 18:57:36 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412177 Nice technique! But as a purely editorial note, everything under "You can probably guess what we find" should go under the fold (the "Post Continuation" rather than "Post Introduction"). Nice technique! But as a purely editorial note, everything under “You can probably guess what we find” should go under the fold (the “Post Continuation” rather than “Post Introduction”).

]]> By: Zubon http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412176 Zubon Mon, 19 Nov 2007 18:30:41 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412176 <em>Still need convincing? Download my death penalty data, and run your own regressions.</em> Now <em>that</em> is a strong signal. Still need convincing? Download my death penalty data, and run your own regressions.

Now that is a strong signal.

]]> By: Robin Hanson http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412175 Robin Hanson Mon, 19 Nov 2007 18:17:30 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412175 So why doesn't this sort of check become a standard feature of such publications? Seems easy enough to do. So why doesn’t this sort of check become a standard feature of such publications? Seems easy enough to do.

]]> By: Douglas Knight http://www.overcomingbias.com/2007/11/publication-bia.html#comment-412174 Douglas Knight Mon, 19 Nov 2007 15:27:27 +0000 http://prod.ob.trike.com.au/2007/11/publication-bias-and-the-death-penalty.html#comment-412174 How often do people do this kind of meta-analysis? I've only ever heard of Card-Krueger doing it, but that paper is invisible because of the better known Card-Krueger paper. Does the word "intuition" signal that people in medicine are aware of the problem, but incapable of doing anything about it? How often do people do this kind of meta-analysis? I’ve only ever heard of Card-Krueger doing it, but that paper is invisible because of the better known Card-Krueger paper. Does the word “intuition” signal that people in medicine are aware of the problem, but incapable of doing anything about it?

]]>