Discussion about this post

User's avatar
Robin Hanson's avatar

Norman, I think the main point is that the paper doesn't offer much evidence one way or the other on the main claim of interest. The likelihood of the data observed seems about the same if author info biases evaluations, and if it improves evaluations. So updating our priors on this evidence just gives us our priors back.

Expand full comment
Overcoming Bias Commenter's avatar

Let's get away from what the paper itself said, and whether Robin was or was not criticizing, and as the underlying question. Robin has proposed a definition of paper quality as resource compensated posterity review. I accept this definition. If our goal is to maximize paper quality, so defined, should referees use the authors' names? I take it that we all accept that the JAMA paper shows that using the authors' names affects acceptance and the dispute is whether the effect increases or decreases quality.

The main objection raised by Robin and Douglas is that the JAMA paper hasn't shown that the effect decreases quality. I don't see why they should have to. As I understand it, there is no data currently which established paper quality as defined by the posterity review standard. This means that one's conclusion will turn on one's prior / null hypothesis / presumption. If we start with the presumption that using the author's name does NOT decrease quality, then it is true, as Robin and Douglas point out, that the JAMA study does not prove the contrary. If we start with the presumption that using the author's name DOES decrease quality, then the JAMA study establishing that there is an effect, and if we combine this with our presumption that the effect is negative, then we can conclude that authors' name should not be used.

Why should our presumption be that using the authors' name reduces quality? I have given theoretical reasons to thing that (1) using the authors' name may reduce reduce quality, and that (2) blind reviewing is likely to be well correlated with quality as defined posterity review. No one has explained why my theoretical argument is flawed on either point. Unless someone can undermine my argument, or at least propose a counter-argument as to why using the authors' name is likely to improve quality, I think that it follows that the correct null hypothesis is that the effect of using the authors' name, if any, is to reduce quality. The JAMA study shows that it does have an effect, thereby eliminating the need for the 'if any' caveat. I think my null hypothesis is also the common sense null hypothesis, which is why the JAMA study apparently didn't make it clear.

Expand full comment
16 more comments...

No posts