In the latest American Journal of Economics and Sociology, Arthur Diamond presents a very disturbing result: Polywater, one of the most famous mistaken scientific research programs of the past half-century, is used as a case study to examine whether polywater researchers later experienced lower citation counts, or less favorable job mobility. The primary result is that simply writing on polywater, either pro or con, has a negative impact on future citations, in comparison with those who never wrote on polywater. The lifetime value of the lost citations is roughly in the range of $13,000 to $19,000. However writing on polywater did not affect the probability of a scientist leaving university employment.
I'm not entirely convinced that writing against polywater was the cause of anyone's career stagnation.
I would say it is worth considering whether correlation (i.e. writing "con" opinions on polywater correlates with lower citations and earnings later) is really causation in this case. (If this is addressed in the larger study, my apologies - this is just looking at the abstract.)
If you have the time to write papers opposing polywater, doesn't this mean less time devoted to your own research? Could it mean your own research career is stalling? Or that you are more interested in getting your name out there rather than producing meaningful results of your own? If you are too busy with your own pressing research, are you really going to take time out to write against what seems to you to be an obvious scientific dead-end?
Engaging contrarian views may be unproductive. Or it may signal an underlying lack of productivity in the rest of one's work.
This was a good discussion of some issues related to my paper---sorry to add to it way late. Another related example: Stigler is not much remembered for his refuting Berle and Means, even though their book was important before the Stigler refutation. I say a bit more on this at:http://www.artdiamondblog.c...
Michael, flawed is not useless. The fact that contrarian views get ignored means they are not investigated enough; it doesn't mean that academics are wrong on average to hold a low opinion of them.
Zac, yes, I had in mind contrarian views that die out.
My graduate adviser told me to always put something positive in a paper, I presume for this reason. A critical paper, if compelling, ends a line of pursuit and does not generate many citations. A critical paper that is not compelling or otherwise easy to circumvent, is pushed aside and ignored. So you never really win.
Nevertheless, I've written critical papers (as have most other people, it seems). One just tries to squeeze in some interesting ideas or results, which are somewhat independent of the viability of the model being criticized.
Maybe this only holds for contrarian views that eventually die out. For instance, I suspect Bryan Caplan's frequent criticism of the Austrian School has greatly increased his citation count, not diminished it. So perhaps an addendum: academic incentives are to just ignore contrarian claims that you do not believe will become mainstream or develop a vocal heterodox following.
I also agree with Diamond's assessment: if you contribute to a branch of knowledge that turns out to be incorrect, rather than the future avenue of exploration, of course those papers have no reason to be cited. Is there any evidence that writing about polywater reduces the number of citations on your non-polywater articles? That is the stigma effect you seem to be interested in.
Also, this study is looking at a theory that failed ex post, to see the effect on researcher outcomes. We need to do a different analysis if we want to see whether non-mainstream theories are worth debating ex ante, which is what matters for determining our future behavior towards new theories.
Ideally we could classify all papers as falling under mainstream or non-mainstream (at the time of publication), and then look at the outcomes of researchers who respond to each in kind. Obviously some mainstream views become discredited, and some non-mainstream views eventually become the mainstream, so it is not clear that writing about currently-mainstream views is necessarily better.
Given results like this, on what grounds do you go about acting as if sub-disciplinary academic consensus in say, economics, or especially in sociology, is worth the non-paper that it's not written on?
I agree with Diamond's assessment you mentioned last. If you've spent your career refuting polywater, then there's no reason for anyone after that to cite you. Compared to someone whose work builds positively towards progress, they'll have relatively fewer citations. But surely this isn't a reason not to debate the other side - as was mentioned, fewer citations doesn't lead to unemployment.
I'm not entirely convinced that writing against polywater was the cause of anyone's career stagnation.
I would say it is worth considering whether correlation (i.e. writing "con" opinions on polywater correlates with lower citations and earnings later) is really causation in this case. (If this is addressed in the larger study, my apologies - this is just looking at the abstract.)
If you have the time to write papers opposing polywater, doesn't this mean less time devoted to your own research? Could it mean your own research career is stalling? Or that you are more interested in getting your name out there rather than producing meaningful results of your own? If you are too busy with your own pressing research, are you really going to take time out to write against what seems to you to be an obvious scientific dead-end?
Engaging contrarian views may be unproductive. Or it may signal an underlying lack of productivity in the rest of one's work.
Interesting study and fantastic blog.
This was a good discussion of some issues related to my paper---sorry to add to it way late. Another related example: Stigler is not much remembered for his refuting Berle and Means, even though their book was important before the Stigler refutation. I say a bit more on this at:http://www.artdiamondblog.c...
Off topic link, I'm not saying this comic necessarily applies to y'all, just thought you'd like it. It's even by another guy named Robin!
Michael, flawed is not useless. The fact that contrarian views get ignored means they are not investigated enough; it doesn't mean that academics are wrong on average to hold a low opinion of them.
Zac, yes, I had in mind contrarian views that die out.
My graduate adviser told me to always put something positive in a paper, I presume for this reason. A critical paper, if compelling, ends a line of pursuit and does not generate many citations. A critical paper that is not compelling or otherwise easy to circumvent, is pushed aside and ignored. So you never really win.
Nevertheless, I've written critical papers (as have most other people, it seems). One just tries to squeeze in some interesting ideas or results, which are somewhat independent of the viability of the model being criticized.
Maybe this only holds for contrarian views that eventually die out. For instance, I suspect Bryan Caplan's frequent criticism of the Austrian School has greatly increased his citation count, not diminished it. So perhaps an addendum: academic incentives are to just ignore contrarian claims that you do not believe will become mainstream or develop a vocal heterodox following.
I also agree with Diamond's assessment: if you contribute to a branch of knowledge that turns out to be incorrect, rather than the future avenue of exploration, of course those papers have no reason to be cited. Is there any evidence that writing about polywater reduces the number of citations on your non-polywater articles? That is the stigma effect you seem to be interested in.
Also, this study is looking at a theory that failed ex post, to see the effect on researcher outcomes. We need to do a different analysis if we want to see whether non-mainstream theories are worth debating ex ante, which is what matters for determining our future behavior towards new theories.
Ideally we could classify all papers as falling under mainstream or non-mainstream (at the time of publication), and then look at the outcomes of researchers who respond to each in kind. Obviously some mainstream views become discredited, and some non-mainstream views eventually become the mainstream, so it is not clear that writing about currently-mainstream views is necessarily better.
Given results like this, on what grounds do you go about acting as if sub-disciplinary academic consensus in say, economics, or especially in sociology, is worth the non-paper that it's not written on?
I agree with Diamond's assessment you mentioned last. If you've spent your career refuting polywater, then there's no reason for anyone after that to cite you. Compared to someone whose work builds positively towards progress, they'll have relatively fewer citations. But surely this isn't a reason not to debate the other side - as was mentioned, fewer citations doesn't lead to unemployment.