Comments on: Popular Fields Less Accurate

By: Eliezer Yudkowsky

Eliezer Yudkowsky — Sat, 05 Dec 2009 23:47:01 +0000

Hypothesis: Popular fields attract people who are relatively more interested in status and relatively more inclined to conformity.

Testable prediction: Run some standard conformity experiments on scientists in fields that were more popular or less popular at the time the scientist joined them.

By: George

George — Wed, 29 Jul 2009 08:29:23 +0000

Very interesting article. Perhaps this recent article in BMJ is related: How citation distortions create unfounded authority

By: George Weinberg

George Weinberg — Mon, 27 Jul 2009 20:18:23 +0000

This phenomena helps explain why we need prediction markets for academic topics, and why most academics may not preceive that need.

I think it’s fair to say that any action which has significant consequences will have unanticipated consequences, and that in general the unintended consequences of a restriction on human action will usually overall be negative.

But for prediction markets to work, what is being predicted must be well defined. It’s easy for me to say something like “your idea may well accomplish what it intends, but it will have negative results overall”, but even if I am quite confident that this is true, given that I don’t know what the negative results will be beforehand and can’t prove that the results followed from the policy after, nor can we agree how bad they are, how can we come up with a reasonable bet?

By: CannibalSmith

CannibalSmith — Mon, 27 Jul 2009 14:52:51 +0000

By: q

Mon, 27 Jul 2009 02:25:10 +0000

truth and scientific progress are illiquid and answers often come after a long time, sometimes decades or centuries. how would you separate duration and outcome uncertainties in financing the market?

By: mike

mike — Mon, 27 Jul 2009 01:21:07 +0000

Couldn’t there be a reverse causation, with false results more likely to be interesting and thus more studies (or even possibly, bad results spur studies to disprove them)

By: Douglas Knight

Douglas Knight — Sun, 26 Jul 2009 17:46:36 +0000

I should have said something more like “‘field’ is a misleading description of the experiment”; the article is already using the word “field,” although more for the general theory than for the experiment. It seems likely that proteins cut across the organization of biology, but it’s not obvious that popular proteins are the same as popular fields. It seems plausible that popular fields also have unpopular proteins. Do people who do sloppy work on popular proteins do sloppy work on unpopular proteins? (if the problem is raw attention leading to publication bias, then no; if competition creates fraud, then maybe)

By: Douglas Knight

Douglas Knight — Sun, 26 Jul 2009 17:28:03 +0000

“field” is a misleading description of the article, though it may be a reasonable inference. The range of the paper is proteins. The measure is number of papers published, but that doesn’t explain why why some proteins are popular. I imagine that it’s because they’re expected to be medically relevant, that is, outside attention, or the expectation thereof. But I don’t think the paper addresses the issue of outsiders wanting particular answers, as if the protein were already relevant to a drug. That would produce bias and not just small error bars, but would probably be hard to test in such a large automated study.

By: Requia

Requia — Sun, 26 Jul 2009 15:47:38 +0000

By popular do they mean more papers published in the field or more people outside the field paying attention?

By: Robin Hanson

Robin Hanson — Sun, 26 Jul 2009 13:42:04 +0000

The usual problem is that popular topics give stronger distorting incentives, and prediction markets do very well are resisting trader incentives to manipulate prices.