[Researchers] obtained data from the National Science Foundation on the number of researchers per capita in each state, and then randomly selected research papers that contained the phrase “test* the hypothes*”. Those papers were characterized as either confirming (positive result) or rejecting (negative result) the hypothesis. …
In my own experience, "proofs" that something can't be done or is wrong are much less reliable than "proofs" that something can be done or is right. As such, I have little respect for negative papers. As the bar above which a paper is attended to at all by others, I would expect a shift towards positive results. We have limited time and attention, and we may believe learning more things that are true and work is a more valuable use of our neuronal limited calories.
So while we can tell ourselves and our colleagues that there is no such thing as a failed experiment, we rightly strive for ourselves and our colleagues to find areas where our results will be positive rather than negative.
Vladimir,I think you've raised some good points. In my own work, which is not for an academic institution, I have virtually no incentive to publish my failures. I am not evaluated on the basis of how many publications I produce, but rather whether the science/technology that I produce or promote appears to solve some customer's (or potential customer's) need and (especially) if it makes money for my employer. Along those lines, I have an incentive to publish my successes, since I can use them to strengthen my claims to expertise, as well as to raise the status of my company. Maybe, if peer review (as you've pointed out) doesn't always work so well, it is because the peer reviewers are not being asked to put their money where their mouths are! This suggests to me a (Robin Hanson-like?) concept: What if peer reviewers had to invest in some tangible manner in the authors and/or papers that they recommended for publication? Papers with results that stood the test of time (yes, I don't know how this would be judged, but perhaps by positive-citation frequency?) would yield positive returns to peer reviewers who recommended them, but negative returns if a published paper was later found to be mostly-useless drivel.
Mikko, your interpretation also praises backwaters.
Robert, your university is more "competitive" if it requires a higher min # publications for tenure. This feature may well correlate with other places in your state.
In what sense does a research compete against people in their State? I compete to publish articles with everyone in the US (and to a lesser extent, other countries). The weakness of the proxy leads me to suspect the results are driven by an omitted variable correlated with productivity-per-PhD-holder and research style (whether and how to state hypotheses and whether to publish).
Trouble is, this leads to a situation where authors have the incentive to misrepresent their results as a great success by all available means short of outright lying.
My impression is that in many fields of applied science, the prevailing attitude is that a fully open and honest discussion of the limitations and shortcomings of one's work is a self-destructive move, and one is normally expected to put a relentless positive spin on it. I've read lots of papers that advertise their results as a magnificent breakthrough, yet on closer inspection it turns out that their usefulness is dubious or nonexistent because of unrealistic assumptions and practical failures that the authors haven't discussed at all, or at best mentioned in a highly obfuscating manner.
This eventually results in a flood of papers that try to sell bad and useless ideas as good using elaborate propaganda, making advanced bullshit-detection skills necessary to separate those from genuinely good work. From my experience, in at least some fields, the peer review system does a miserable job of filtering in this regard. Thus, making sense of the literature requires laborious dredging through bad ideas couched in spin and propaganda, and lots of resources are wasted in manufacturing such worthless (or worse) publications.
I don't see it as obvious that this system is better than one that would allow authors to publish papers honestly describing and analyzing a disappointing evaluation of some promising-looking idea.
In the applied sciences, the purpose of the research is typically to improve, advance, invent, or otherwise contribute meaningfully to some technique, technology, etc. If one "succeeds," then the result may be worth publishing. If one "fails," then publishing is only of value if one believes that many others are very likely to waste time following the same path and that publication would somehow help them avoid making that mistake. After all, in the applied sciences, one acquires status specifically via one's successes. No one doubts, after all, that any fool can fail! Finally, since most new/untested ideas are bad ones, to publish all the failures would require a huge increase in the number of papers published, while at the same time consume enormous amounts of labor and resources that could instead be spent on pursuing the investigation of new, potentially-promising, ideas.
Formal sciences such as mathematics and TCS do not use statistical hypothesis testing but they do evaluate conjectures, and falsifying a long-standing or intuitively compelling conjecture is generally seen as a notable result.
It would be interesting to know whether the bias against negative results extends to rejections of hypotheses which are known from the literature, as opposed to novel proposals.
Much simpler explanation: when financing is scarce, people tend to write more papers from the same data and results. There may be asymmetry in publishing positive vs. negative results several times.
A very interesting paper. However, one thing I find puzzling is that its data include some areas where typical publications are not framed in terms of hypothesis testing, so I don't understand how they classified those.
In particular, computer science papers are normally either theoretical (and thus, like math papers, don't test any hypotheses), or propose and experimentally evaluate some new idea in software or hardware. In the latter case, however, negative results are, as a general rule, not publishable at all -- to get a paper published, you must present positive experimental results and tout your idea as a great breakthrough. (This of course creates incentives for putting every conceivable spin on the presentation, and often makes it hard to judge papers, since you have to judiciously dissect the spin and self-serving propaganda to see the work's real worth.)
So, I'd be really curious to see those ~19% CS papers that supposedly report negative results according to these authors. It could be that my impressions are wrong and what I've seen is not representative of the field as a whole, but from what I've seen, this figure honestly seems incredible.
In my own experience, "proofs" that something can't be done or is wrong are much less reliable than "proofs" that something can be done or is right. As such, I have little respect for negative papers. As the bar above which a paper is attended to at all by others, I would expect a shift towards positive results. We have limited time and attention, and we may believe learning more things that are true and work is a more valuable use of our neuronal limited calories.
So while we can tell ourselves and our colleagues that there is no such thing as a failed experiment, we rightly strive for ourselves and our colleagues to find areas where our results will be positive rather than negative.
Vladimir,I think you've raised some good points. In my own work, which is not for an academic institution, I have virtually no incentive to publish my failures. I am not evaluated on the basis of how many publications I produce, but rather whether the science/technology that I produce or promote appears to solve some customer's (or potential customer's) need and (especially) if it makes money for my employer. Along those lines, I have an incentive to publish my successes, since I can use them to strengthen my claims to expertise, as well as to raise the status of my company. Maybe, if peer review (as you've pointed out) doesn't always work so well, it is because the peer reviewers are not being asked to put their money where their mouths are! This suggests to me a (Robin Hanson-like?) concept: What if peer reviewers had to invest in some tangible manner in the authors and/or papers that they recommended for publication? Papers with results that stood the test of time (yes, I don't know how this would be judged, but perhaps by positive-citation frequency?) would yield positive returns to peer reviewers who recommended them, but negative returns if a published paper was later found to be mostly-useless drivel.
Mikko, your interpretation also praises backwaters.
Robert, your university is more "competitive" if it requires a higher min # publications for tenure. This feature may well correlate with other places in your state.
In what sense does a research compete against people in their State? I compete to publish articles with everyone in the US (and to a lesser extent, other countries). The weakness of the proxy leads me to suspect the results are driven by an omitted variable correlated with productivity-per-PhD-holder and research style (whether and how to state hypotheses and whether to publish).
Trouble is, this leads to a situation where authors have the incentive to misrepresent their results as a great success by all available means short of outright lying.
My impression is that in many fields of applied science, the prevailing attitude is that a fully open and honest discussion of the limitations and shortcomings of one's work is a self-destructive move, and one is normally expected to put a relentless positive spin on it. I've read lots of papers that advertise their results as a magnificent breakthrough, yet on closer inspection it turns out that their usefulness is dubious or nonexistent because of unrealistic assumptions and practical failures that the authors haven't discussed at all, or at best mentioned in a highly obfuscating manner.
This eventually results in a flood of papers that try to sell bad and useless ideas as good using elaborate propaganda, making advanced bullshit-detection skills necessary to separate those from genuinely good work. From my experience, in at least some fields, the peer review system does a miserable job of filtering in this regard. Thus, making sense of the literature requires laborious dredging through bad ideas couched in spin and propaganda, and lots of resources are wasted in manufacturing such worthless (or worse) publications.
I don't see it as obvious that this system is better than one that would allow authors to publish papers honestly describing and analyzing a disappointing evaluation of some promising-looking idea.
In the applied sciences, the purpose of the research is typically to improve, advance, invent, or otherwise contribute meaningfully to some technique, technology, etc. If one "succeeds," then the result may be worth publishing. If one "fails," then publishing is only of value if one believes that many others are very likely to waste time following the same path and that publication would somehow help them avoid making that mistake. After all, in the applied sciences, one acquires status specifically via one's successes. No one doubts, after all, that any fool can fail! Finally, since most new/untested ideas are bad ones, to publish all the failures would require a huge increase in the number of papers published, while at the same time consume enormous amounts of labor and resources that could instead be spent on pursuing the investigation of new, potentially-promising, ideas.
Formal sciences such as mathematics and TCS do not use statistical hypothesis testing but they do evaluate conjectures, and falsifying a long-standing or intuitively compelling conjecture is generally seen as a notable result.
It would be interesting to know whether the bias against negative results extends to rejections of hypotheses which are known from the literature, as opposed to novel proposals.
Much simpler explanation: when financing is scarce, people tend to write more papers from the same data and results. There may be asymmetry in publishing positive vs. negative results several times.
A very interesting paper. However, one thing I find puzzling is that its data include some areas where typical publications are not framed in terms of hypothesis testing, so I don't understand how they classified those.
In particular, computer science papers are normally either theoretical (and thus, like math papers, don't test any hypotheses), or propose and experimentally evaluate some new idea in software or hardware. In the latter case, however, negative results are, as a general rule, not publishable at all -- to get a paper published, you must present positive experimental results and tout your idea as a great breakthrough. (This of course creates incentives for putting every conceivable spin on the presentation, and often makes it hard to judge papers, since you have to judiciously dissect the spin and self-serving propaganda to see the work's real worth.)
So, I'd be really curious to see those ~19% CS papers that supposedly report negative results according to these authors. It could be that my impressions are wrong and what I've seen is not representative of the field as a whole, but from what I've seen, this figure honestly seems incredible.
Some researchers constructed a "hierarchy of the Sciences" based on publication of negative results.