Discover more from Overcoming Bias
Three and a half years ago I proposed results-blind journal review:
Consider conclusion-blind review. Authors would write, post, and submit at least two versions of each paper, with opposite conclusions. Only after a paper was accepted would they say which conclusion was real.
On reflection, I’d modify that proposal. I’d add an extra round of peer review. In the first found, all conclusions about signs, amounts, and significance would be blanked out. After a paper had passed the first round, the reviewers would see the full paper. While reviewers might then allow the conclusions to influence their evaluation, they could not as easily hide such bias. Reviewers who rejected on the second round after accepting on the first round would feel pressure to explain what about the actual results, over and above the method, suggested that the paper was poor.
Result Blind Review
Nice thinking, but partial. Especially for the benefit of economists of all academics, I recommend the following funny (even though bitter and cynical) reading:http://www.labspaces.net/vi... as a start of cure against economists' now famous autism.
Are peer reviewed journals relevant anymore?
They were invented in the days before cheap copiers fuhgeddaboud web pages, WYSIWYG, matlab, and excel. The peer reviewed journal system is unchanged from the day I would write a paper longhand on a pad of paper, would redline it (with a red pen) would cut and paste (with scissors and a glue stick), would hand it to a typist, would hand-draw some figures which would be sent to a graphic artist who would do them up with black ink and other tricks of his trade on vellum. The typists copy of the paper might get a 2nd edit, but as you can infer the costs of editing cut the rate of revisions down. The final paper would be sent to a journal where it would be carefully reviewed before the scarce space in the only "broadband" media available to the profession was used on this paper.
At this point any paper can be more effectively published than a journal paper ever was 50 years ago by putting it on the web.
Now publicizing it, drawing in readers, is a different thing. We all have methods for choosing what we sped our time reading. This blog is an important one for me, for instance. And indeed, in my experience as an electrical engineer, journals are replaced by IEEE Xplore, which is the web presence of all IEEE journals. All papers published are on that site. So the reviewers still matter, somewhat. Its not clear what would be lost if IEEE opened it up so all properly formatted papers were "published" in this way.
I think the problem with making the reviewing process HARDER is that it is simply ineffective. I'd imagine in most small science, we all change the details of our studies as we go along. And "randomly" chosen reviewers never have the background to fully appreciate what we are trying to say, while our fans will read our papers irrespective of, and indeed prior to, their review status is determined.