Yet my instinct is to charge ahead and to expand the number of interesting, important questions that academics are encouraged to think systematically about. In particular, if the methodological tools that can be developed in cognitive forensics turn out to be so weak that they become misused on a massive scale, then (I’d expect) norms will develop that discount arguments constructed with these tools, so not much damage will be done.

These often themselves reflect an extreme bias. For example, they trash almost any kind of private funding (e.g. by oil companies on subjects that effect oil companies) but blithely ignore the extreme and endemic conflicts of interest among government-funded scholars who promote the virtues of government generally and government by their kind of expertise in particular.

It’s a great example of bias being more extreme due to it being more centralized and authoritative and thereby even less accountable. A “bias police” is the perfect way to create more extreme and unfixable bias. Extreme bias that we must all accept as normal.

Paul, I think that the pattern of error may be different depending on how it originated. Motivated errors might have distinct signatures that one could detect (and they might also lack some signatures by which one can identify inadvertent error). Thus, they pose a special challenge which we (or somebody) should think about.

My impression of the status quo is that scientists have off-loaded understanding of the use and abuse of statistical methods (eg, overfitting) to the statisticians, and then proceeded to ignore them, the worst of both worlds.

Nick Bostrom: The few examples I know of meta-analyses making accusations of bias are of type (B). They are also politicized. Mainly there are papers about funding bias. Would RH’s meta-analysis contrasting control variables to focal variables be as well-received? Would people bother to write such papers without a preidentified bad guy?

For example, my own field of software development, the most interesting techniques, ideas, and even implementations are researched, discussed, critiqued, and posted to the web, on blogs, email lists, and bulletin boards. Of course not every line of code of every software product is available on the internet, but many of them are (Linux, apache group, many frameworks and components) along with tons of “here’s how you do this” sample code and lots of troubleshooting, all more-or-less easily navigable via google.

I would suspect that subjecting academic research to the same kinds of open environments and discussions would be likely to lead to the kinds of iterative, evolutionary progress in methods and rigor. The biggest barrier to this is probably the stranglehold that the academic press holds over journal articles.

I think I might have given the wrong impression when I wrote about “apprehending the culprits”. I wasn’t actually thinking about some sort of academic police force tasked with combating scientific fraud. Maybe such a thing would be useful; I’m not sure. Yes, scientific fraud does happen, and it’s a serious offense, but my guess is that it’s a small fraction of the problem. The great bulk of the problem is the vast grey zone between fraud and inadvertent error.

I was thinking more about this grey zone, and even there I was thinking primarily about detecting biases on a topic rather than biases of particular individuals or research groups. And the “enforcement” I had in mind is merely the publication of academic critique.

Let me distinguish three things that a cognitive forensics could be or include:

A. Detecting scientific fraud. Developing better tools and methods for this might be useful for journal editors and referees, and could help scientific communities to better police themselves. I’m not proposing any kind of new police force for this, just better tools.

B. Detecting motivated error on some topic, primarily through data mining and statistical analysis. This is what I had in mind. I see a need for better tools for this, and for specialists who are skilled at developing and using these tools. The only enforcement power these specialists would have would be to publish meta-analyses and critiques.

C. Combating bias in general. This is a much more open-ended project, what we are trying to do on this blog. A great many disciplines are needed to do this well. It’s also more of an art than a science. I’m not sure this is yet ready to become a recognized “field” in its own right, although something like that might one day emerge if we can build up a community of people who are interested in these issues. But one could start with something more narrow, such as (A) and (B).