Overcoming Bias

Share this post

Neglecting Conceptual Research

www.overcomingbias.com

Discover more from Overcoming Bias

This is a blog on why we believe and do what we do, why we pretend otherwise, how we might do better, and what our descendants might do, if they don't all die.
Over 11,000 subscribers
Continue reading
Sign in

Neglecting Conceptual Research

Robin Hanson
Mar 12, 2008
Share this post

Neglecting Conceptual Research

www.overcomingbias.com
7
Share

Scott Aaronson and CS theory colleagues complain conceptual insights are slighted relative to technical results:

The trends that worry us are … Assignment of little weight to "conceptual" considerations, while assigning the dominant weight to technical considerations. … by "conceptual" we mean the aspects that can be communicated succinctly, with a minimum amount of technical notation, and yet their content reshapes our view/understanding. Conceptual contributions can be thought of as contents of the work that are most likely to be a part of a scientific hallway discussion. … Once understood, conceptual aspects tend to be viewed as obvious, which actually means that they have become fully incorporated in the worldview of the expert. … our community should be warned of dismissing such contributions by saying "yes, but that’s obvious"; when somebody says such a thing, one should ask "was it obvious to you before reading this article?"

Scott elaborates:

People will often say, "sure, but as soon as you’ve asked the question / defined the model that way, the answer is obvious." They recognize, but don’t sufficiently appreciate, the fact that before the paper in question no one had asked the question or defined the model that way.

Here are a few of the 76 comments.  Travis:

Unfortunately this problem is present in many fields – not just computer science.

Piotr:

Conceptual, notably new model papers, are high risk (and potentially high gain) at the time of evaluation. … Technical papers, notably those solving open problems, are low-risk, and their gain can be easily assessed at eval time. … In a situation involving shrinking resources, a natural (if not a rational) approach is to focus on low-risk entities

Greg Kuperberg:

You can sometimes gain security through obscurity by writing long papers that make your results look hard.

Maleq Khan: 

We got some nice results using a very simple approach … we could not get these results [published]. The most annoying part was the reviewers’ comments, like "this is very simple". Throughout the long journey (i.e., several submissions) of our paper, only one reviewer found our simple approach as "an asset". …  Couple of years earlier … a paper, which achieved much worse bound using a complicated (and inefficient) algorithm and complex proof, got published. 

Most agreed this was a problem, but a few dissented.  Gil Kalai:

The best papers with conceptual breakthroughs were usually also very good in terms of the formalism and other technical aspects.

Yes, the pattern observed is clearly a "bias" relative to the goal of promoting intellectual progress.  But my working model of academia is that it functions mainly to allow folks to affiliate with people certified as impressive – intellectual progress is only a side effect. So reviewers try to to seem clearly impressive to less well-informed observers.  Approving hard solid technical work a reviewer clearly signals his technical abilities, but approving unclear-to-observers conceptual contributions risks both seeming an ignorant lightweight, and seeming a deep thinker.   In general, "certification" tends to be a risk-averse process – it much prefers a high confidence that quality is above a certain minimum, relative to equal chances of very high quality and very low quality. 

I do think there are possible academic institutions that could better reward intellectual progress, but I’m skeptical that people actually want to adopt them, if they make it harder to certify people as impressive.

Share this post

Neglecting Conceptual Research

www.overcomingbias.com
7
Share
7 Comments
Share this discussion

Neglecting Conceptual Research

www.overcomingbias.com
Overcoming Bias Commenter
May 15

Aren't professors certified (with PhDs) by the same institutions which certify students?

Although I think my question was stupid for other reasons. Might a better means of funding knowledge be solely with prizes? Institutions of certification would arise from that sort of arrangement. Of course there would still be the problem of what prizes to pick, but I think that problem is more self-evident than the problem of who to fund, and how much.

Expand full comment
Reply
Share
Overcoming Bias Commenter
May 15

I'd be interested to hear of a counter-example of a field where hard to evaluate conceptual work is over-weighted.

Maybe the "other" complexity theory (i.e., the kind Wolfram does)?

Expand full comment
Reply
Share
5 more comments...
Top
New
Community

No posts

Ready for more?

© 2023 Robin Hanson
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing