Neglecting Conceptual Research

Scott Aaronson and CS theory colleagues complain conceptual insights are slighted relative to technical results:

The trends that worry us are … Assignment of little weight to "conceptual" considerations, while assigning the dominant weight to technical considerations. … by "conceptual" we mean the aspects that can be communicated succinctly, with a minimum amount of technical notation, and yet their content reshapes our view/understanding. Conceptual contributions can be thought of as contents of the work that are most likely to be a part of a scientific hallway discussion. … Once understood, conceptual aspects tend to be viewed as obvious, which actually means that they have become fully incorporated in the worldview of the expert. … our community should be warned of dismissing such contributions by saying "yes, but that’s obvious"; when somebody says such a thing, one should ask "was it obvious to you before reading this article?"

Scott elaborates:

People will often say, "sure, but as soon as you’ve asked the question / defined the model that way, the answer is obvious." They recognize, but don’t sufficiently appreciate, the fact that before the paper in question no one had asked the question or defined the model that way.

Here are a few of the 76 comments.  Travis:


Unfortunately this problem is present in many fields – not just computer science.

Piotr:

Conceptual, notably new model papers, are high risk (and potentially high gain) at the time of evaluation. … Technical papers, notably those solving open problems, are low-risk, and their gain can be easily assessed at eval time. … In a situation involving shrinking resources, a natural (if not a rational) approach is to focus on low-risk entities

Greg Kuperberg:

You can sometimes gain security through obscurity by writing long papers that make your results look hard.

Maleq Khan

We got some nice results using a very simple approach … we could not get these results [published]. The most annoying part was the reviewers’ comments, like "this is very simple". Throughout the long journey (i.e., several submissions) of our paper, only one reviewer found our simple approach as "an asset". …  Couple of years earlier … a paper, which achieved much worse bound using a complicated (and inefficient) algorithm and complex proof, got published. 

Most agreed this was a problem, but a few dissented.  Gil Kalai:

The best papers with conceptual breakthroughs were usually also very good in terms of the formalism and other technical aspects.

Yes, the pattern observed is clearly a "bias" relative to the goal of promoting intellectual progress.  But my working model of academia is that it functions mainly to allow folks to affiliate with people certified as impressive – intellectual progress is only a side effect. So reviewers try to to seem clearly impressive to less well-informed observers.  Approving hard solid technical work a reviewer clearly signals his technical abilities, but approving unclear-to-observers conceptual contributions risks both seeming an ignorant lightweight, and seeming a deep thinker.   In general, "certification" tends to be a risk-averse process – it much prefers a high confidence that quality is above a certain minimum, relative to equal chances of very high quality and very low quality. 

I do think there are possible academic institutions that could better reward intellectual progress, but I’m skeptical that people actually want to adopt them, if they make it harder to certify people as impressive.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • Silas

    You can sometimes gain security through obscurity by writing long papers that make your results look hard.

    We got some nice results using a very simple approach … we could not get these results [published]. …

    So I wasn’t out on a limb when I mentioned it last time?

  • http://yudkowsky.net/ Eliezer Yudkowsky

    I’d say: Just go ahead and write up your ideas on superior institutions first, wait a bit to see what happens, and then moan and groan about how no one really wants to do better.

    I’ve been pleasantly surprised by the absence of backlash from the AI field against my own ideas and the more general intelligence-explosion concept – though it may just be due to sheer absence of notice. We’ll see how that develops, but I was expecting much worse (perhaps based on selective experience of ideas that had been controversial enough for me to be aware of how they had been received).

    Motives are not singleplexes. People don’t want just one thing, with only one part of themselves. Yes, academics want to make things easy on themselves; and they also want, via quite a different sort of wanting, to promote deep conceptual progress.

  • http://www.scottaaronson.com Scott Aaronson

    Maybe I should clarify that I think the bias can go both ways: it’s entirely possible for a field to err in the direction of too much pontificating about big ideas and not enough nitty-gritty technical work. But within CS theory, my feeling is that that’s been the least of our problems.

  • Grant

    Robin,

    Might a solution be to divorce institutions of education from institutions of certification? If organizations devoted solely to certification could do a better job at certifying than universities do today, the demand for pure academic credentials could drop couldn’t it? Sometimes I wonder how much the blogosphere approximates this, although I think its effects are more or less limited to the Internet.

  • http://hanson.gmu.edu Robin Hanson

    Eliezer, does twenty years count yet as having waited “a bit”?

    Scott, I’d be interested to hear of a counter-example of a field where hard to evaluate conceptual work is over-weighted.

    Grant, you have in mind certifying students, I’m talking about certifying the professors.

  • http://www.scottaaronson.com Scott Aaronson

    I’d be interested to hear of a counter-example of a field where hard to evaluate conceptual work is over-weighted.

    Maybe the “other” complexity theory (i.e., the kind Wolfram does)?

  • Grant

    Aren’t professors certified (with PhDs) by the same institutions which certify students?

    Although I think my question was stupid for other reasons. Might a better means of funding knowledge be solely with prizes? Institutions of certification would arise from that sort of arrangement. Of course there would still be the problem of what prizes to pick, but I think that problem is more self-evident than the problem of who to fund, and how much.