Mutual Admiration Societies

Recently I talked at an event for an investment fund that specializes in startups run by high school and college age kids. They told me that they prefer these to be STEM-type startups, as kids with other-type startup ideas tend more often to be crooks. 

I also noticed recently: 

When intellectual A famously gets insight into topic X, it is quite rare for intellectuals interested in topics Y to say "let's try to attract A to study Y!". A might in fact wander into thinking on Y, but rarely because Y folks recruited him/her. An obvious explanation: most intellectuals care less about making progress on their topics, and more about being seen as the source of whatever progress is made.

But then my friend Scott Aaronson told that this does happen in his field of theoretical computer science. Others told me this happens in some areas of math.

These facts suggest to me that intellectual areas do actually vary in how healthy they are, at least in terms of how well people coordinate to promote progress. That is, in some areas people achieve equilibria that better promote progress, while in others they more pay lip service to progress while sitting in equilibria that push other efforts.

This possibility has inspired me to outline a simple model of how intellectual norms could be more or less healthy, at least when peer review matters. But before describing that model, let’s review the basic game theory of peer review. 

Imagine that a set of reviewers decide which authored submissions are approved for valued slots, such as jobs, publications, grants, and reviewers. If a set of reviewers can coordinate and is completely free in its approval choices, then the obvious equilibrium is for a group of reviewers to form a “mutual admiration society” (MAS), which only approves submissions from its members. 

In a simple MAS equilibrium, there’s no need for MAS members to put any effort into their submissions beyond what it takes to identify themselves as MAS members following their equilibrium strategy. Now such a MAS equilibrium might be unstable in the face of a proposed new MAS which retains a majority of the prior MAS members but swaps out some old members for new members. But there’s no point in reviewers approving anyone not part of what they see as a prior or new MAS. 

If outside patrons or customers want an area run by peer review to not devolve into this sort of MAS equilibrium, they need to check that review criteria do not simply reduce to whether submissions are authored by MAS members. The more and stronger are other review criteria, then the more work submission authors will put into looking good according to such criteria, instead of just being MAS members.

Or alternatively, patrons and customers might force one area to compete with another, evaluating each one in terms of some absolute measure of product or progress. This can create incentives for area members to promote area norms that induce better area products or progress.

The intellectual areas where I got most of my early grad-school-plus publications, and thus the areas most responsible for my tenure, were in economic theory. And I quickly learned that such areas are full of substantial MAS type effects. 

A typical research paper in economic theory picks some topic area, makes (very) roughly a dozen specific theoretical assumptions, and then proves roughly two to four results given those assumptions, results intended to be suggestive of econ activity in the paper’s topic area. Such papers also vary in how hard their math looks. 

When such a paper is submitted to a journal, peer reviewers are chosen who have published similar papers before. Reviewers must decide whether or not to recommend the submission, and what reasons to give for this choice. Usually they reject. If reviewers can find a major mistake in the math, they can reject for that reason. But reviewers are typically unwilling to put in the effort to find such mistakes, and most mistakes are minor. 

Sometimes reviewers reject saying that the math looks too easy, which is why authors don’t work as hard as they might to make their math easy to understand. And sometimes reviewers reject saying the topic is boring, which is why authors tend to stay close to reviewer-favored topics. But in economic theory, most rejections complain instead about the paper’s assumptions. 

For each of a paper’s roughly dozen assumption “slots”, there were ~2-4 possible assumptions one could have made at that point in the analysis, to fill that slot. For each result theorem, there were also a similar number of other results one could instead have sought. Thought results are heavily constrained by feasibility, as it is typically just not possible to prove anything re most possible results.

Now consider a number of possible peer review norms that a review community could hold re each such assumption slot. 

• A) One right answer - there is one specific right assumption to make in this slot; papers without it should be rejected.

• B) Any is okay - any assumption is okay in that slot, as long as the paper consistently works out its implications.

• C) Precedent required - an assumption is okay as long as a similar paper has made a similar assumption in that slot. 

• D) Arbitrary is okay - reviewers are given discretion to accept or reject a paper on the basis of that assumption.

Now we finally get to my model. Norms A,B,C might be more or less good at promoting progress, depending on how well the community can identify the better assumptions to make for that slot. But they don’t give much discretion to reviewers, whose publication recommendations are forced by context. So they don’t support MAS type equilibria. 

However norm D, which gives reviewers discretion, does allow reviewers to use MAS type reasons for their choice. After all, there are many ways to identify other MAS members from paper submissions. For example, reviewers can favor the papers of their friends, authors affiliated with allied institutions, papers that favorably cite their own work, and papers whose conclusions match their prior conclusions, or their political priors. 

Note that if a community adopts constraining norms of type A,B,C for ten out of a dozen assumptions slots, and only uses the unconstrained norm D for the last two slots, that still gives reviewers plenty of room to reject papers for MAS reasons. Thus for areas dominated by peer review, outside patrons or customers who want to limit MAS type effects must attend to most all of the evaluation criteria used. Having 95% of criteria allow little discretion, and only 5% allow discretion, still fails if reviewers are free to put enough decision weight on that last 5%. 

Thus we see that depending on the peer review norms adopted in some area, submission authors may be either mostly forced to do good work valued by ouside patrons or customers, or they may be mostly free to follow MAS equilibria where they need only attend to pleasing other MAS members.

And so this is a model of how intellectual areas can vary in their health. Some many actually achieve valued progress and earn the deserved respect of outsiders, while others may devolve into internal political battles of little value to outsiders.

Note while the above analysis has been specific to econ theory papers and norms re their assumptions, this analysis easily generalizes to submissions with many different dimensions, where in some reviewers choices are forced, but in others reviewers have discresion.

Added 15Dec: While I talk about progress in the above, that’s just one example. The core idea is that when a group organized by peer review gets paid by outsiders to do something, the internal peer review standards can be more or less aligned with the outsider priorities. From the outsider point of view, more aligned standards are “healthier”.