Fillers Neglect Framers
We can usefully divide intellectual tasks into two sets: filling and framing. Fillers add more useful detail or content within some framework, while framers explore possible new frameworks. While both tasks are essential, framing has higher variance; most frame attempts fail, but a few produce great value.
We tend to be best at evaluating people with skills like our skills. So good fillers can best evaluate other fillers, and good framers can best evaluate other framers. You might think then that these groups would separate, so that fillers choose the next generation of fillers and framers choose the next generation of framers.
What I see in the intellectual world around me, however, is very different. When people choose new subordinates and assistants, who will eventually take over their institutional roles in the next generation, I see both fillers and framers choosing fillers. Each person tends to prefer successors who will increase his own fame and reputation by build directly on his own work. The more people who have followed up on your work, the more important you must be.
Since fillers tend to more directly build on the work of others, both fillers and framers tend to choose fillers as successors. For fillers it is a no-brainer; fillers are more like them. For framers there is tension; fillers are less like them, and harder to evaluate. But fillers are more likely to add to the glory of framers’ great new frames, and so framers choose fillers as successors nonetheless.
As a result, our institutions do not do very well at selecting promising framers; while we do a decent job of selecting and encouraging promising fillers from the next generation, successful framers tend to be accidents, people we chose as fillers who turned out to be good framers. Presumably our production of new frames suffers as a result.
Addendum: I think I notice a similar trend in referee reports, which tend to reject framing-style papers, even those with high expected values compared to typical filling-style publications.
I can't quite claim to be in the framer mold, but I left grad school because I saw no value in adding the 10th decimal place to someone else's research. At any rate, I believe there are major institutional problems, at least in humanities, that occlude the creation of bold new ideas. One of the main problems, as you've pointed out, has to do with the patronage and nepotism endemic to the transistion from undergrad to graduate to professorial studies. To my thinking, one proposal might be to prevent members of a department from choosing the entering grad student class. In philosophy for example, I think it terribly quaint to let Kantians cultivate Kantians, Rawlsians Rawlsians and so on. Recommendations, while important, are overrated, largely because they're prone to confirmation bias: professors will tend to like recommendations that exemplify their own perceived virtues. (Well that's best case. Worst case, someone will respect a recommendation because they respect or like or are friends with the recommender.) In general, to foster genuine creativity in graduate school and further, academia needs greater fragmenting, separating research interests from professional power.
It's always much more politically expedient to bash the newest framers and laugh at them. . . Until it turns out that they are right.
The problem is, like other institutions before it, academia has become a sinecure for fillers. Being a framer is bad for your career, and the bigger your frame, the more likely that you will be dismissed by the fillers of the previous frames (who are not at all wont to admit that there might be something amiss with the frames they are busy filling).
The existence of this blog is an encouraging step in the right direction, though. . .