Impressive Power

Monday I attended a conference session on the metrics academics use to rate and rank people, journals, departments, etc.:

Eugene Garfield developed the journal impact factor a half-century ago based on a two-year window of citations. And more recently, Jorge Hirsch invented the h-index to quantify an individual’s productivity based on the distribution of citations over one’s publications. There are also several competing “world university ranking” systems in wide circulation. Most traditional bibliometrics seek to build upon the citation structure of scholarship in the same manner that PageRank uses the link structure of the web as a signal of importance, but new approaches are now seeking to harness usage patterns and social media to assess impact. (agenda; video)

Session speakers discussed such metrics in an engineering mode, listing good features metrics should have, and searching for metrics with many good features. But it occurred to me that we can also discuss metrics in social science mode, i.e., as data to help us distinguish social theories. You see, many different conflicting theories have been offered about the main functions of academia, and about the preferences of academics and their customers, such as students, readers, and funders. And the metrics that various people prefer might help us to distinguish between such theories.

For example, one class of theories posits that academia mainly functions to increase innovation and intellectual progress valued by the larger world, and that academics are well organized and incentivized to serve this function. (Yes such theories may also predict individuals favoring metrics that rate themselves highly, but such effects should wash out as we average widely.) This theory predicts that academics and their customers prefer metrics that are good proxies for this ultimate outcome.

So instead of just measuring the influence of academic work on future academic publications, academics and customers should strongly prefer metrics that also measure wider influence on the media, blogs, business practices, ways of thinking, etc. Relative to other kinds of impact, such metrics should focus especially on relevant innovation and intellectual progress. This theory also predicts that, instead of just crediting the abstract thinkers and writers in an academic project, there are strong preferences for also crediting supporting folks who write computer programs, built required tools, do tedious data collection, give administrative support, manage funding programs, etc.

My preferred theory, in contrast, is that academia mainly functions to let outsiders affiliate with credentialed impressive power. Individual academics show exceptional impressive abstract mental abilities via their academic work, and academic institutions credential individual people and works as impressive in this way, by awarding them prestigious positions and publications. Outsiders gain social status in the wider world via their association with such credentialed-as-impressive folks.

Note that I said “impressive power,” not just impressiveness. This is the new twist that I’m introducing in this post. People clearly want academics to show not just impressive raw abilities, but also to show that they’ve translated such abilities into power over others, especially over other credentialled-as-impressive folks. I think we also see similar preferences regarding music, novels, sports, etc. We want people who make such things to show not only that they have have impressive abilities in musical, writing, athletics, etc., we also want them to show that they have translated such abilities into substantial power to influence competitors, listeners, readers, spectators, etc.

My favored theory predicts that academics will be uninterested in and even hostile to metrics that credit the people who contributed to academic projects without thereby demonstrating exceptional abstract mental abilities. This theory also predicts that while there will be some interest in measuring the impact of academic work outside academia, this interest will be mild relative to measuring impact on other academics, and will focus mostly on influence on other credentialed-as-impressives, such as pundits, musicians, politicians, etc. This theory also predicts little extra interest in measuring impact on innovation and intellectual progress, relative to just measuring a raw ability to change thoughts and behaviors. This is a theory of power, not progress.

Under my preferred theory of academia, innovation and intellectual progress are mainly side-effects, not main functions. They may sometimes be welcome side effects, but they mostly aren’t what the institutions are designed to achieve. Thus proposals that would tend to increase progress, like promoting more inter-disciplinary work, are rejected if they make it substantially harder to credential people as mentally impressive.

You might wonder: why would humans tend to seek signals of the combination of impressive abilities and power over others? Why not signal these things separately? I think this is yet another sign of homo hypocritus. For foragers, directly showing off one’s power is quite illicit, and so foragers had to show power indirectly, with strong plausible deniability. We humans evolved to lust after power and those who wield power, but to pretend our pursuit of power is accidental; we mainly just care about beauty, stories, exciting contests, and intellectual progress. Or so we say.

So does anyone else have different theories of academia, with different predictions about which metrics academics and their customers will prefer? I look forward to the collection of data on who prefers which metrics, to give us sharper tests of these alternative theories of the nature and function of academia. And theories of music, stories, sport, etc.

GD Star Rating
Tagged as: , , , ,
Trackback URL:
  • Douglas Knight

    Have you said anything about this theory of music, stories, and sport?

    • Robin Hanson

      I’ve said a lot about such things signaling ability. This is the first time I’ve discussed signaling the combo of ability and power.

  • Siddharth

    As a corollary, if you wanted the purpose of academia towards generating useful innovations, then you can attempt to change the metrics. For example, there is a strong movement in science and engineering that if you use someone else’s code, you should cite it.

    • Stephen Diamond

      if you wanted the purpose of academia towards generating useful innovations, then you can attempt to change the metrics.

      I don’t think that follows from the analysis.

      What about the institutions providing research grants? Don’t they have their own motives, which might be more aligned with innovation? In other words, the societal function of the institution doesn’t necessarily coincide with the dominant motives of individuals within the institution. Metrics would reflect motives while they might miss the more overriding function, which the grantors supply.

      • Robin Hanson

        Funders are part of the game, not independent rulers. Like other customers, they seek status by affiliating with credentialed impressives.

      • Steve

        The US govt funds a lot of that research. How does it gain status from this, especially since most people dont even know what is funded? Why would would businesses fund for status rather than profits? A lot of people in STEM fields leave academia to start their own businesses? They are doing this to be impressive, and not to try to get rich like most entrepreneurs?


      • Robin Hanson

        US folks feel proud to think that their nation has the world’s best academics. They support politicians who tax to make that happen. Firms also look more impressive, and so get more customers, when they are associated with top academics.

  • Christian Zimmermann

    I conducted a survey on the RePEc blog on whether blog mentions and wikipedia links should be counted like regular citations (link). Both were soundly rejected. The economics profession is not ready for this, but others seem to be, seeing the success of Altmetric, which tracks the discussion of research in social media and has been able to sell its “badge” to several publishers.

  • Pingback: Assorted links

  • KevinH

    I think most likely the motivations of scientist (like myself) are a mixture of a lot of things, probably these two included. I think one other motivation you are missing is the ‘just leave me alone with my toys’ motivation. There metrics would mostly be used to erect a barrier between an in-group and and out-group. It would probably look a lot like the ‘impressive power’ motivated metrics, but with even less emphasis on social media and even broad scientific journals.

  • F.E. Guerra-Pujol (Enrique)

    Robin’s theory appears to be especially true in academic disciplines in which there is no real progress (i.e., law, literature, philosophy), just arguments over competing normative values

    • Robin Hanson

      Most of law, literature, and philosophy are not arguments over competing normative values.

      • F.E. Guerra-Pujol (Enrique)

        What is justice? beauty? truth? Are these not normative questions?

      • Stephen Diamond

        Yes, you could say that, but that doesn’t collapse into disputes about normative values. Should we pursue justice or should we pursue truth instead (where they conflict)? is a dispute about normative values. Is truth correspondence or is it coherence? could be called a “normative question” but it isn’t about values. It’s about the nature of the norms, not their value.

      • F.E. Guerra-Pujol (Enrique)

        Fair enough … but my larger point is that such questions as whether truth is correspondence or coherence (to use your example) cannot be falsified in any meaningful way … and to the extent many academics are engaged in research whose results are not falsifiable, then maybe academia is just about signaling how clever or original one is

  • JasonL

    This theory blends interestingly with my instinct that there is more action around the novel idea than around the idea that drives progress in some marginal but unflashy manner. The freakonomicsification of academia, if you will. Influence within academic circles may be one thing, but influence of The Public Discussion is also food for the beast I’d guess.

  • Stephen Diamond

    Your theory seems to make it something of a mystery that any innovation comes out of academia. The the set of useful thoughts is a miniscule subset of clever thoughts, leaving aside the useful unclever thoughts.

    Do you think the societal usefulness of academia is an illusion?

  • B_For_Bandana

    Robin, before your project is done, you will have changed everything about how society sees academia! I would like to invite you to a fancy cocktail party.

    (That was a very cheap and unworthy joke which anyone could mindlessly apply, with very little modification, to lots of OB posts. Actually I do think this is a cool theory, and intuitively it seems exactly on the money. “Einstein changed the way we think about the universe” is much more typical of pop science than “Einstein greatly improved our models of the universe.” The second one seems too dry and dorky, even though their meanings are the same. The difference? First one emphasizes power).

  • light reading guide

    It would be fairly easy to rank academic subjects by how much credibility or respect non-academics get within the subject field from a defined subset of aficionados (i.e., the 2010s equivalent of “underground” novelists probably outrank, measured by fiction readers, the current equivalent of “underground” mathematicians, who are generally considered cranks or, at best, recreational mathematicians by people who can follow math). This could extend to current or recently (like 40 years ago) non-academic subjects as well, such as cookery, jazz, and pamphleteering i.e. blogging. Thinking about this, I am wondering if the sum total of respect in the world, outside of immediately appreciated individual performance (home or restaurant-cooked food, entertainment, fictions, math puzzle solving), is not a lot lower than I thought it was before reading Robin’s post, (since my gut feeling is few people really actually respect academic successes)

  • Axa

    Innovation as a byproduct of the fight to become dean. A pesimistic view but not unfeasible at all.

  • Stephen Diamond

    A puzzle within the puzzle, based on construal-level theory: Why does academic high status depend primarily on the complexity and elaboration of near-mode thought (if I’m right that it does), when far-mode is associated with greater status in general?

    One obvious possibility is that near-mode is easier to credential reliably. Its near-mode character may account for the fact that this high-status academic operation doesn’t really garner consistently high status among ordinary people, who often see academics as narrow specialists who aren’t truly intelligent.

    If the “abstract thinking” of academics were far-mode, this could supply a compromise version of your theory: a sensitivity to broad societal concerns and a grasp of their essence would be incorporated into the abstract impressiveness the academic must prove. But my impression is that far-mode thinking is usually absent from the precursors to academic prestige because of narrow specialism. (Inter-disciplinary studies would probably introduce more far-mode thinking.)

    On the other hand, the very rise of academia as a status nexus would seem to partly depend on the prestige of far-mode thought. What was impressive intellectual power like in primeval times? I’m thinking it consisted in facility with far-mode thought. These contradictions seem to augur status anxiety among academics.

  • dmytryl

    Suppose you have 2 persons. You know that one won a chess game against a good chess engine on easy, and you know that the other won a chess game against a good chess engine on hard. That’s all you know. Who’d you pick to be your chess coach?

    Same applies when you need to choose an academic. High intelligence is rare, and mere absence of evidence for high intelligence implies lower expected intelligence.

    • Stephen Diamond

      It’s also true that if you gave the two persons an IQ test and all you know are the scores, you would pick the one with the highest. But that doesn’t tell you why you should use the chess game in preference to the IQ test to choose a chess coach. You’d pick the chess game winner because of the relevance of the particular skill demonstrated.

      Hanson’s thesis is that people choose academics based on something closer to an IQ test than to a chess game. This requires explanation; the one he proposes is that the choice is made for reasons of intellectual status rather than practical expedience.

      You could deny the premise, that people go irrationally gaga about powerful high abstract ability, which may part your point. But it’s an empirical question, whereas you’re saying, I think, that Hanson hasn’t specified his theory enough to say what would count against it; that ordinary expediency would explain the same facts. If that’s what you’re saying, I think you’ve incompletely understood Hanson’s thesis.

      • dmytryl

        > Hanson’s thesis is that people choose academics based on something closer to an IQ test than to a chess game.

        I see nothing of that kind, except maybe in the fields where there’s *no game* to begin with, no testing or anything. I assumed that his point was that the achievements deemed impressive are the ones requiring high intelligence rather than the ones supposedly advancing the field the most (if there’s even a distinction between the two).

      • Stephen Diamond

        I find the impressionistic evidence for Hanson’s theory more persuasive than you do, but I’m wary for two specific reasons:

        1. The idea that impressive intellectual credentials award high status based on the power drive seems a bit too much like a nerd’s wet dream.

        2. Speaking of which, if well-credentialed academics are experienced as being powerful, why aren’t people’s erotic responses to academics consistent with this.

      • dmytryl

        But what would be the IQ like metrics by which academics are ranked? In the fields I am familiar with (physics, neurobiology), there’s literally nothing resembling “status by IQ test” what so ever. Status requires intelligence in those fields, but that’s because actual advances require intelligence – if you are not of exceptionally high intelligence you just won’t do anything important before someone else will. In economics, well, the game is presumably making the most money, and people who win at the game are way more powerful than people with citations or anything of that kind.

      • Stephen Diamond

        I think you have a more favorable idea of universities than many people, even in the selected fields. The contrary claim is that academicians are overspecialized, research projects often trivial, and programs subject to intellectual fads. In physics, there’s at least a camp that claims that string theory has been oversubsidized and is more responsive to the mathematical interests of investigators than it is to its subject matter. Even if it’s a blind alley, string theory would remain, wouldn’t it, an excellent way to signal intellect. Advances require high intellect but that doesn’t mean that advances are the most efficient way to signal intellect.

      • dmytryl

        Well, no one gets a lot of status from working on string theory, nowhere close to the status you’d get from actual working predictive ToE. Yes, you do get some status from something like work on string theory, as you very well should as such work demonstrates capability to do work, thus demonstrating that you are worth funding or worth listening to. I’m thinking people with less favourable view are bitter for their own lack of status which is a product of lack of achievement.

      • VV

        Specialization is a necessity. If you are to make any significant improvement, you have to spend at least a few years studing in a particular subfield, therefore repeatedly changing subfields and fields is not an efficient use of your time.

        While it is true that some projects may be trivial or faddish, or even some research fields may be pathological (e.g. cold fusion, maybe string theory or even hot fusion), academia overall (or at least science and technology departments) do generate significant innovation that would not happen otherwise.

      • the dabbler

        To gain a deep understanding of a mature science you have to invest thousands of hours in diligent study. At the end of it all you have a corpus of knowledge that’s utterly useless to outsiders, a group which includes nearly every single person in the world. Unless you translated your erudition into lots and lots of money/prestige, or lots and lots of experiences that are story-friendly, most people will think, rightly or wrongly, that you made a stupid investment.

        Dilettantes are not jealous of specialists. That’s wishful thinking.

  • Pingback: Edu-Links | Education 2027

  • Pingback: Tiwesdæg: the Left-hand of Linkage » Duck of Minerva

  • Pingback: Overcoming Bias : Beware Celebrity Academia

  • Pingback: Overcoming Bias : Math: Over-Used, But Useful