59 Comments
User's avatar
Sarah Constantin's avatar

I don't think the problem with any of the failed approaches you mentioned was the nature of the approach.

And I don't think your current proposed alternative is any likelier to succeed.

I think you're missing some intuitions about momentum and marketing. ANY approach, to solving ANY problem, will only get "off the ground" in a real way if resources are devoted to hammering home the message, repeating it, simplifying it, promoting it, over and over and over and over and over again, from multiple voices, gradually climbing the prestige ladder. That is what it takes to make something "a thing".

Then, when you have a Thing, it is time to evaluate whether the approach is fundamentally or structurally flawed, whether it is incentive-compatible, etc etc. And flawed Things tend to fail.

But almost none of your ideas are even Things. The only one that has gained momentum is prediction markets, and that took *decades* and the rise of a true prediction-market "community" and sustained funding and multiple companies and so on. And, as you often point out, the implementation of these successful prediction markets is often flawed and doesn't follow the details you've written up earlier. THAT IS HOW IT GOES. it takes 30 years for a two-word phrase to gradually become "mainstream" enough that people in a medium-sized subculture will have heard of it.

"Science reform" is a Thing today, just not a successful thing.

Specific science-reform policies mostly aren't big enough to be Things.

Now, you don't just want an idea to be "a Thing", you want it to be strong enough to *actually win*. That is extremely ambitious. I would like to believe that this is possible but to be honest I have little faith that humans can actually set out to do something on the scale of "make academia accountable" and have the overall arc play out in a way similar to what they intend. But you are unusually good at gaming out the incentive-compatibility of ideas. Perhaps, if any of them became Things, they would then succeed. But they're *not.*

I don't feel, reading this, that there's enough "oomf" in historical ranking that it would connect to forward-looking reforms. This is policy; you are trying to get people on board with changing their behavior; it's not clear to me how developing any new information source would do so, if you don't think information moves people in general...

It's too abstract. You're not *pitching* me. You're not reassuring me that it would totally work, that People Are Making It Happen, that all the what-if uncertainties that come to mind can be dispelled and we are marching along the royal road to Totally Winning Forever. You are not doing *even enough of this to convince me to spend a tiny amount of my own cash on the idea*, let alone connecting with anybody more mainstream.

Now ^ is a confident and un-caveated version of my intuition; it could be that you regularly move in circles where the emotional gap doesn't matter.

But if you suspect I'm right I think you really need to talk to some kind of coach on marketing and really *learn* how to do emotional appeal.

Expand full comment
Robin Hanson's avatar

I agree that these aren't close to being "things" yet. It is also plausible that someone with better rhetorical skills, or elite connections, than I have might have a better shot at pushing some of them closer. But seems unlikely that I could acquire either of those at a low enough cost to make that a better strategy persuade some who already have these things via my writings and talks. Not clear what other realistic options I have.

Expand full comment
Sarah Constantin's avatar

here's a suggestion:

1. Find out which of your ideas (of the ones you still endorse) is the *closest* to being able to motivate someone with better rhetorical skills or elite connections.

2. then focus on trying to get that idea to someone who can "run with it".

this will, I predict, force you to do at least a *little* workshopping of how you present your ideas, filling in the (perceived) gaps, in order to get even one person one notch more "mainstream" than you to take one nontrivial action towards pursuing your ideas in practice (like writing a grant, or creating a more rhetorically compelling presentation of the idea).

Expand full comment
Robin Hanson's avatar

There are several startups now pursuing my futarchy concept, which I am helping. So investors have already been sufficiently inspired there.

Expand full comment
Robin Hanson's avatar

"You're not reassuring me that it would totally work, that People Are Making It Happen, that all the what-if uncertainties that come to mind can be dispelled and we are marching along the royal road to Totally Winning Forever". You mean, I'm not lying.

Expand full comment
Sarah Constantin's avatar

I'm exaggerating to give you the idea of what it feels like to be effectively pitched; I'm not advocating you literally tell lies.

Expand full comment
Robin Hanson's avatar

It should feel like I'm lying?

Expand full comment
Sarah Constantin's avatar

It should feel *safe.*

I'm specifically thinking of a pitch deck I saw for a genuinely technically impressive company. It included a key fact about their approach that made it instantly clear to me "oh, this has a better chance of success than the average thing", and they had plausible answers for all my points of skepticism. They pulled me along a narrative path towards the good outcome. In principle, *something* could go wrong with their plan, of course, but right now I can't imagine what it might be.

Expand full comment
JamesLeng's avatar

If you treat successful persuasion as conflicting with intellectual integrity on a zero-sum sliding scale, you're gonna have problems conquering much of anything.

Expand full comment
Jared Gordon's avatar

It's not a solution in all areas, but for experimental academic research it seems like improving replication norms would help a lot. And that's something that's already the norm in most physical sciences, to at least some degree.

One way to strongly enforce replication is to require any new finding to also have a replication study performed independently before it was published, with the replication study published at the same time. However, this could significantly slow down research in some areas, and I'm not sure that's a trade off we should make.

Expand full comment
Jack's avatar

I think what bears remembering is that status markers, above all else, have to be simple. A nonspecialist needs to be able to apply it INSTANTLY. Successful examples include: Money, prizes won, titles, degrees conferred, GPAs, test scores, institution names, number of publications, number of citations. All trivial for a nonspecialist to evaluate instantly.

I don't know much about economics, but I have a strong hunch that people who win the Nobel Memorial Prize in Economic Sciences are probably very good economists. Now that may not be a *perfect* heuristic, but whatever inaccuracy it has is more than made up for by its simplicity. Likewise we use GPAs and institutions and test scores to evaluate job applicants, and while we all know those aren't perfect signals, sometimes we need heuristics to help us make decisions. When you're staring at a pile of 1000 resumes you can't dig into the nuances of every person's life story.

So the problem you have is packaging. If you have a panel of historians or a prediction market, how do you distill down to a simple heuristic that an idiot like me can apply? Something like LEED certification (platinum/gold/silver/bronze)? The marketing here is at least as hard to get right as the details behind it.

Expand full comment
Lawrence's avatar

I don't think academia is reformable until it loses its prestige. The best hope that I can see that private sector advances in things like AI or space exploration, alongside increased insularity in academia causing it to focus on less and less relevant things causes people to stop paying attention to it, and then funding to dry up. After academia is humiliated in that way it could be built back up again to the heights of prestige, with some hard earned lessons to make its future incarnation more robust.

Expand full comment
Aaron Weiss's avatar

To hurry it along, compete with it on functional terms

Expand full comment
Russ Abbott's avatar

Your post was about research and ignored teaching entirely--even though teaching is presumably one of the more important jobs that academia does for society.

Expand full comment
Robin Hanson's avatar

Yes my proposal was directed at reforming research. Futures markets in personal outcomes could be used to reform teaching, but alas that wouldn't be prestigious now.

Expand full comment
Aaron Weiss's avatar

People notice when certain institutions have higher/lower quality of alumni.

You don't need to get prestige, which is hard, you need to cut away at the prestigious institution you want dismembered until there's nothing left but prestige.

Then it withers

Expand full comment
Stefano's avatar

I think the problem with teaching is that no one really cares about it.

(At least in Europe) it doesn’t carry any weight in an academic career, especially compared to research, published papers, and winning grants.

So once again, it all comes down to a matter of prestige.

Expand full comment
Aaron Weiss's avatar

This is very wrong.

Teaching confers status of the teacher to the students and raises the teacher above the students.

Professor of X at Yale is impressive.

Going to Harvard is impressive

Expand full comment
Mike's avatar

Why not go back to prizes?

Expand full comment
JamesLeng's avatar

Yeah, advantage there is it doesn't strictly require overturning much anything that already exists - just start posting bounties and wait for somebody with the right skills to get hungry enough. Gates foundation already does some of that, I think?

Expand full comment
Aaron Weiss's avatar

It feels like you haven't addressed the conflation between undergrads, grads, and academia.

Currently, one of the largest ways academia leverages prestige is through their undergrad (and to a lesser degree graduate) programs.

Business owners are looking for a credential for a mix of

-can do as told

-has knowledge/skills

-intelligence

-prestige

Prestige is least important for standard employment, and an institution will lose its prestige if its graduates are not competitive for long enough.

Ivy league generally wins on all of these.

Outcompeting Universities with high quality colleges on the undergraduate programs can rip away a chunk of their leverage.

Expand full comment
Jay McCarthy's avatar

I think your three possible responses of academics (A, B, and C) misses a plausible defense: conquering the future ranking itself just like they have conquered the current prestige and status system.

I think that you think it is going to be easy or at least objective to compute these rankings in the future, but I think it will be extremely political and controlled by the status regime.

As an example, take Tyler’s GOAT book: does he exalt a forgotten economist that would not have been recognized before? How different are his rankings from what contemporaries would have thought? Does this mean that he is bad at ranking? I suspect most of your readers would assume they’d be aligned with him, but his ranking doesn’t strike me as controversial or much better than the status quo status regime. To make the problem harder, how could Tyler rank two nobodies, like Milton Friedman’s seventh PhD student and Mankiw’s ninth, both of which published papers and then went on to be nobodies? Presumably these people need to be on the ranking in 100 years for the current prediction market to make a tenure/promotion decision.

My opinion is that Sauron knows he will be able to beat any power that goes against him because he will twist it to his ends. You have argued that peer review is example of the corruption of academia, but ask any man on the street and they will tell you that peer review is good… and they will believe it! Oh woe to them that call sweet bitter and bitter sweet, that put light for darkness and darkness for light.

My taste is that academia (meaning government backed education, research, and prestige) needs 100% abolishment and liquidation, as well as salting the earth so Carthage can never be rebuilt.

Academia delendo est

Expand full comment
Dain Fitzgerald's avatar

That AI art could be much better

Expand full comment
Robin Hanson's avatar

Give me a better one and I'll use it.

Expand full comment
smopecakes's avatar

I think that the peak time for this idea may be in 10 or 20 years if academia has failed to reclaim respect and people start to shift from critique to solutions, thus this is a good time to introduce it

Expand full comment
Robin Hanson's avatar

Academia has plenty of respect now, and that won't be much less in 20yrs.

Expand full comment
Jake's avatar

academia and its findings, at least in social sciences, is generally only respected or held to be legitimate by certain political circles, at least in the current period. When it supports their arguments, others will cite academic findings, but generally this is a political divide.

Expand full comment
Alexander Kurz's avatar

"readers care far more about the prestige of publications than what criticisms are effective with those who read them with care."

Interesting. Why is that?

Expand full comment
Robin Hanson's avatar

Humans really care a LOT about status.

Expand full comment
Alexander Kurz's avatar

why do readers care about the status of a journal?

Expand full comment
TGGP's avatar

The real problem is that few are going to read such criticisms with care, most are ignorant.

https://www.overcomingbias.com/p/the-real-problemhtml

Expand full comment
Alexander Kurz's avatar

you hint a two different problems, ignorance and not-enough-time. Is there research on how important each of the factors is? Are there known strategies how to best mitigate them?

Expand full comment
TGGP's avatar

I didn't mention time. Even with more time, people would rather spend it being entertained than informed.

Expand full comment
Alexander Kurz's avatar

I thought we were talking about academic publishing.

Expand full comment
TGGP's avatar

Academia is prestigious in large part because people outside academia regard it that way. Academic publishers prioritize buzziness over replicability because the public won't penalize them for that.

Expand full comment
Paul Brassey's avatar

People outside academia have little idea what is in the journals, because the most prestigious ones are paywalled. We can read abstracts sometimes, but they are often written in obscurantist jargon. Maybe one of the reforms needed is to break the journal publishing cartel and make all scientific papers public access. After all, most of the research is paid for by the public through taxes (and government debt).

Expand full comment
Alexander Kurz's avatar

I am interested in why authors and readers would choose prestige over quality. It should be possible to investigate this question using standard methods from economics. Do you know whether there is research on this topic?

Expand full comment
Alexander Kurz's avatar

I am in computer science and afaik the role of prestige in publishing works very differently in economics and CS. So there may be sth interesting to get out of a comparison.

Expand full comment
Robin Hanson's avatar

CS publishes mainly in conferences, not journals. But otherwise what is different?

Expand full comment
Tony Skove's avatar

Are there potential patrons who don't care so much about prestige? Or could academic X-Prizes be crowd-funded? I suppose the organizers of the crowd-funding would have to also be indifferent to prestige.

Expand full comment
Robin Hanson's avatar

Have you met many?

Expand full comment
Andy G's avatar

Peter Thiel and Patrick Collison seem like two very likely ones. And AFAIK you are at most only one degree separated from each.

Expand full comment
Tony Skove's avatar

No, I haven't. I wonder about anonymous donors. Are they motivated by prestige? Honest question.

Expand full comment
Robin Hanson's avatar

Most "anonymous" donations are known by key associates.

Expand full comment
Lawrence Kummer's avatar

Big results might follow the public’s large-scale loss of confidence in Universities and their associated institutions. Including diminished credibility of the credentials it prints and the ‘facts’ it creates.

New social mechanisms might include online teaching and certification programs, for which there are already many successful models. These could fill similar functions at vastly lower cost in money and years of student’s lives. Education systems developed when books were expensive and educated people scarce are obsolete when the opposite are true.

As usual, such changes will astonish people, then be seen as inevitable.

Expand full comment
Robin Hanson's avatar

I don't see how "online", by itself, solves any of the key problems of academia.

Expand full comment
Lawrence Kummer's avatar

A shift to online is technically feasible today, but has not happened due to university’s social pre-eminence. As that diminishes, alternative social mechanisms will arise to replace them.

A shift to online certification would reduce the size, cash flow, and prestige of today’s universities. It could begin a positive feedback loop, a negative network effect.

(1) Size : Fewer students, whose spending supports then need fewer teachers and administrators.

(2) Less cash flow, as student’s spending supports much of the universities fixed costs.

(3) Less prestige, since the networks of alumni provide many kinds of support for universities.

This is not theory. Marketing changes have thrown many smaller private colleges into such a doom loop. A loss of prestige could do the same to America’s large universities, with the erosion slow moving up to the top.

America’s universities have priced themselves to unsustainable levels while both their education and research loses public trust. A shift to new certification methods (online is just one such) would “solve” universities’ problems. Just as euthanasia “solves” an individual’s problems.

Expand full comment
Robin Hanson's avatar

Demographics is forcing colleges to cut back, but otherwise their demand remands strong.

Expand full comment
Lawrence Kummer's avatar

Yes. By “marketing changes” I meant changes in the balance of supply and demand balance as the market for degrees shrinks due to fewer young people. This illustrates how a change in demand can throw universities into a doom loop, especially since they have large fixed costs.

But none of this addresses my point: that universities might have begun to crash. As Hemingway described it, “gradually then suddenly.”

Expand full comment
Hollis Robbins (@Anecdotal)'s avatar

I think you are calling for AI when you write "there’s a decent chance that we can find a way to fund “historians” (who might not be professional credentialed as such) to look back at particular areas of research, and rank past researchers in terms of who should have been listened to more."

Expand full comment
Robin Hanson's avatar

AIs should be allowed to compete in this process, but they should be held to the same standards of evaluation as humans.

Expand full comment
Hollis Robbins (@Anecdotal)'s avatar

I wrote about the role of administrators and AI here in improving academia (similar goals as you are promoting) recently here: https://www.chronicle.com/report/free/what-higher-ed-will-look-like-in-10-years

"The fundamental reorganization of university library holdings into reliable and unreliable works may also result in new disciplines, subdisciplines, or in some cases disciplinary homogenization. Currently there is a great deal of quantitative behavioral scholarship that could fall under economics, psychology, communication, sociology, or epidemiology, for example. These historical categories may disappear as AI proposes new groupings without disciplinary bias.

A not-insignificant percentage of faculty members will lose their current positions at research universities when their work does not survive rigorous vetting and scrutiny.

Beyond ensuring the integrity of research and teaching, administrators at large institutions (or consortiums of smaller ones) will also draw on AI to better understand what is growing in their own walled garden. AI can scan and absorb all papers, publications, and works in progress across a given institution and offer up a clear and detailed picture of the range of research produced by its faculty and graduate students.

AI will give administrators a clearer picture of the overlap and potential for collaboration among labs and faculty members, and will allow for more efficient institutional support for faculty research. Administrators will be able to perceive new clusters and nurture them. Those who work alone, outside the gravitational pull of past or future groupings, will be at risk of increased invisibility or precarity. In those cases, administrators will also have a role in assessing the viability of research that may be truly innovative — like mRNA technology — or a dead end."

Expand full comment
Robin Hanson's avatar

I can't see AI getting remotely enough prestige for academics to defer to their strong criticisms as much as you propose.

Expand full comment
Hollis Robbins (@Anecdotal)'s avatar

elite humans or regular humans?

Expand full comment
Robin Hanson's avatar

All held to the same standards.

Expand full comment