As there’s been an uptick of interest in prediction markets lately, in the next few posts I will give updated versions of some of my favorite prediction market project proposals. I don’t own these ideas, and I’d be happy for anyone to pursue any of them, with or without my help. And as my first reason to consider prediction markets was to reform academia, let’s start with that.

Back in 2014, I restated my prior proposals that research patrons subsidize markets, either on relatively specific results likely to be clearly resolved, such as the mass of the electron neutrino, or on simple abstract statements to be judged by a distant future consensus, conditional on such a consensus existing. Combinatorial markets connecting abstract questions to more specific ones could transfer their subsidizes to those the latter topics.

However, I fear that this concept tries too hard to achieve what academics and their customers say they want, intellectual progress, relative to what they more really want, namely affiliation with credentialed impressiveness. This other priority better explains the usual behaviors of academics and their main customers, namely students, journalists, and patrons. (For example, it was a bad sign when few journals showed interest in using prediction market estimates of which of their submissions were likely to replicate.) So while I still think the above proposal could work, if patrons cared enough, let me now offer a design better oriented to what everyone cares more about.

I’d say what academics and their customers want more is a way to say which academics are “good”. Today, we mostly use recent indicators of endorsement by other academics, such as publications, institutional affiliations, research funding, speaking invitations, etc. But we claim, usually sincerely, to be seeking indicators of long term useful intellectual impact. That is, we want to associate with the intellectuals about whom we have high and trustworthy shared estimates of the difference that their work will make in the long run toward valuable intellectual progress.

A simple way to do this would be to create markets in assets on individuals, where each asset pays as a function of a retrospective evaluation of that individual, an evaluation made in the distant future via detailed historical analysis. By subsidizing market makers who trade in such assets, we could today have trustworthy estimates to use when deciding which individuals among us we should consider for institutional affiliations, funding, speaking invitations, etc. (It should be easy for trade on assets that merge many individuals with particular features, such as Ph.Ds from a particular school.)

Once we had a shared perception that these are in fact our best available estimates, academics would prefer them over less reliable estimates such as publications, funding, etc. As the value of an individual’s work is probably non-linear in their rank, it might make sense to have people trade assets which pay as a related non-linear function of their rank. This could properly favor someone with a low median rank but high variance in that rank over someone else with a higher median but lower variance.

Why wait to evaluate? Yes, distant future evaluators would know our world less well. But they would know much better which lines of thought ended up being fruitful in a long run, and they’d have more advanced tech to help them study intellectual connections and lineages. Furthermore, compound interest would give us access to a lot more of their time. For example, at the 7% post-inflation average return of the S&P500 1871-2021, one dollar becomes one million dollars in 204 years. (At least if the taxman stays aside.)

Furthermore, such distant evaluations might only be done on a random fraction, say one percent, of individuals, with market estimates being conditional on such a future evaluation being made. And as it is likely cheaper to evaluate people who worked on related topics, it would make sense to randomly pick large sets of related individuals to evaluate together.

Okay, but having ample resources to support evaluations by future historians isn’t enough; we also need to get clear on the evaluation criteria they are to apply. First, we might just ask them to sort a sample of intellectuals relative to each other, instead of trying to judge their overall quality on some absolute scale. Second, we might ask them to focus on an individual’s contributions to helping the world figure out what is true on important topics; being influential but pushing in the wrong directions might count against them. Third, to correct for problems caused by scholars who play organizational politics, I’d rather ask future historians to rate how influential an individual should have been, if others had been a bit more fair in choosing to whom to listen.

The proposal I’ve sketched so far is relatively simple, but I fear it looks too stark; forcing academics to admit more than they’d like that the main thing they care about is their relative ranking. Thus we might prefer to pay a mild complexity cost to focus instead on having future historians rate particular works by intellectuals, such as their journal articles or books. We could ask future historians to rate such works in such a way that the total value of each intellectual was reasonably approximated by the sum of the values of each of their work’s.

Under this system, intellectuals could more comfortably focus on arguing about the the total future impact of each work. Derivatives could be created to predict the total value of all the works by an individual, to use when choosing between individuals. But everyone could claim that is just a side issue, not their main focus.

To pursue this project concept, a good first step would be to fund teams of historians to try to rank the works of intellectuals from several centuries ago. Compare the results of different historian teams assigned to the same task, and have teams seek evaluation methods that can be both reliable and also get at the key questions of actual (or counterfactual) impact on the progress that matters. Then figure out which kinds of historians are best suited to applying such methods, and which funding methods best induce them to do such work in a cost-effective manner.

With such methods in hand, we could with more confidence set up markets to forecast the impact of particular current intellectuals and their works. We’d probably want to start with particular academic fields, and then use success there to persuade other fields to follow their example. This seems easier the higher the prestige of the initial academic fields, and the more open are they all to using new methods.