Prediction Markets Need Trial & Error

We economists have a pretty strong consensus on a few key points: 1) innovation is the main cause of long-term economic growth, 2) social institutions are a key changeable determinant of social outcomes, and 3) inducing the collection and aggregation of info is one of the key functions of social institutions. In addition, better institutional-methods for collecting and aggregating info (ICAI) could help with the key meta-problems of making all other important choices, including the choice of our other institutions, especially institutions to promote innovation. Together all these points suggest that one of the best ways that we today could help the future is to innovate better ICAI.

After decades pondering the topic, I’ve concluded that prediction markets (and closely related techs) are our most promising candidate for a better ICAI; they are relatively simple and robust with a huge range of potential high-value applications. But, alas, they still need more tests and development before wider audiences can be convinced to adopt them.

The usual (good) advice to innovators is to develop a new tech first in the application areas where it can attract the highest total customer revenue, and also where customer value can pay for the highest unit costs. As the main direct value of ICAI is to advise decisions, we should thus seek the body of customers most willing to pay money for better decisions, and then focus, when possible, on their highest-value versions.

Compared to charities, governments, and individuals, for-profit firms are more used to paying money for things that they value, including decision advice. And the decisions of such firms encompass a large fraction, perhaps most, of the decision value in our society. This suggests that we should seek to develop and test prediction markets first in the context of typical decisions of ordinary business, slanted when possible toward their highest value decisions.

The customer who would plausibly pay the most here is the decision maker seeing related info, not those who want to lobby for particular decisions, nor those who want to brag about how accurate is their info. And they will usually prefer ways to elicit advice from their associates, instead of from distant curated panels of advisors.

We have so far seen dozens of efforts to use prediction markets to advise decisions inside ordinary firms. Typically, users are satisfied and feel included, costs are modest, and market estimates are similarly or substantially more accurate than other available estimates. Even so, experiments typically end within a few years, often due to political disruption. For example, market estimates can undermine manager excuses (e.g., “we missed the deadline due to a rare unexpected last-minute problem”), and managers dislike seeing their public estimates beaten by market estimates.

Here’s how to understand this: “Innovation matches elegant ideas to messy details.” While general thinkers can identify and hone the elegant ideas, the messy details must usually come from context-dependent trial and error. So for prediction markets, we must search in the space of detailed context-dependent ways to structure and deploy them, to find variations that cut their disruptions. First find variations that work in smaller contexts, then move up to larger trials. This seems feasible, as we’ve already done so for other potentially-politically-disruptive ICAI, such as cost-accounting, AB-tests, and focus groups.

Note that, being atheoretical and context-dependent, this needed experimentation poorly supports academic publications, making academics less interested. Nor can these experiments be enabled merely with money; they crucially need one or more organizations willing to be disrupted by many often-disruptive trials.

Ideally those who oversee this process would be flexible, willing and able as needed to change timescales, topics, participants, incentives, and who-can-see-what structures. An d such trials should be done where those in the org feel sufficiently free to express their aversion to political disruption, to allow the search process to learn to avoid it. Alas, I have so far failed to persuade any organizations to host or fund such experimentation.

This is my best guess for the most socially valuable way to spend ~<$1M. Prediction markets offer enormous promise to realize vast social value, but it seems that promise will remain only potential until someone undertakes the small-scale experiments needed to find the messy details to match its elegant ideas. Will that be you?