News Error Markets

I left grad school (physics & phil at U Chicago) back in 1984, to move to Silicon Valley to work on the then AI boom (which I did at Lockheed and NASA), and to help Xanadu in its quest for hypertext publishing. Just as I returned to grad school in 1993, this quest was achieved in the form of the World Wide Web.

The hypertext dream had been to improve public conversation, via backlinks making it easier to find criticism. But I came to doubt that concept around 1988, and soon turned to betting markets as a maybe better way to improve public conversation, via creating consensus estimates built on strong incentives. So like most who came to the topic of prediction markets, I first saw them as a way to improve the conversations we have in news, politics, and policy.

However, by 1996 I came to see much greater social value in using related markets to more directly advise important decisions, especially big ones by big orgs, including in public policy. I’ve since struggled to get others to see this. I’ve also since settled on a quite different way to use such markets to reform academia. Even so, I still see value in more directly improving public conversation, so let me now return to that topic.

A big problem is that news customers don’t seem very interested in news accuracy. For example, though it has been feasible for centuries for news sources to post bonds payable to those who can prove their stories false, few have ever done so. An example of a news practice that seems designed to aid accuracy is fact-checking. But only ~1-5% of news today is from sources that use fact checkers, only ~1/4 of consumers of such sources know this fact about them, and no one has ever bothered to do a study to see if fact-checked sources have lower error rates.

It seems that customers value fact-checking more as a costly signal of prestige, and less for the accuracy it adds. But this does offer hope that future customers might see news sources improved by prediction markets as also a costly sign of prestige. Even if news customers don’t directly demand the added accuracy, they might value the prestige that the appearance of such accuracy gives.

Today we have many prediction markets on topics related to public conversation. Even so, mainstream media cites such prices only rarely (~12 cites in NYT, WP, WSJ over last year). So I’m afraid that instead of getting media to cite prediction markets in their stories, we need to try more directly to make markets on pre-existing public statements by reporters, pundits, policy makers, and academics.

Let us start with the reader/viewer as our key initial customer. A reader/viewer might be interested in annotations that indicate the chance of a text being judged in error, conditional on it being judged. For example, as the average error rate of particular news text claims is ~1-2%, the color of the text at a claim might turn pink if its error chance was >5%, red if >25%, light green if <0.5%, and full green if <0.1%. (Different kinds of claims might have different thresholds for these colors.) While readers might not want to pay much to see any given annotation, they may pay more to be known as the sort of person who attends to such things.

To gain prestige, branded news sources might want to encourage and subsidize such annotations to their news stories. I propose that the process start with someone paying to create a betting market on if a particular public claim made in a news or policy text would be judged erroneous (and maybe how erroneous) by a particular type of judging regime. This payment might be subsidized by the news source, and might give whomever pays a right to some revenue from later fees charged on induced market activity. People likely to pay for such markets might be given early access to news stories, so markets can be there ready when the public starts to read them.

Different judging regimes use different standards of evaluation, and processes for picking judges and how they interact. For example, the accuracy of a quote might be judged by asking the person quoted or checking interview recordings, while the accuracy of a calculation can be judged by having others inspect the details of the calculation. Judging orgs should compete to imagine and realize different kinds of judging regimes.

A plan to judge a claim includes a judging date, fee, type of judging regime, and other choices for details of the judging process. Each type of judging regime has a standard plan for a first triage judging round. This plan might be conditional on a few very easy to check conditions of the doubted text. This first round might give a fast summary judgement, perhaps “This claim can’t be judged by this type of judging regime.” But usually it just looks more at the details of this statement and picks a next duration and fee more appropriate to that particular statement.

Later rounds are more intended to be final, but sometimes they end up punting to a further round, by picking a new judging plan. And sometimes their final judgement is that this claim just can’t be judged by this type of judging regime. Those who bet on that kind of outcome then get paid off.

The initial fee to create a market pays for an initially-thin automated market-maker, with a sponsor-set initial price, but a market-maker that tends to get thicker as people use it to trade. The bet assets held by market participants are all subject to a max-% tax to create a judging budget that can pay for the next judging round.

When the next judging date arrives, and it is time for the next judging round, if this judging budget is not enough to cover the next judging fee, bet assets are gambled to become enough, or to become nothing. If the budget is more than needed, assets are subject to a smaller tax. All assets held by bettors in this market, and most held by the market maker, are gambled at the same odds at the same time. If the gamble wins, the judging fee can be paid, judging proceeds, with the original creator of the market is paid winning assets held by the market maker. If the gamble loses, all bettor assets are gone, and a min thickness market maker returns with a triage judging plan, except now set to the market price just before this gambling event.

And that’s my proposed design. In this system, authors and editors of texts might want to set of mechanisms that, when such markets are created, automatically post large offers to sell at just below key error thresholds, as a way to signal confidence in their text. Reader/viewers might also want to see indications of such “truth bonds” as indicators of text quality.

If there are scale economies in judging related statements together, judging plans should plan to have them judged together, with correlated gambles of their judging fees.