Decision Selection Bias

As futarchy interest and activity are way up lately, this seems a good time to elaborate on one of its most technical issues, one that @metaproph3t also discussed recently: decision selection bias.

To make a decision, one wants estimates of the outcome that would be caused by each possible decision. But decision markets instead give estimate of outcomes conditional on each possible decision. Selection biases can cause a difference between conditional and causal outcome estimates.

This figure describes a concrete example:

The two dimensions here are of the value of a company’s stock if the CEO is dumped, and if the CEO is kept. Assume that market speculators know only that the truth is uniformly distributed within the oval shown, but the actual decision to dump or keep will be made according to the actual point in this space. Risk-neutral speculators should then estimate the value the stock conditional on dumping by averaging the x-axis value over the yellow region, and also estimate the value of the stock conditional on keeping by averaging the y-axis value over the red region.

Combining these (x,y) values gives an “apparent center” point, while the “true center” of the oval gives the speculators best info. According to the true center, the CEO should be dumped, but according to the apparent center, the CEO should be kept. Thus a decisions selection bias here distorts the market advice, making it seem like the CEO should be kept, when they should be dumped.

Now this example has been designed to create a worst case, by basing the decision on the exact point and also by finding a worse-case (x,y) correlation. Here are many other correlations, showing that this problem is rare overall:

I’ve discussed this issue many times in talks over the years (e.g., 2002), and in a 2006 journal article I said:

To avoid this decision selection bias problem, one can either directly reveal decision maker information to speculators, or allow people with access to decision maker information to trade in these markets. One should also clearly inform speculators that a decision is about to be made, so that they do not fear that the decision will be made later when they know more. …

Another general approach … imagine … committed rolling two dice and then if snake eyes came up, flipping a coin to change policy or not. In this case it would sufficient to look at prices conditional on snake eyes and on changing policy or not.

I blogged in 2007 and 2011 on how to address this issue for election markets, and a 2006 paper suggests a different approach.

There is only a difference between causal and conditional outcome estimates in decision markets when then the decision is made using different info that the market prices. So make sure the market directly decides, or that decision makers trade in the markets, to tell all they know. Also make sure that the decision moment is clear, by measuring prices over a short know time period, and paying more for info just before that period to flush out any new info that might instead appear while measuring.

Conditioning on random decisions should very effective, but seems expensive. However, it seems much less expensive to sometimes randomly not do a policy change that you were going to do, than to sometimes randomly do a policy change you were not going to do. So I suggest that a futarchy system for considering and adopting proposals randomly reject say 5% of the changes that it would otherwise have accepted. This should ensure good estimates conditional on not adopting proposals, leaving only the potential problem of a decision selection bias distorting estimates conditional on adopting some proposal.

Given the complexity of human behavior, there’s only so far one can go with theory to address such problems. This is why I hope to run some lab experiments soon, so look for robust ways to pick decisions in the context of price conditional price histories.