Confident Proposer Bias
My recent exchange with Tyler highlighted a common bias: people presume that policy proposal authors claim high success probabilities.
Because I had written sympathetically about futarchy (public policy decision markets) and private law, Tyler presumed that I assigned high probabilities to the success of such alternatives. I countered:
I think Tyler and I both agree that private law and decision markets are among the twenty most interesting big potential policy ideas in the last few decades, that such big ideas have little chance of being adopted wholesale and even then would work out very differently than anticipated, but that such ideas are nevertheless well worth exploring and trying out first on small scales.
A novel approach to policy deserves more attention, including sympathetic discussions, when there is a positive expected payoff from further explorations of it. Such explorations can include math models, lab experiments, and field experiments. A positive payoff comes if such explorations can refine the approach into useful fielded implementations, while a negative payoff comes from wasted effort and harmful implementations. If the cost of experimenting is low and the final positive payoff could be very high, a novel policy approach can be worth exploring even with a very low probability of success.
I agree as well. It often find myself explaining that I am in favour of trying something when I believe the chance of success is small or, more commonly, scared of some risky situation when the chance of failure is small. It takes a surprising amount of effort to convey this to someone as it tends to be assumed that to be deeply worried about ecophagy or falling of an ill-maintained cliff path you have to think the chance is greater than 1%, rather than that the product of the probability and badness of outcome exceeds the badness of the prevention. I now pre-empt this possibility by phrasing my concern in terms of 'it would be worth putting in a lot of effort to avoid X, even if the chance was 1% and I see no way to justify that it is lower than that'. This works quite well, and even if it doesn't play to the 'arguments are soldiers' game, it is often quite convincing because of its revealed honesty.
More generally, people presume that those who accurately explain policy proposals--cautiously and carefully and with a straight face--believe those policies are optimal. This is in line with Eliezer's "arguments are soldiers" approach to politics.