A Model of Extraordinary Claims

Last week I claimed that the saying "extraordinary claims require extraordinary evidence" is appropriate anytime people too easily make more extreme claims than their evidence can justify.  Eliezer, however, whom I respect, thought the saying appropriate anytime people make claims with a very low prior probability.  So I have worked out a concrete math model to explore our dispute.  I suggest that if you are math averse you stop reading this post now.

Consider the example of power law disasters, where the chance of an event with severity greater than x goes as a negative power x^-v.  If we talk in terms of the magnitude y = log(x) of the disaster, then our prior on y is distributed exponentially as exp(-v*y).

Imagine that a claimer gets a signal s about our next disaster magnitude y, a signal normally distributed with mean y and standard deviation d.  His maximum posterior estimate (MPE) of y should rationally be z = s – d*v, and his posterior for y should be normally distributed with mean z and standard deviation d.   The claimer will claim to us either that his signal was s’, or equivalently that his MPE was z’.   We must then decide what to believe about the distribution of y (or x) from this claim.

Let us assume that the payoff U of this claimer is the sum of two parts: attention U1 and truth U2.  First, he gets an attention payoff U1 that goes as a*z’, rewarding him more for claiming that a bigger disaster looms.  Second, he gets a truth payoff U2 that is a logarithmic proper scoring rule.  That is, he must declare a probability distribution p(y) and will be paid as log(p(y)), using the chance he assigned to the actual disaster magnitude y that occurs.  (So U = a*z(s’) + Int_y p(y|s) log(p(y|s’)) dy.)

Let us also assume that the claimer is not an exact rational agent; the chance he makes any claim z’ (or s’) is exponential in his total payoff, as in exp(r*U), where r is his rationality.  Finally, assume that we know parameters like a,d,r,v, and that the claimer’s signal s is well away from the lower boundary of possible y, so we can treat integrals over normal distributions as if they went to infinity.

Putting this all together and turning the crank, we find that a claimer with zero attention payoff will claim a z’ that is normally distributed with a mean of the infinitely rational z (or equivalently a s’ that is normally distributed around a mean of the actual s).  The standard deviation c of this distribution depends on his rationality r and the strength of his payoffs U.  For a claimer with a positive attention payoff, his z’ (or s’) is also normally distributed with the same standard deviation c, but the mean of this distribution is biased toward larger disasters, with a bias b that is proportional to a

When we hear the claimer’s claim z’ (or s’), we make an inference about y.  Our rational posterior for y should have a mean z = z’-b-c*v and a standard deviation c, which depends on both d and r.   That is, we are skeptical about the claim for two reasons.  First, we correct for the bias b due to the claimer distorting his claims to gain more attention.   Second, we correct for the fact that larger disasters are a priori less likely, a correction that rises as the claimer’s rationality falls.

So this model contains effects similar to those Eliezer and I were discussing.  My claim was that in practice the attention payoff effect is larger than the irrational claimer effect, but I have not shown that yet.  Note, however, that the correction, and the relative size of the two effects, does not depend on how a priori unlikely is the claim z’.

GD Star Rating
Tagged as: ,
• http://profile.typekey.com/johnthacker/ John Thacker

His maximum likelihood estimate (MLE) of y should rationally be z = s – d*v,

Technically, no. This is the posterior mode, or maximum a posteriori, the maximum value of the posterior distribution, which you have correctly calculated. The MLE, by its name, is applied to a likelihood ratio and would, in this case, be simply equal to s. The MLE and maximum a posteriori (MAP) are equal when the prior is uniform. (Also of course we’re assuming that d is known a priori for this calculation to make sense.)

• John Thacker

The standard deviation c of this distribution depends on his rationality r and the strength of his payoffs U. For a claimer with a positive attention payoff, his z’ (or s’) is also normally distributed with the same standard deviation c, but the mean of this distribution is biased toward larger disasters, with a bias b that is proportional to a.

c = d/r and b = ad, I believe, assuming that my brief sketch is correct.

However, you have set up the model here yourself. It’s easy enough to change your attention payoff or other model features so that it does depend on how unlikely a priori the claim is. Some of the alterations will leave it no longer a nice normal distributions, though. Other examples don’t do that, but do change the model. One easy example is to suppose that the attention payoff is proportional to the square of the magnitude, so that U1 = a*y^2. In that case, the attention payoff does not affect the mean of the normal given, and introduces no bias term b, but affects the standard deviation of the given distribution.

We may also imagine an attention payoff that rather than depending on the magnitude alone, also depends on the probability of the unlikely event, perhaps incorporating just v or the entire probability. In that case then, yes, it would depend on the probability of the event.

It seems to me that you’ve excluded the probability of the event from the attention payoff, and then proceeded to show that under that model the probability of the event doesn’t affect the prediction. I’m sorry, but I don’t see how it’s extremely useful as an argument.

• http://profile.typekey.com/sentience/ Eliezer Yudkowsky

Couple of comments: First, I think you meant to say MAP not MLE. The MLE is (of course) z=s. I check your figures on the MAP. But could you explicitly spell out your equation for the agent’s combined payoff as a function of fixed s and variable claim?

Second, I think that this fails to capture the real problem of “extraordinary claims”. Mathematics has to capture the structure of the environment before solving the equation actually helps us, and we all know the danger of treating humans as Bayesians.

One, “extraordinary claims” tend to be those that violate *qualitative* generalizations. When dams and levees are built, they reduce the frequency of floods, and thus apparently create a false sense of security, leading to reduced precautions. While building dams decreases the frequency of floods, damage per flood is so much greater afterward that the average yearly damage increases. [Burton, I., Kates, R. and White, G. 1978. Environment as Hazard. New York: Oxford University Press.] People don’t model the quantitative power-law distribution in their heads – they regard the prospect of a large flood as violating the qualitative generalization, “We’ve never had a flood that large!” or “We haven’t had a flood that large since we built the dam.”

Two, social payoffs (such as attention) don’t increase automatically as log-scope – there comes a point of diminishing returns, when your claim simply causes incredulity. (An environmentalist researcher announces that the greenhouse effect is more severe than expected, and is going to destroy the entire galaxy.) There is a claim-of-scope with some maximum social payoff, rather than an infinite increase. We may end up with a bimodal distribution, one mode centered around the rational estimate but biased in the directon of increased reward, and one mode centered around the maximum social reward but biased in the direction of the rational estimate. And note that I have said nothing about which mode is greater or less than the other.

On a model which yielded that sort of shape, people would be “rational” or “play to the public”, biased maybe in the direction of the other reward, but tending to fall into one category or the other.

• http://profile.typekey.com/sentience/ Eliezer Yudkowsky

(Incidentally, what happens if agents anticipate the attempt to discount their claims, and pick the claim such that, after being discounted, it maximizes their reward function, etc…)

• http://profile.typekey.com/robinhanson/ Robin Hanson

You are both right; my error; I meant max posterior instead of max likelihood. Also I should have written U1 = a*z’ instead of a*y. I’ve corrected these in the text above.

So, Eliezer, U = a*z(s’) + Int_y p(y|s) log(p(y|s’)) dy.

John, I took Eliezer’s argument to be that there was a low probability effect that wasn’t due to an attention payoff at all.

Yes, other models might give different results; this was the first one I tried. Yes, an obvious easy generalization is a general quadratic U1, which can express the diminishing returns Eliezer suggests.

Anyone want to co-author something a write up of this? ðŸ™‚

• http://profile.typekey.com/johnthacker/ John Thacker

Also I should have written U1 = a*z’ instead of a*y.

Ah yes, right, of course.

Well, I grant that it is a reasonable sounding model. A pretty good one for a few days work, absolutely. Perhaps you should submit it to the Mathematical Contest in Modeling as a problem for them? Reading it reminded me of my years competing there. ðŸ™‚

I’m not convinced that it’s necessarily much more accurate than some other plausible models, but it does have the advantage of much nicer looking mathematics than many alternatives, and that counts for a lot.

Anyone want to co-author something a write up of this? ðŸ™‚

Well, I am in Fairfax County and a probabilist… ðŸ˜‰