# Why Common Priors

Do I contradict myself? Very well, then I contradict myself, I am large, I contain multitudes.  Walt Whitman

A key issue for the (epistemic) rationality of disagreement is whether different Bayesians can rationally have different priors.  Bayesians with different priors could easily disagree, though they would see no point in offering information to resolve it.   But a standard practice has been to assume rational priors are common.  For example, the vast majority of economic models of multiple decision makers are models of Bayesians with common priors.   And even when philosophers allow priors to be different between people, philosophers usually insist that different parts of a mind, or different versions of that mind on different days, have the same prior.

Can rational priors be different?   On the one hand, some don’t see why priors can’t be different, especially since disagreement often feels rational.  On the other hand, some say part of the meaning of rational belief is that it should not depend on arbitrary individual features, and others suggest Dutch Book arguments apply to groups as well as to individuals.  (One can claim rational priors are common without needing to give exact formulas for them, just as one can claim that P(A) + P(notA) = 1 without giving a formula for P(A).)

After eight rejections at other journals, Theory and Decision just published my paper (see also this ppt) offering a new argument for the rationality of common priors.  It only has few lines of math, which formalize this key idea: a rational prior must be consistent with reasonable beliefs about the processes that produced everyone’s priors.

That is, while priors are usually fully known to everyone (and everyone knows that everyone knows etc.), each agent is asked to consider the information situation of a "pre-agent" who is not sure which agents will get which priors.  Each agent can have a different pre-agent, but each agent’s prior should be consistent with his pre-agent’s "pre-prior," in the sense that the prior equals the pre-prior conditional on the key piece of information that distinguishes them:  which agents actually get which priors.

The main result is that an agent can only have a different prior if his pre-agent believed the process that produced his prior was special; reality correlated with his prior, but not with other priors.

Consider, for example, two astronomers who disagree about whether the universe is open (and infinite) or closed (and finite). Assume that they are both aware of the same relevant cosmological data, and that they try to be Bayesians, and therefore want to attribute their difference of opinion to differing priors about the size of the universe.

This paper shows that neither astronomer can believe that, regardless of the size of the universe, nature was equally likely to have switched their priors. Each astronomer must instead believe that his prior would only have favored a smaller universe in situations where a smaller universe was actually more likely. Furthermore, he must believe that the other astronomer would not track the actual size of the universe in this way; other priors can only track universe size indirectly, by tracking his prior. Thus each person must believe that prior origination processes make his prior more correlated with reality than others priors.

As a result, these astronomers cannot believe that their differing priors arose due to the expression of differing genes inherited from their parents in the usual way. After all, the usual rules of genetic inheritance treat the two astronomers symmetrically, and do not produce individual genetic variations that are correlated with the size of the universe.

Since it seems unreasonable to believe that the process that made your prior was this special, it also seems unreasonable to have differing priors.

By the way:  We are "blog of the week" at the Economist

GD Star Rating
Tagged as: , ,
• Paul Gowder

But how do agents actually get to common priors, given that they start with different ones? It’s not as if agents could follow some kind of Aumann-esque idea and adjust their probability estimates on the accuracy of their priors to match. An agent’s probability estimate of the accuracy of his prior would depend on some other prior, which might not be common, so it would have to be adjusted first, and that would depend on some other prior, which might not be common, and as someone said in an earlier comment, it’s “turtles all the way down.”

• http://profile.typekey.com/robinhanson/ Robin Hanson

Paul, you are confusing a prior with an earlier belief. A prior is counterfactually what you should believe if you had the minimum possible info; it need not be what you believed yesterday. If you realize that the beliefs you had yesterday are not what they should have been, given the evidence you had yesterday, then you realized that your beliefs had errors, i.e., deviations from your Bayesian ideal.

• http://yootles.com Daniel Reeves

To echo Robin: your prior is the bottom-most turtle.

• Paul Gowder

I’m really starting to have a problem with the whole prior thing. It seems to be the get-out-of-difficulty-free-card for normative Bayesianism. A prior isn’t an earlier belief, it isn’t based on any earlier belief (“bottom-most turtle”) and it isn’t (per Robin’s reply to my comment on the impossible worlds post) based on empirical evidence. It’s just this entity hanging out there bestowing probability distributions on beliefs. What’s the basis for it? Well, there isn’t one!

So where on earth do priors come from? And can anyone utter an example of one — of a belief that (a) is based on no empirical evidence, (b) is not based on any other belief, (c) is somehow still reasonable, and (d) has enough content to it to bestow probability estimates on other beliefs? (And say what makes such a belief reasonable?) It seems like the traditional synthetic a priori claims (like the axioms of mathematics) meet a and c, and maybe b, but surely don’t meet d.

And I repeat my question: given that it’s irrational to have different priors, how do agents adjust their priors to match? We can’t have probability statements about the truth of priors, because, hey, bottom-most turtle.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Paul, virtually all of your beliefs are in situations of uncertainty, and in a Bayesian framework all such beliefs require a prior of some sort. So if any such beliefs seem reasonable, that shows that there are priors that are reasonable.

Regarding your question “how do agents adjust” I could just as well ask how should you adjust when you find you have been violating P(A) + P(notA) = 1? In both cases I could give you algorithms for changing beliefs to show that algorithms are possible, but it would be easy to criticize such algorithms as not always optimal.

• http://profile.typekey.com/halfinney/ Hal Finney

Paul, I found a kind of sketchy page on where priors come from:

http://www.optimizelife.com/wiki/Prior_selection

I like the idea of Solomonoff induction, which basically formalizes Occam’s Razor. More complex possibilities get lower probability than simpler ones. I am not familiar with the technical objections to this proposal, I need to learn more about it.

Robin, is it a separate question to ask whether people do in practice have different priors, than to ask if they should have different priors? Are a person’s priors something that could in principle be studied empirically, like his genome? A new baby is born, and along with his gene map the doctors produce his prior map? Or are these questions totally confused?

• Tim Fowler

Hal Finney – Re: “More complex possibilities get lower probability than simpler ones.” That might be a reasonable rule of thumb, but it hardly seems like a universal prior, or the basis for having rational priors all match. People might reasonable disagree, or assuming they do agree, I still don’t think its enough. There is no simple and totally adequate measure of complexity of a possibility that I can think of. And if you had one different people who agreed that the more complex one’s where less likely, could still have disagreements about how much less likely, each “unit” of extra complexity makes a possibility. Also if you assume this as your criteria, does that mean that equally complex possibilities are always equally likely? That doesn’t make a lot of sense to me.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Hal, yes, I’ve emphasized Bayesian beliefs as a normative ideal; people are not Bayesian. One may in some ways be able to approximate people as Bayesian and if so then one can ask whether they have different priors. But they may well have different priors at different times, or regarding different ways a question is asked, etc. There is no guarantee that people are simple.

• Daniel Greco

My prior assigns a very low probability to the proposition that shortly after my birth, I was drugged, kidnapped, and my brain was extracted and put in a vat, where it was then stimulated in such a way that I had experiences continuous with my pre-vat life (and consistent with my current experiences).

So, I believe my prior favors anti-vat beliefs. However, had I actually been drugged, kidnapped, and envatted, my prior would still have favored anti-vat beliefs. I do not believe that my prior would only have favored anti-vat beliefs in situations in which anti-vat beliefs were more likely to be true. If envatting kidnappers were operating frequently around the time of my birth, then pro-vat beliefs would have been more likely to be true, but I would still have had a prior that favored anti-vat beliefs (assuming we hold my genetic history and pre-vat life constant).

Does this mean that my anti-vat beliefs are irrational? In discussing an example in your paper, you say: “Each astronomer must instead believe that his prior would only have favored a smaller universe in situations where a smaller universe was actually more likely.” This certainly seems like it does mean that my anti-vat beliefs must be irrational, if I believe that my prior would have favored anti-vat beliefs even in situations where the anti-vat beliefs were not likely to be true. This seems like a counterintuitive consequence, if we think that I (and others) are rational in assigning very low probabilities to bizarre skeptical hypotheses like the brain-in-a-vat one.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Daniel, I say if your prior is different then the situation is different, but I do not say the converse. The situation can be different without your prior being different.

• http://profile.typekey.com/nicholasshackel/ Nicholas Shackel

Robin: As I see it, the central issue to be resolved if Bayesianism is to have a notion of rational belief that is something more than probabilistic coherence is to justify on Bayesian grounds the existence of a uniquely rational set of priors. I take it that this is the objective of this paper.

Before I pursue it, please tell me what you mean by ‘a reasonable belief about the processes that produced everyone’s priors’.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Nicholas, in this context “reasonable” beliefs about the origins of priors are beliefs consistent with our usual stories about our origins, such as our beliefs about genetic inheritance and about cultural transmission of ideas.

Although I would guess that there is in fact a uniquely rational set of priors, one doesn’t have to believe this to think Bayesianism can go beyond mere coherence. One might accept further rationality constraints beyond coherence, but not enough constraints to determine a unique prior. For example, a common accepted constraint is that the different versions of a self at different times should have the same prior. This goes beyond coherence, but stops well short of uniqueness.

• http://profile.typekey.com/nicholasshackel/ Nicholas Shackel

Robin:
reasonable origin beliefs = ‘consistent with usual stories about our origins, such as our beliefs about genetic inheritance and about cultural transmission of ideas.’ That’s what I thought, but (and I grant all the maths) I don’t see how this satisfies the explanatory burden: ‘to justify on Bayesian grounds the existence of a uniquely rational set of priors’.

re coherence: But ‘the different versions of a self at different times should have the same prior’ is implied by standard Bayesianism, since an ideal Bayesian believer will start with priors and only modify beliefs by conditionalising on evidence as it is received, so will always have the same foundational prior, so this is not a constraint additional to coherence.

• http://www.nickbostrom.com Nick Bostrom

Robin, you wrote: “Bayesians with different priors could easily disagree, though they would see no point in offering information to resolve it.”

They might see a point in offering information to resolve disagreements because they might not yet know whether a particular disagreement is based on diffferences in their priors. The easiest way to check this might be for each of them to tell the other all the relevant evidence and arguments they have, and see if agreement results. Moreover, even if two Bayesians have different priors for some propositions, it might still be the case that when enough empirical information becomes available it will cause their posterier probabilities to converge.

One could turn the claim around and say rather that if it is true that Bayesians cannot agree to disagree, then they should see no point to offer information to resolve their (initial) disagreements. They could simply repeatedly express their (revised) estimates of the probability of the proposition in question, and their beliefs would converge even without them giving any other arguments or evidence.

I also think it is not necessary for you to claim that “it also seems unreasonable to have differing priors”. I think your claim should at least be weakened to: “it seems unreasonable to have priors that differ in such ways that even after they have been conditionalized on (a) standard information about the origins of human priors, and (b) the facts about other people’s priors, the posterior probabilities they give still disagree.” This might be a terminological issue, but the way I think about these things is something like this:

1. There is a set of all possible priors
2. A subset (not a singleton set) of these are “reasonable”. A reasonable prior must allow for reasonably efficient induction and enable an agent to learn at a normal pace about the actual world.
3. Part of what is required for a prior to be reasonable is also that it takes information about its origins into account in certain ways when conditionalized on such information.
4. We actually have relevant information about the origins of our priors which is such that after we conditionalize on it we find that some disagreements seem irrational.
5. It becomes important to determine just which disagreements (and opinions in general) are irrational in light of this evidence.
6. Robin has presented an interesting argument that tries to derive a very strong conclusion about 5.
(7. I have not yet fully made up my mind about the extent to which Robin’s argument succeeds or what exactly the limitations of his argument might be.)

• http://profile.typekey.com/robinhanson/ Robin Hanson

To be very clear: I do not claim that this paper shows that there is a unique rational prior. No one analysis could plausibly do that. Analysis of rationality is in many ways a matter of collecting rationality constraints; the more constraints one accepts, the smaller the set of rational beliefs. We can each make guesses about how small that set will eventually be, once we have all relevant constraints. In this paper, I suggest a new constraint, which I think offers a strong argument for common priors in our usual situations.

Nick, yes, my statement about info to resolve disagreements is vague and unsatisfactory; I’ll have to keep thinking if I can find a better way to say what I had in mind.

Nicholas, one can think of me today and me yesterday as two different agents. So the weakest coherence one could impose would be coherence for each agent. Coherence for a person, including all the agents associated with that person, is a stronger constraint.

• http://profile.typekey.com/jdannan/ James Annan

What do you mean by “rational prior”? AIUI the concepts are pretty much orthogonal.

• http://profile.typekey.com/robinhanson/ Robin Hanson

James, in this context “rational” is understood epistemically, as more likely to track truth. A “prior” is a counterfactual set of beliefs one would hold in a situation of minimum imaginable information.

• http://profile.typekey.com/jdannan/ James Annan

Sorry, that still leaves me none the wiser. Surely all priors will converge to the truth in the presence of increasing information – unless one actually assigns the truth a prior probability of zero. Is this what you mean?

• http://profile.typekey.com/robinhanson/ Robin Hanson

James, convergence with evidence has nothing to do with what I’m saying. If the abstract of a paper doesn’t explain all the details you are curious about, perhaps you should read the paper?

• http://profile.typekey.com/jdannan/ James Annan

I did read the paper before posting my first comment, but didn’t understand it.

• http://profile.typekey.com/robinhanson/ Robin Hanson

James, do you mean you didn’t at all understand anything in the paper? If so, I’m at a loss with how to respond. If you can articulate a more specific question in the context of the paper, I can try to respond.

• http://profile.typekey.com/nicholasshackel/ Nicholas Shackel

Robin: ‘I do not claim that this paper shows that there is a unique rational prior. No one analysis could plausibly do that.’ But your argument that disagreement cannot be reasonable requires there to be a unique rational prior.

‘one can think of me today and me yesterday as two different agents’. Not without contradiction you can’t. Proof: Let the two different agents be x and y. Being different, x is not y. But each is you, so Robin is x and Robin is y; therefore x is y by the transitivity of identity, contradicting x is not y.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Nicholas, the Bayesian argument against disagreement requires only that rational priors be common, not that they be unique. And surely you have read Parfit (there at Oxford) on our being selves across times.

• http://profile.typekey.com/nicholasshackel/ Nicholas Shackel

Robin: I’ve put the answer to your first remark in a post.

What exactly is it that Parfit says about identity over time that you think gets you out of the contradiction?

• http://profile.typekey.com/robinhanson/ Robin Hanson

Nicholas, today I am awake, but last night I was asleep. We do not conclude from this that awake = asleep. Similarly, it could be consistent to say today I have prior P1, while last night I had prior P2, and that these are both rational. There is no more a logical requirement that P1 = P2 than that asleep = awake.

• http://profile.typekey.com/sentience/ Eliezer Yudkowsky

Nicholas, the “trivial” Unique Rational belief is the probability distribution that assigns probability 1 to the actual world and probability 0 to all other possibilities. Bayesians with a common prior cannot agree to disagree in their posteriors, but this doesn’t mean their agreed-upon posterior is the Unique Rational distribution – it means they’ve shared information. Similarly, if you could prove that Bayesians couldn’t disagree about their prior, it still might not prove that there was a nontrivial Unique Rational prior, or that they had attained it.

Note that everything a Bayesian does is interpretable as an attempt to approximate the “trivial” Unique Rational belief as much as possible: “Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.”

• http://profile.typekey.com/nicholasshackel/ Nicholas Shackel

Eliezer: Beliefs are not probability distributions since beliefs are mental states and probability distributions are mathematical objects; a true belief isn’t necessarily a rational belief, let alone a trivially rational belief. Aumann’s theorem on its own doesn’t show that no disagreements are reasonable. If you could prove that ideal Bayesian believers will have the same prior, and Normative Bayesianism is true, then you would have proved that ‘there was a … Unique Rational prior’.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Nicholas, this thread started from my saying “one can think of me today and me yesterday as two different agents. So the weakest coherence one could impose would be coherence for each agent. Coherence for a person, including all the agents associated with that person, is a stronger constraint.” Apparently you use the word “agent” differently from me, and refer to a network of definitions and concepts of which I am not familiar. Is there must be another word we could substitute for “agent” in that paragraph that would be acceptable to you?

• http://profile.typekey.com/sentience/ Eliezer Yudkowsky

Nicholas, all of these results are about Bayesian agents in communication with each other – not just communication, but a state of common knowledge. So an Aumannlike result for priors would say, “If you have common knowledge of each other’s priors you must have the same prior” or some such. I’m not sure such a thing is true, mind you, but it can be true without implying anything in the way of “all rational agents have a common prior”, just as the classic Aumann result shows that Bayesians with common knowledge of each other’s beliefs have the same beliefs, but not that “all rational agents have a Unique Rational belief”.

FYI: So far as I’m concerned, a probability distribution is a unified mathematical way of *viewing* beliefs of various kinds, high and low anticipations of particular experiences, and so on. A Unique Rational probability distribution would determine, up to freedoms of mere representation, a unique set of beliefs and anticipations with respect to facts and experiences.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Eliezer, FYI, Bayesians always have common knowledge of their priors.

• http://profile.typekey.com/unclemarx/ Michael Greinecker

1. This is a problem that is more basic than Baysianism. If I have the same information as you but disagree with you I cannot really attribute my disagreement to a reason that make my opinion better than yours. But why should rationality determine much?

2. I think that this is of no help in Baysian games. In these cases one has to accept that the agents DO have different information (otherwise one couldn’t apply them in the social sciences). Even if there were a coherent argument that there is a unique rational common prior- when one is born(!)- that would not justify the assumption of a common prior in economics.

• http://profile.typekey.com/sentience/ Eliezer Yudkowsky

Robin, I’m not sure I understood that comment – did you mean that most Aumannish papers make that assumption? I certainly couldn’t write down my own prior, but of course I’m not a Bayesian.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Michael, yes, previously the usual argument given for common priors was that rational beliefs should not vary with arbitrary personal characteristics. And yes, if you are modeling irrational humans, then rational Bayesians may not be what you want.

Eliezer, I mean that in *every* model where an agent has a prior, priors are common knowledge.

• http://www.weidai.com Wei Dai

I have a couple of late comments. (And a good excuse for being late: the earthquake near Taiwan made parts of the Internet inaccessible for a few days.)

One is that I don’t agree with the intuition that apparently inspired this paper. Quoting from it:

“For example, if you learned that your strong conviction that fleas sing was the result of an
experiment, which physically adjusted people’s brains to give them odd beliefs, you might
well think it irrational to retain that belief (Talbott, 1990). Similarly it might be irrational
to be more optimistic than your sister simply because of a random genetic lottery.”

I don’t see why it’s irrational to be more optimistic than your sister simply because of a random genetic lottery. Consider the analogous argument with preferences. If you learned that your strong taste for bitter foods was the result of an experiment which physically adjusted people’s brains to give them odd preferences, you might think it irrational to retain that preference. Does it follow that it’s irrational to enjoy being outdoors more than your sister simply because of a random genetic lottery?

The other comment is that I can’t figure out the connection between this intuition and its formalization: “This condition in essence requires that each agent’s ordinary prior be obtained by updating his pre-prior on the fact that nature assigned the agents certain particular priors.” I’ve tried to accept Robin’s intuition for the purpose of trying to figure out what this sentence means, but so far without success. I can accept that either an agent’s prior is assigned randomly by nature, or it’s obtained from some pre-prior by updating. But how can both be true at the same time?

• http://profile.typekey.com/robinhanson/ Robin Hanson

Wei, preferences are about you, beliefs are about the world. Beliefs should only change when the world or your info about it changes, but not otherwise change when you change. Regarding your second comment, I claim that if you find nature has assigned you a prior which violates a rationality constraint, you should reject it and replace it with a better one.

• conchis

Nicholas – the problem with your proof is that it assumes what it sets out to prove. The error lies in the claim “each is you, so Robin is x and Robin is y”. The point is precisely to deny that Robin(t) and Robin(t+s) are identical, and so x=Robin(t) and y=Robin(t+s) are not identical either.

Now, you might think that the reasons for denying “Robin(t) is identical to Robin(t+s)” are bad ones, but I’d suggest you read Parfit on this. In any event, you’re a far cry from establishing the contradiction you claim.

• http://www.weidai.com Wei Dai

I think I’m starting to get it. Let me restate the main idea, and someone let me know if I got it right.

Human beings are not born as generic reasoners without any information about the world we live in. Instead evolution has provided us with a prior that is partially optimized for this world. However since evolution is random and unfinished, some aspects of this prior are arbitrary. Robin’s idea is that we can remove the randomness and keep only the useful information in the prior by taking the nature-provided prior as the first data point of a generic reasoner with a generic prior (which Robin calls a pre-prior, and which truly has no information about the world), instead of adopting it directly as the prior.

If I’ve understood it correctly so far, my remaining question is, is it assumed that all of the pre-agents have the same pre-prior? It seems to me that this updating process will remove the arbitrary differences in the nature-provided priors, but differences in the pre-priors continue to be reflected in the post-update priors. Is that correct?

• http://profile.typekey.com/robinhanson/ Robin Hanson

Wei, your restatement looks fine to me, and no we need not assume pre-agents have the same pre-prior. We need only assume that your pre-prior does not think that your prior had a special origin.

• http://www.weidai.com Wei Dai

In the astronomers example, suppose one astronmer has a pre-prior that assigns a higher probability to the universe being open than the other’s pre-prior. Even if they both agree that the nature-provided priors are not special and give no information about the actual world on this issue, wouldn’t they still end up with different post-update priors just from the pre-priors being different?

• http://profile.typekey.com/robinhanson/ Robin Hanson

Wei, the whole point of math is that one’s assumptions and conclusion can be described concisely, even if the proof has more detail. I’ve proven that common priors follow from believing your origins are not special. It may be that such a belief also constrains the pre-priors; I don’t know if so or not, but that would be an interesting question to explore.

• http://www.weidai.com Wei Dai

Ok, I think I know what is going on here. Robin’s “pre-rationality” assumption is that each agent’s pre-prior, when conditioned on the assignment of priors, equals the prior that was “assigned” to him. This means that the “assignment of priors” in this assumption cannot be the assignment of priors by nature that I talked about earlier in my restatement of the main idea, since the nature-provided priors must contain randomness that wouldn’t survive this update process.

Instead, the “assigned” prior (the one that the assumption refers to) must actually be an agent’s pre-prior updated by nature’s assignment of priors. Only then would updating the pre-prior by this “assigned” prior give you back the “assigned” prior. (I put “assigned” in quotes, because these priors are not actually assigned in the normal sense of the word.)

At this point it becomes clear that Robin’s other assumption, that each pre-agent does not think his “assigned” prior has a special origin, is equivalent to saying that all pre-agents have the same pre-prior. After all, if you have a pre-prior different from everyone else, then your “assigned” prior is special, since it is just the updated version of your special pre-prior.

• http://profile.typekey.com/robinhanson/ Robin Hanson

Wei, it make be clear to you that my assumption “is equivalent to saying that all pre-agents have the same pre-prior”, but it is not at all clear to me. I think I could come up with a counter example – can you come up with a proof of this equivalence?