“Impossible possible worlds” doesn’t suffer from this problem.

One distinction between factual claims and moral claims is that uncertainty about factual claims can be accompanied with evidence of the type that is particularly suggestive of probability judgments. For example, if I’m uncertain whether Joe is a liberal, evidence that Joe is an academic is useful evidence. Thus, we often describe the effect of that evidence in terms of conditional probabilities. If I know that 85% of academics are liberals, I have some real world, non-arbitrary basis for assigning the probability .85 that Joe is a liberal once I find out that he’s an academic. But there’s no similar basis for making such an assignment to moral claims, because there’s no factual evidence for moral claims. If we were to describe conditional probabilities for moral claims, we would be very hard-pressed indeed to say what it was that we were conditioning on, and to point to something observable in the world that corresponded to same.

But without some observable evidence to condition on, it’s hard to see why any given number is anything but a random stab. I would challenge anyone who assigns a numerical (or even some kind of merely ordinal “moral claim A is more likely than moral claim B,” where A and B are not just mutually exclusive) probability assignment to a claim to give reasons why it isn’t some other number. In particular, I’d challenge anyone who has a stated probability above .5 to explain the difference between .51 and .85 and .9999 and justify their specific choice.

I think Hal’s suggestion on this one is extraordinarily clever — perhaps we can condition on the strength of our moral sentiment/feeling/intuition. However, it still remains to be seen whether there’s some ex post way of determining how right our moral judgment was after we observe the intuition, so as to have some basis for this probability assignment. I suppose we could observe our intuitions and judge how often they match up with rational arguments. But note that this procedure would equate having a rational argument to having certain knowledge of the truth of a proposition, and I can’t see how two people, both of whom had rational arguments, could ever reconcile their opposing views of a moral claim via a Bayesian procedure. Nonetheless, I hasten to concede to Hal that if one’s only reason for believing a moral claim is one’s intuition, it might be reasonable to assign a probability to the correctness of one’s intuitions generally.

Incompatible moral propositions don’t directly correspond to different *worlds* because, no matter how you arrange the atoms in a solar system, that’s not going to make it okay to kill people. Try to imagine a world that is exactly like this world – the same people, the same brain states, all the atoms in the same place, all fixed computations have the same outputs, all existing philosophers including you have written the same words in their essays – only in that world, killing people is good, instead of evil. If you call moral propositions worlds, then how do you tell which world you’re in?

Similarly, probabilities themselves are not direct, ontological attributes of observed reality. If we don’t know whether our shoelaces are tied, that is a fact about our state of mind, not a fact about the shoelaces. (Jaynes named this the Mind Projection Fallacy.) So, just because you feel a state of uncertainty about what “probability” you ought to assign to something, doesn’t mean that “probability” is a variable-in-the-world. You could, nontrivially, state an ideal computation that you ought to be doing with the evidence you have, and then wonder about the unknown result of this fixed computation. For example, you might guess there’s a 10% probability that this ideal computation would output an answer between “90%” and “95%” that your shoelaces are tied. But, in real life, either this computation outputs “92%” or it doesn’t. And even if it is a definite fact that the computation outputs “92%”, that doesn’t mean there *really is* a 92% chance that your shoelaces are tied. Either they’re tied or they’re not. The actual world is not one in which the shoelaces have a little number 0.92 woven into the threads. The actual world is one in which a certain fixed computation has the definite output of 0.92, and your shoelaces are actually tied.

If you want to put probability distributions over probabilities, you’ve got to do some additional work to justify putting a probability distribution on something that isn’t part of the world. You might, for example, introduce a fixed computation whose unknown answer you’re putting a probability distribution over. But it is not permissible to just make the jump directly, based on the fact that you *feel unsure* about what probability you ought to assign. You can’t trust your brain – in this case, as in so many others. Your brain has a tendency to collapse its own perceptual states into the objects that they refer to. We say, “The coin has a 50% probability of coming up heads,” not “I assign a 50% probability to the coin coming up heads”. Evolution saving a few clock ticks again.

Likewise it takes some extra work to build a framework for moral possible worlds. By treating moral propositions as possible worlds, you are essentially *formalizing* that case of the Mind Projection Fallacy, which may not be a wise course of action. It may lead you to forget that not all minds are in the same frame of reference that you happen to use, the way that all minds you meet will be in the same real world as you. If you try to treat with an entity that doesn’t share your resolution procedure – Andromedan babyeaters, or a paperclip maximizer – then there’s no single variable for the two of you to both be uncertain about.