I’m not offering you a phony seventeen-step “proof that murder is normally wrong.”  Instead, I begin with concrete, specific cases where morality is obvious, and reason from there.  Bryan Caplan.

My debate with Bryan Caplan made me reflect again on our differing attitudes toward intuition.  While we still differ, Bryan has greatly influenced my thinking.

For each of our beliefs, we can ask our mind to give our "reasons" for that belief.  Our minds usually then offer reasons, though we usually don't know how much those reasons have to do with the actual causes of our belief.  We can often test those reasons through criticism, increasing confidence when criticism is less effective than expected, and decreasing confidence when criticism is more effective than expected.

For some of our beliefs, our minds don't offer much in the way of reasons.  We say these beliefs are more "intuitive."  In a hostile debating context this response can seem suspicious; you might expect one side in a debate to refuse to offer reasons just when they had already tested those reasons against criticism, and found them wanting.  That is, we might expect a debater to pretend he didn't have any reasons when he knew his reasons were bad. 

But this doesn't obviously support much distrust of our own intuitive beliefs.  Not only is our internal mind not obviously like a hostile debating context, but we must admit that our minds are built so that the vast majority of our thinking is unconscious.  It is unreasonable to expect our minds to be able to tell us much in the way of reasons for most of our beliefs. 

Furthermore, we must admit that even when we do have reasons for our beliefs, not only do our chains of reasoning usually end at pretty intuitive beliefs, but we usually can't even fully articulate why each step in a reasoning chain supports the next one.  We clearly have little choice but to rely greatly on intuition.

But there still remains the question of how much to rely on intuition, when we do have a choice.  Sometimes we seem to have a choice between just accepting belief confidence levels suggested by opaque subconscious processes, or employing more explicit reasoning processes, even if those more explicit processes still rely heavily on intuitive beliefs. 

We find ourselves managing complex networks of beliefs. Bryan's picture seems to be of a long metal chain linked at only one end to a solid foundation; chains of reasoning mainly introduce errors, so we do best to find and hold close to our few most confident intuitions.  My picture is more like Quine's "fabric," a large hammock made of string tied to hundreds of leaves of invisible trees; we can't trust each leaf much, but even so we can stay aloft by connecting each piece of string to many others and continually checking for and repairing broken strings. 

But having identified differing pictures, what then?  Perhaps Bryan is satisfied to just have an intuition preferring his long chain picture, but I want to find more explicit reasons to choose among pictures.  I have to admit, however, that I don't have very much yet I can point to.  I'm not even sure we have much in the way of empirical data on belief accuracy of folks who rely more versus less on intuition. 

We remain sadly ignorant about intuition, an important neglected research area.

Added: A week ago some professional philosophers struggled here to demarcate reliable and unreliable intuitions, with little success. HT to Michael.

More added: We clearly vary across topics and people in when we rely how much on intuition.  So the more precise question is if we can identify any biases in these judgements, when we rely too much or too little.

But if you want to find the folks who most love cars for their main purpose, getting folks around in their daily lives, you'll have to filter out the sellers, hobbyists, and do-gooders to find ordinary people who just love their cars.  For the most part, car companies love to sell cars to make cash, car hobbyists love to use cars to show off their personal abilities, and do-gooders use cars to show off their compassion.  By comparison, those who just love to drive from point A to B don't shout much.

Truth loving is similar.  Most folks say they prefer truth, but the folks most vocal about loving "truth" are usually selling something.  For preachers, demagogues, and salesmen of all sorts, the wilder their story, the more they go on about how they love truth.  The next most vocal in their enthusiasm for truth are those who, like car hobbyists, use public demonstrations of truth-finding to show off personal abilities.  Academics, gamers, poker players, and amateur intellectuals of all sorts are proud of the fact that their efforts reveal truth, and they make sure you notice their proficiencies. And do-gooders earnestly talk about the importance of everyone understanding the truth of the uninsured, the illiterate, etc.

The people who just want to know things because they need to make important decisions, in contrast, usually say little about their love of truth; they are too busy trying to figure stuff out.  These are the "truth lovers" I most respect in the sense of trusting their efforts to be directly targeted to actually uncovering truth.  Sellers, hobbyists, and do-gooders are instead more likely to pretend to seek truth while actually seeking cash or respect. 

What if you wanted to convince others that you were actually devoted primarily to truth about some topic, and to an unusual degree?  Perhaps the clearest signal would be to show that you are buying truth, not selling it or making it, especially if you use some sort of auction to show you buy truth from the least cost provider.  Instead of offering purported truths for others to believe, or paying ideologues to "discover" already-agreed-on "truths," or inviting others to admire your truth-discerning skills in practice, offer prizes payable to whomever actually uncovers truths on your topic. 

When there are identifiable accomplishments that would clearly indicate truth had been uncovered, you can fund prizes payable to whoever creates such accomplishments.  Or if you want to encourage accurate estimates now on topics where truths will eventually be revealed by other means, you can fund "info prizes," such as by subsidizing prediction markets. 

If these are such clear signals of truth-loving, why do we see so few people sending such signals?  One obvious explanation is that the signal works; it shows us how few big truth fans there really are out there.

Added 13Apr:  Arnold riffs here.  Of course even ordinary people buy cars for reasons other than getting from A to B.  And they can buy newspapers to prepare themselves for clever in-fashion conversations; they needn't want truth.

But sometimes, not only is the best combination not the average, it's more extreme than either original belief.

Let's say Jane and James are trying to determine whether a particular coin is fair.  They both think there's an 80% chance the coin is fair.  They also know that if the coin is unfair, it is the sort that comes up heads 75% of the time.

Jane flips the coin five times, performs a perfect Bayesian update, and concludes there's a 65% chance the coin is unfair.  James flips the coin five times, performs a perfect Bayesian update, and concludes there's a 39% chance the coin is unfair.  The averaging heuristic would suggest that the correct answer is between 65% and 39%.  But a perfect Bayesian, hearing both Jane's and James's estimates – knowing their priors, and deducing what evidence they must have seen - would infer that the coin was 83% likely to be unfair.  [Math footnoted.]

Perhaps Jane and James are combining this information in the middle of a crowded tavern, with no pen and paper in sight.  Maybe they don't have time or memory enough to tell each other all the coins they observed.  So instead they just tell each other their posterior probabilities – a nice, short summary for a harried rationalist pair.  Perhaps this brevity is why we tend to average posterior beliefs.

However, there is an alternative.  Jane and James can trade likelihood ratios.  Like posterior beliefs, likelihood ratios are a condensed summary; and, unlike posterior beliefs, sharing likelihood ratios actually works.

Let's listen in on a conversation where Jane and James trade likelihood ratios:

JANE: My observations are seven and a half times as likely if the coin is unfair, as if it is fair.

JAMES:  My observations are two and half times as likely if the coin is unfair, as if it is fair.

BOTH, in unison: That means our joint observations are about nineteen times as likely if the coin is unfair. But our prior for unfair coins is 20%, which means a prior odds ratio of 1:4.  Combining with Bayes' theorem, we get (1:4)*(19:1), which is about 5:1 in favor of an unfair coin.

[BAR PATRONS edge away slightly.]

Now that you see how sharing likelihood ratios can work, you'll probably be itching to put them to work in your daily life.  As with most rationalist tricks, it helps to have a number of cached examples of places they can be used.

(1) Distinguish evidence from priors. I've been in several conversations that went roughly like this:

Person A:  So, what do you think of Jack?

Person B: My guess is that he's moderately (smart/trustworthy/whatever), but not extremely so.

Person A: Is the "not extremely so" because you observed evidence Jack isn't, or because most people aren't and you don't have much data that Jack is?  Where's the peak of your likelihood function?

This type of dialog is useful.  Let's say that A's initial impression is that Jack is amazing, and B's impression is that Jack is somewhat less amazing.  If B knows Jack well, A should lower her estimate of Jack.  But if B's impression come from a tiny amount of amazing-looking data from Jack — just not enough to pull Jack all the way from "probably normal" to "probably amazing" — A should raise her estimate.  B's posterior expectations about Jack's amazingness are identical in the two cases, even though B's observations in the two cases have opposite implications for A.  Trading likelihoods notices the difference, but trading average posterior impressions doesn't.

(2)  Avoid double-counting your priors. Robin Hanson suggested adjusting women's SAT-math scores toward the mean (downward for high-scoring women, upward for low-scoring women) if women's math aptitudes have a smaller standard deviation than men's.  Moral intuitions aside, adjusting in this manner would improve the scores' accuracy if used as stand-alone estimates of strangers' SAT-math abilities; perhaps a woman with a single score of 800 has the same expected score on subsequent SAT-math tests (the same best point estimate for "true SAT-math ability") as a man who received a single score of, say, 770.

However, adjusting scores in this manner mixes likelihoods in with priors.  SAT scores are best interpreted as likelihood functions: an SAT score of 800 has one likelihood from a person whose "true ability" is superb, another from a person whose "true ability" is moderate, etc.  If you mix these likelihood functions with your prior (as gender-adjusted SAT scores would mix them), combining multiple indicators becomes more difficult.  For example: suppose again that a single SAT-math score of 800 from a woman implies the same best point estimate of "true ability" as a single score of 770 in a man (because of differing priors plus the chance of testing error).  Two SAT-math retests of 800 in a woman will then imply a higher best point estimate of true ability than two 770's in a man.  The "gender-adjustment" would work for single, stand-alone SAT measurements, but it breaks when multiple indicators are combined.  If you mix the likelihood function with your prior and then combine it with other mixed indicators (e.g., multiple gender-adjusted SAT scores, or gender-adjusted SAT scores plus gender-adjusted letters of rec), you pull too strongly toward the prior.

The take-home in all these cases is to keep hold of your likelihood ratios.  Instead of tracking how likely your lead theory is to be true, or remembering a single theory which was most representative of the range of remaining possibilities (like Jack's average expected amazingness), try to track how likely your data-set is under one hypothesis vs. another.  (You'll need to separately remember the prior on each hypothesis.)  I suspect such tracking may also help with confirmation bias; I don't know if it ends up confusing people in other ways.

One major caveat: in our example with the coin, and in A's and B's estimations of Jack's amazingness, combining likelihood ratios led to more extreme beliefs.  (In more general examples, combining likelihood ratios may not lead to more extreme beliefs, but it almost always leads to more specific beliefs.)  When trying this yourself, make sure the likelihood ratios you're combining are independent indicators of the variable you're trying to infer.  Otherwise, you and your co-rationalists may pull one another to beliefs that are unjustifiably extreme (or specific).

(Math for original example:
James, to end up with a 39% posterior on the coin being heads-weighted, must have seen four heads and one tail:
P(four heads and one tail| heads-weighted) = (0.75^4 ∙ 0.25^1) = 0.079.  P(four heads and one tail | fair) = 0.031.  P(heads-weighted | five heads) = (0.2∙0.079)/(0.2∙0.079 + 0.8∙0.031) = 0.39, which is the posterior belief James reports.
Jane must similarly have seen five heads and zero tails.
Plugging the total nine heads and one tail into Bayes' theorem:
P(heads-weighted | nine heads and a tail) = ( 0.2 ∙ (0.75^9 ∙ 0.25^1) ) / ( 0.2 ∙ (0.75^9 ∙ 0.25^1) + 0.8 ∙ (0.5^9 ∙ 0.5^1) ) = 0.83, giving us a posterior belief of 83% that the coin is heads-weighted.)

It doesn't seem you can simply plug your uncertainty into expected utility decision theory and crank the wheel; because many moral theories state that you should not always maximize expected utility.

Even if we limit consideration to consequentialist theories, it still is hard to see how to combine them in the standard decision theoretic framework.  For example, suppose you give X% probability to total utilitarianism and (100-X)% to average utilitarianism.  Now an action might add 5 utils to total happiness and decrease average happiness by 2 utils.  (This could happen, e.g. if you create a new happy person that is less happy than the people who already existed.)  Now what do you do, for different values of X?

The problem gets even more complicated if we consider not only consequentialist theories but also deontological theories, contractarian theories, virtue ethics, etc.  We might even throw various meta-ethical theories into the stew: error theory, relativism, etc.

I'm working on a paper on this together with my colleague Toby Ord.  We have some arguments against a few possible "solutions" that we think don't work.  On the positive side we have some tricks that work for a few special cases.  But beyond that, the best we have managed so far is a kind of metaphor, which we don't think is literally and exactly correct, and it is a bit under-determined, but it seems to get things roughly right and it might point in the right direction:

The Parliamentary Model.  Suppose that you have a set of mutually exclusive moral theories, and that you assign each of these some probability.  Now imagine that each of these theories gets to send some number of delegates to The Parliament.  The number of delegates each theory gets to send is proportional to the probability of the theory.  Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting.  What you should do is act according to the decisions of this imaginary Parliament.  (Actually, we use an extra trick here: we imagine that the delegates act as if the Parliament's decision were a stochastic variable such that the probability of the Parliament taking action A is proportional to the fraction of votes for A.  This has the effect of eliminating the artificial 50% threshold that otherwise gives a majority bloc absolute power.  Yet – unbeknownst to the delegates – the Parliament always takes whatever action got the most votes: this way we avoid paying the cost of the randomization!)

The idea here is that moral theories get more influence the more probable they are; yet even a relatively weak theory can still get its way on some issues that the theory think are extremely important by sacrificing its influence on other issues that other theories deem more important.  For example, suppose you assign 10% probability to total utilitarianism and 90% to moral egoism (just to illustrate the principle).  Then the Parliament would mostly take actions that maximize egoistic satisfaction; however it would make some concessions to utilitarianism on issues that utilitarianism thinks is especially important.  In this example, the person might donate some portion of their income to existential risks research and otherwise live completely selfishly.

I think there might be wisdom in this model.  It avoids the dangerous and unstable extremism that would result from letting one’s current favorite moral theory completely dictate action, while still allowing the aggressive pursuit of some non-commonsensical high-leverage strategies so long as they don’t infringe too much on what other major moral theories deem centrally important.

But maybe somebody here has better ideas or suggestions for improving this model?

the pope said the book “explained with great clarity” that “an interreligious dialogue in the strict sense of the word is not possible.” In theological terms, added the pope, “a true dialogue is not possible without putting one’s faith in parentheses.”

What are we to make of a statement like this?

Let’s start with the basics. 

Basic 1: Theological claims are beliefs — they are statements with propositional content that refer to the world.  "There is a God."  "God loves me."  "The world was created ten thousand years ago."  One might reinterpret religion to strip away the propositional content, but then one loses everything that makes religion different from any random social activity. 

Basic 2:  Beliefs should be held for reasons.  Many of our beliefs are not held for reasons, in the sense that I’m using the term (normative, rather than explanatory reasons).  As Robin says in the linked post, those beliefs — random beliefs, beliefs that are contingent on one’s social and physical history rather than on the sorts of things that actually justify beliefs (evidence, argument) ought to be rejected.  This includes beliefs that are based on un-swamped priors from, e.g., how one was raised. 

Basic 3: Beliefs have objective and universal truth-values.  It is flat-out incoherent to utter sentences like "God exists for me but not for you."  This might not be true about everything, but it is manifestly true about ontological claims about the world, historical claims, etc. 

In light of those three basics, what can we make of the pope’s statement?  I take there to be four possibilities for making sense of it. 

1.  The pope believes that his beliefs are not held for reasons.  This, of course, counsels revising those beliefs. 

2.  The pope believes that his beliefs are held for reasons, but that the beliefs of those with whom he disagrees are not. Given that his beliefs and the beliefs of, say, Muslims, are roughly on an epistemic par, this belief is unwarranted. 

3.  The pope believes that his beliefs, and those of his interlocutors, are underdetermined by the available reasons.  That is, the evidence and arguments are consistent with both his beliefs and those of his interlocutors.  If that’s the case, however, it seems to call for his putting equal credence in his beliefs and in those of his interlocutors.

4.  He believes he has access to reasons that are fundamentally incommunicable.  Gnosis.  There are two worries about this claim.  First, is truly incommunicable belief really impossible?  Agents ought to be able to communicate the fact of their gnosis: the pope ought to be able to turn to the mullah and say "I experienced a gnosis, and so did these millions of other people," and that ought to count for the mullah, if the mullah holds his beliefs for reasons (that is, is a bayesian).  Second, if it doesn’t count for the mullah, maybe it’s because the mullah experienced a gnosis too, and that possibility, of course, ought to count for the pope.

Any way we take it, it seems like the pope’s evident belief that it’s impossible to discuss religious differences with those who have different beliefs ought to lead the pope to reduce his credence in his own beliefs.  The pope ought not to be Catholic. 

- John Stuart Mill, from Utility of Religion

Much electronic ink has been spilled on this blog about Pascal’s wager.  Yet, I don’t think that the central issue, and one that relates directly to the mission of this blog, has been covered.  That issue is this: there’s a difference between the requirements for good (rational, justified) belief and the requirements for good (rational, prudent — not necessarily moral) action.

Presented most directly: good belief is supposed to be truth and evidence-tracking.  It is not supposed to be consequence-tracking.  We call a belief rational to the extent it is (appropriately) influenced by the evidence available to the believer, and thus maximizes our shot at getting the truth.  We call a belief less rational to the extent it is influenced by other factors, including the consequences of holding that belief.  Thus, an atheist who changed his beliefs in response to the threat of torture from the Spanish Inquisition cannot be said to have followed a correct belief-formation process. 

On the other hand, good action is supposed (modulo deontological moral theories) to be consequence-tracking.  The atheist who professes changed beliefs in response to the threat of torture from the Spanish Inquisition can be said to be acting prudently by making such a profession.

A modern gloss on Pascal’s wager might be understood less as an argument for the belief in God than as a challenge to that separation.  If, Modern-Pascal might say, we’re in an epistemic situation such that our evidence is in equipoise (always keeping in mind Daniel Griffin’s apt point that this is the situation presumed by Pascal’s argument), then we ought to take consequences into account in choosing our beliefs. 

There seem to be arguments for and against that position… 

In its favor we can imagine situations where it’s not the nastiness of an all-knowing deity that makes our beliefs consequential, but something about our own psychologies.  Imagine Allen. He’s an alcoholic. He makes an all-things-considered judgment that it would be best for him to stop drinking. He also holds the belief that the only way for someone with his psychological characteristics to stop drinking is to join Alcoholics Anonymous. Allen is also an atheist. However, he believes that if he joins Alcoholics Anonymous, his psychological characteristics are such that he will be induced by social pressure to believe in God. Because he’s an atheist, he believes that if that belief change happens, it’ll be because his reasoning process will be warped by social pressure, and his new beliefs will be false and (more importantly) unwarranted by the evidence. 

Let’s assume that all of Allen’s present beliefs are warranted by the evidence — that they’re rational by the standards of belief that epistemically competent agents hold.  Allen is, in effect, choosing to cause himself to adopt a belief that would be false and irrational by his current lights, in order to bring about better personal consequences.  But it’s hard to call Allen’s decision wrong. 

If we think that the belief in God is what causes AA to work — if we think it’s the belief itself that’s operative in bringing about the good consequence, then the AA question is structurally indistinguishable from the problem at the heart of Pascal’s wager: the problem of making our beliefs dependent on consequences, rather than just the evidence. 

So it seems like the AA example gives us some reason to swallow Pascal’s wager, modulo the other objections (like a multiplicity of religions).  But there are arguments on the other side.  For one thing, again, remember that Pascal’s original argument suggests that the evidence is in equipoise.  It’s somewhat plausible to think of consequences as a "tiebreaker" between beliefs that are uncertain in that way.  But it’s less plausible to think that we can sensibly use consequences where evidence is not in equipoise.  One major reason for this is that it’s totally unclear how we might relate consequences and evidence in one unified process of belief formation.  For example, suppose that I think there’s a 70% change that P is true, but that my believing P is true will cause one puppy to die.  Is the death of the puppy worth 20% + epsilon chance of truth, so that I should change my beliefs?  How about two puppies?  What if someone offers me one dollar?  How about a million dollars?  What’s the function to convert badness or goodness of consequence into weight of evidence? 

This is a problem that’s very difficult, and I don’t purport to offer a solution.  But we should think of it as a serious line of objection to the Pascal’s wager type of argument: if consequences are simply inadmissible in belief-formation processes, Pascal’s argument fails on the spot. 

(This is a revised version of a post that I originally wrote a couple of weeks ago, which appears in its original form as a lengthy excursus on doxastic voluntarism on my personal blog, Uncommon Priors.  If you’re interested, you might check that out, though it’s less sound, I think, than the current presentation.)