One of the schools of cognitive psychology that addresses biases describes "system 1" and "system 2" thinking, where "system 1" is everyday automatic processing, deciding by intuition, relying on heuristics, and totally filled with biases, and "system 2" is thoughtful and careful consideration, logical and methodical. But this seems inadequate, because we can slip into automatic cognitive patterns even when we are consciously trying to be careful.

A few examples from personal experience below the fold…

I experienced this directly when playing the gatekeeper in an AI-box experiment arranged with another OB reader. I agreed not to give the details of what happened, but I think I can say this much without violating that agreement: I found my own mind responding to the person playing the AI as if he were actually an AI. Even though I knew it was a person who was playing the role of an AI, and even though I knew it was perfectly permissible to completely disregard everything my partner/opponent was saying and recite Blake poetry at him, something in the fact of the assigned roles — some kind of automatic suspension of disbelief — caused me to think as if there was actually an AI on the other end of the internet. I still won, but it was surprisingly difficult.  I think that surprising difficulty came about in part because I responded in that way: I took what my partner was saying <i>seriously</i>, even though there was no need to do so. 

Similarly, I was recently on the receiving end of a bizarre bit of rote behavior. Someone did something obnoxious, I caught him and called him out, and he tried to lie his way out of it with the classic phrase "I have no idea what you're talking about." I'm pretty sure nobody has ever uttered that phrase honestly — it's like something that's culturally encoded in us: someone calls us out on something, we activate the Outraged Denial Subroutine, and the phrase "I have no idea what you're talking about" flies out of our mouths (or out of an e-mail, which took some time to write, from an intelligent person!). (Dear readers: ask yourself whether you have ever used that phrase sincerely.) But do we believe it by rote too?

A third example: I play a lot of chess, particularly three-minute speed chess on FICS. At that speed, there's no time really to calculate long lines of moves, there's just time to play tactical swipes that you see (for non chess-players, a tactic is a short series of moves that are meant to achieve some kind of advantage, usually material — like attacking two of your opponent's pieces at once). There's also time to follow intuitions: to move pieces into positions that you know are strategically good, or play intuitive attacks (sacrificing material when the enemy king looks vulnerable, for example). Chess is a favorite game for studying bounded rationality and the ways that skilled versus unskilled players think — Herbert Simon influentially thought that the difference between grandmasters and ordinary patzers mostly was in the number of "chunks" — of memorized positions — accessible to immediate cognition. But it seems like there's more, in this sense. I notice in three-minute chess that I have tactically good periods and bad periods, even thought my overall (low) level of skill is presumably the same throughout, and even though I am trying just as hard. Sometimes, I play by pure intuition, and sometimes I have a calculation capacity available to me and can perform tactics, and there's no visible difference between those times, except that I lose a lot in the first case and win a lot in the second.  Chess players talk a lot about stamina, and this might be what they mean: the mental discipline to play well might just run out after a while.

This is all very strange, and it makes me wonder if we can think properly even if we try. How much of rationality — of being a good Bayesian Ninja or whatever — isn't about intelligence, or knowing how to think, but about having the self-control and discipline to exercise those capacities? And what does it mean for our attempts to become more rational if, as a lot of recent psychological research has been suggesting, our self-control generally is a limited resource?

How can we overcome rote cognition, if it sticks around even when we're trying our best to be mentally alert and careful?

the pope said the book “explained with great clarity” that “an interreligious dialogue in the strict sense of the word is not possible.” In theological terms, added the pope, “a true dialogue is not possible without putting one’s faith in parentheses.”

What are we to make of a statement like this?

Let’s start with the basics. 

Basic 1: Theological claims are beliefs — they are statements with propositional content that refer to the world.  "There is a God."  "God loves me."  "The world was created ten thousand years ago."  One might reinterpret religion to strip away the propositional content, but then one loses everything that makes religion different from any random social activity. 

Basic 2:  Beliefs should be held for reasons.  Many of our beliefs are not held for reasons, in the sense that I’m using the term (normative, rather than explanatory reasons).  As Robin says in the linked post, those beliefs — random beliefs, beliefs that are contingent on one’s social and physical history rather than on the sorts of things that actually justify beliefs (evidence, argument) ought to be rejected.  This includes beliefs that are based on un-swamped priors from, e.g., how one was raised. 

Basic 3: Beliefs have objective and universal truth-values.  It is flat-out incoherent to utter sentences like "God exists for me but not for you."  This might not be true about everything, but it is manifestly true about ontological claims about the world, historical claims, etc. 

In light of those three basics, what can we make of the pope’s statement?  I take there to be four possibilities for making sense of it. 

1.  The pope believes that his beliefs are not held for reasons.  This, of course, counsels revising those beliefs. 

2.  The pope believes that his beliefs are held for reasons, but that the beliefs of those with whom he disagrees are not. Given that his beliefs and the beliefs of, say, Muslims, are roughly on an epistemic par, this belief is unwarranted. 

3.  The pope believes that his beliefs, and those of his interlocutors, are underdetermined by the available reasons.  That is, the evidence and arguments are consistent with both his beliefs and those of his interlocutors.  If that’s the case, however, it seems to call for his putting equal credence in his beliefs and in those of his interlocutors.

4.  He believes he has access to reasons that are fundamentally incommunicable.  Gnosis.  There are two worries about this claim.  First, is truly incommunicable belief really impossible?  Agents ought to be able to communicate the fact of their gnosis: the pope ought to be able to turn to the mullah and say "I experienced a gnosis, and so did these millions of other people," and that ought to count for the mullah, if the mullah holds his beliefs for reasons (that is, is a bayesian).  Second, if it doesn’t count for the mullah, maybe it’s because the mullah experienced a gnosis too, and that possibility, of course, ought to count for the pope.

Any way we take it, it seems like the pope’s evident belief that it’s impossible to discuss religious differences with those who have different beliefs ought to lead the pope to reduce his credence in his own beliefs.  The pope ought not to be Catholic. 

All this stuff is old news.  But, I wouldn’t expect myself to fall for it (which is the point of this post: I did).  I’m a pretty cynical and suspicious guy, a cynicism and suspicion that rises to almost downright paranoia when it comes to marketing.  (I’ve been known to heavily discount a health threat the moment someone starts selling a product to protect against it, for example.)  I flatter myself by thinking I’m somewhat intelligent.  And I’m well aware of the above research. 

Yet every few weeks until today, I’d walk into a Taco Bell and order one of those combo meals.  This is so even though I often don’t particularly want one of the items on the combo — I’m usually fairly indifferent between, say, having a soda and just drinking water.  Since water’s free and soda isn’t, rationally, I should just drink water every time.  So why do I order the combo meal?  Well, it’s in a combo meal — presumably, it’s cheaper than buying the items separately.  I’m saving money!*  Or, at least, this is the rationalization my brain would supply, on a level just below consciousness except on those rare, fleeting, and unproductive moments when I’d bother to think before ordering.**

Recently, in order to live a little healthier, I made a firm decision to stop consuming sodas.  So it was actually easy to figure out how much I was "saving" by ordering the combo meal instead of all three items.***

Guess how much I saved.  Go ahead.  Guess.  In the comments, even, if you want (status points to the first person who gets it right).  Highlight the space between the brackets to see, after you’ve guessed. 

[Combo meal savings over ordering all three items separately: $0.08.  Extra combo meal cost over ordering just the two items I wanted: $1.61]

I fell for this kind of stupidity even though I know the research.  Do you? 

I really think this bears emphasis.  I know this research really well, and I have known it for over a decade.  If they can get me, they can get anyone.  Everyone, even serious experts, even the guy who largely invented the study of these common biases, can fall prey to this kind of thing.  Dare you think you’re exempt? 

Do you think maybe this contributes to our obesity problem? Or do you still think that overeating can casually be described as a "free choice" for which people are personally responsible?  (While Taco Bell profits from selling unwanted sodas…)

Policy message: if even informed people can be suckered like this, maybe it is time for a legislative solution? 

—-
* Even if this were true, one isn’t saving money if one buys something one doesn’t actually want, in order to get it at a discount!  There’s another sinister anchoring effect at work: one’s comparison case becomes "buying all three items at full price," not "buying only the two items one wants" (which is invariably cheaper than buying all three). 

** That is, I’d basically process sodas as free because of the imagined "combo meal discount", and hence be indifferent between them and water, even though it’s actually more expensive.  Then, ordering the combo meal is the path of least resistance.

*** That’s not so easy to do unless you actually order the items individually and compare: not all the prices for individual items show up on the menu.  Fortunately, the sodas do, so I could add the price of a soda to what I paid for the other two items and thereby learn the non-combo price for the three items. 

- John Stuart Mill, from Utility of Religion

Much electronic ink has been spilled on this blog about Pascal’s wager.  Yet, I don’t think that the central issue, and one that relates directly to the mission of this blog, has been covered.  That issue is this: there’s a difference between the requirements for good (rational, justified) belief and the requirements for good (rational, prudent — not necessarily moral) action.

Presented most directly: good belief is supposed to be truth and evidence-tracking.  It is not supposed to be consequence-tracking.  We call a belief rational to the extent it is (appropriately) influenced by the evidence available to the believer, and thus maximizes our shot at getting the truth.  We call a belief less rational to the extent it is influenced by other factors, including the consequences of holding that belief.  Thus, an atheist who changed his beliefs in response to the threat of torture from the Spanish Inquisition cannot be said to have followed a correct belief-formation process. 

On the other hand, good action is supposed (modulo deontological moral theories) to be consequence-tracking.  The atheist who professes changed beliefs in response to the threat of torture from the Spanish Inquisition can be said to be acting prudently by making such a profession.

A modern gloss on Pascal’s wager might be understood less as an argument for the belief in God than as a challenge to that separation.  If, Modern-Pascal might say, we’re in an epistemic situation such that our evidence is in equipoise (always keeping in mind Daniel Griffin’s apt point that this is the situation presumed by Pascal’s argument), then we ought to take consequences into account in choosing our beliefs. 

There seem to be arguments for and against that position… 

In its favor we can imagine situations where it’s not the nastiness of an all-knowing deity that makes our beliefs consequential, but something about our own psychologies.  Imagine Allen. He’s an alcoholic. He makes an all-things-considered judgment that it would be best for him to stop drinking. He also holds the belief that the only way for someone with his psychological characteristics to stop drinking is to join Alcoholics Anonymous. Allen is also an atheist. However, he believes that if he joins Alcoholics Anonymous, his psychological characteristics are such that he will be induced by social pressure to believe in God. Because he’s an atheist, he believes that if that belief change happens, it’ll be because his reasoning process will be warped by social pressure, and his new beliefs will be false and (more importantly) unwarranted by the evidence. 

Let’s assume that all of Allen’s present beliefs are warranted by the evidence — that they’re rational by the standards of belief that epistemically competent agents hold.  Allen is, in effect, choosing to cause himself to adopt a belief that would be false and irrational by his current lights, in order to bring about better personal consequences.  But it’s hard to call Allen’s decision wrong. 

If we think that the belief in God is what causes AA to work — if we think it’s the belief itself that’s operative in bringing about the good consequence, then the AA question is structurally indistinguishable from the problem at the heart of Pascal’s wager: the problem of making our beliefs dependent on consequences, rather than just the evidence. 

So it seems like the AA example gives us some reason to swallow Pascal’s wager, modulo the other objections (like a multiplicity of religions).  But there are arguments on the other side.  For one thing, again, remember that Pascal’s original argument suggests that the evidence is in equipoise.  It’s somewhat plausible to think of consequences as a "tiebreaker" between beliefs that are uncertain in that way.  But it’s less plausible to think that we can sensibly use consequences where evidence is not in equipoise.  One major reason for this is that it’s totally unclear how we might relate consequences and evidence in one unified process of belief formation.  For example, suppose that I think there’s a 70% change that P is true, but that my believing P is true will cause one puppy to die.  Is the death of the puppy worth 20% + epsilon chance of truth, so that I should change my beliefs?  How about two puppies?  What if someone offers me one dollar?  How about a million dollars?  What’s the function to convert badness or goodness of consequence into weight of evidence? 

This is a problem that’s very difficult, and I don’t purport to offer a solution.  But we should think of it as a serious line of objection to the Pascal’s wager type of argument: if consequences are simply inadmissible in belief-formation processes, Pascal’s argument fails on the spot. 

(This is a revised version of a post that I originally wrote a couple of weeks ago, which appears in its original form as a lengthy excursus on doxastic voluntarism on my personal blog, Uncommon Priors.  If you’re interested, you might check that out, though it’s less sound, I think, than the current presentation.)

"But!," the anticynic might say, "cynicism is a self-fulfilling prophecy!  If we all believe that politics is run by crooks, we won’t demand better at the voting booth [for example, because we vote strategically for the least offensive guy we think can win rather than the one we trust]!  If enough people are optimistic, your optimism will be self-fulfilling too!" 

So imagine the following belief/payoff correspondences.  If you hold a true cynical belief, you get payoff A.  If you hold a false cynical belief (cynicism in a nice world), you get payoff B.  If you hold a true optimistic belief, you get payoff C, and if you hold a false optimistic belief, you get payoff D.  Suppose C>A>B>D (or C>A>D>B — it doesn’t matter.)  And suppose that the world is nice if M people are optimistic (where N is the number of people in the world, and N>M>1) and nasty otherwise.

Anyone who knows game theory will immediately see that this world amounts to a coordination game with two nash equilibria: everyone optimistic in a nice world and everyone cynical in a nasty world.  And the nice world equilibrium has higher payoffs for all.

Now suppose we’re in a nasty world.  How do we get to the nice world?  It seems like we’d do best if someone came along and deceived at least M people into thinking we’re in the nice world already! 

This shows us that not only can individually rational behavior be collectively suboptimal, so can individually rational (truth-maximizing) belief.  Should we support demagoguery? 

I imagine the self-fulfilling false belief problem works on some individual cases too.  For example, suppose I have more success in dating if I’m confident?  Suppose I’m a person who has poor success in dating.  True beliefs for me are not confident ones, but I’ll do better if I adopt falsely confident beliefs, which will then be retroactively justified by the facts.  Should I engage in self-deception? 

At the most basic level (which is all we need for present purposes), an argument is nothing but a chain of dependence between two or more propositions.  We say something about the truth value of the set of propositions {P1…Pn}, and we assert that there’s something about {P1…Pn} such that if we’re right about the truth values of that set, we ought to believe something about the truth value of the set {Q1…Qn}. 

If we have that understanding of what it means to make an argument, then we can see that an argument doesn’t necessarily have any connection to the universe outside itself.  The utterance "1. all bleems are quathes, 2. the youiine is a bleem, 3. therefore, the youiine is a quathe" is a perfectly logically valid utterance, but it doesn’t refer to anything in the world — it doesn’t require us to change any beliefs.  The meaning of any argument is conditional on our extra-argument beliefs about the world.

One important use of this principle is reflected in the oft-quoted line "one man’s modus ponens in another man’s modus tollens."  Modus ponens is a classical form of argument: 1. A–>B.  2.  A. 3.  .: B.  Modus tollens is this: 1.  A–>B.  2. ¬B.  3. .: ¬A.  Both are perfectly valid forms of argument!  (For those who aren’t familiar with the standard notation, the horizontal line is meant to indicate negation.)  Unless you have some particular reason outside the argument to believe either A or B, you don’t know whether the claim A–>B means that B is true, or that A isn’t true! 

Why am I elucidating all this basic logic, which almost everyone reading this blog doubtless knows?  It’s a rhetorical tactic: I’m trying to make it salient, to bring it to the top of the cognitive stack, so that my next claim is more compelling.

And that claim is as follows:

Eliezer’s posts about the specks and the torture [1] [2], and the googleplex of people being tortured for a nanosecond, and so on, and so forth, tell you nothing about the truth of your intuitions.

Argument behind the fold…

At most, at most!, Eliezer’s arguments establish an inconsistency between two propositions.  Proposition 1: "utilitarianism is true." Proposition 2: "your intuitions about putting dust specks in people’s eyes, sacred values, etc., to the extent they recommend inflicting a small harm on lots of people rather than a lot of harm on one person, when that the aggregate pain from the first is higher than the aggregate pain from the second, are true."  As I’ve noted before, I don’t think Eliezer has even established that.  (The short version: utilitarianism is a lot more complicated than that, it ain’t easy to figure out how to aggregate harms, it ain’t easy to map those harms onto hedonic states like pleasure and pain, etc.) 

But let’s give Eliezer that one, arguendo.  Suppose his argument has established the inconsistency.  In symbols, where P = utilitarianism, and Q = your intuitions about dust specks etc., Eliezer has established ¬P∨¬Q.  (Not P, or not Q.)  It doesn’t establish ¬Q!  Unless there’s more exogenous reason to believe P than there is to believe Q, Eliezer’s argument shouldn’t be any more likely to cause us to disbelieve P than to disbelieve Q.  This is the step that should make your heart sing, now that I’ve primed you with the review of basic logic above.

Now let’s take the next step.  Why should there be more exogenous reason to believe P than to believe Q?  Why might one want to believe that utilitarianism is true? 

This post is already far too long to go over the abstract reasons why one might accept utilitarianism.  But let me make the claim, which you might find plausible, that many of those reasons come down to intuitions.  Those intuitions might be about specific cases which lead to inductive generalizations about rules ("I think it’s better to kill one person than to kill five, and better to torture for a week than torture for a year, therefore, it must be best to maximize pleasure over pain!"), or intuitions directly about the rules ("well, obviously, it’s best to maximize pleasure over pain!").  Regardless, intuitions they be.

And now let’s subjectivize things a little further.  I’ll bet that the vast majority of the people reading this post, people who hold utilitarian beliefs, came to those utilitarian beliefs largely as a result of articulating their moral intuitions, or reading an argument about normative ethics that spoke to their moral intuitions.  Eliezer’s own case is a perfect example: he has expressed his utilitarian beliefs as being a direct consequence of his seemingly intuitive choices. 

And now the final step.  You get your moral intuitions about the dust specks case from wherever it is that your intuitions come from.  You get your utilitarianism from wherever it is that your intuitions come from.   They’re on equal footing — you have no more reason to believe your utilitarian intuitions than you have to believe your dust speck intuitions!  Therefore, by the claims above, Eliezer’s argument shouldn’t cause you to reject your dust specks intuition.

A summary:
1.  An argument establishing that two propositions are inconsistent doesn’t tell you which of those propositions you should reject, unless you have more reason outside the argument to accept one or the other. 
2.  For any two propositions P and Q, if you accept P for only the same reasons you accept Q, you don’t have more reason to accept P than Q.
3.  Your reasons to believe dust specks are better than torture are identical to your reasons to believe utilitarianism is true.
4.  Therefore, an argument (Eliezer’s) establishing that dust specks>torture is inconsistent with utilitarianism doesn’t give you any reason to reject dust specks>torture.

Q.E.D. 

(A couple objections to this argument: 1) "But what if my intuitions about utilitarianism come from many, many cases, and I only have renegade non-utilitarian intuitions about a few cases — doesn’t that mean I should believe my utilitarian intuitions more strongly?"  Answer:  Sure, if and only if you think that the strength of intuitions can be summed that way, and it’s not obvious that’s true.  Also, I can come up with many more cases than just the dust specks where your intuitions likely get non-utilitarian outcomes.  2) I was recently handed a paper where an undergrad argued that the intuitions of utilitarians tend to [always, even] match the results of utilitarian calculations [should she read this post, I invite her to defend that claim in the comments].  If true, that would cause problems… but does anyone actually believe it?) 

This all connects back quite strongly to the point of this blog.  Taking an argument of the form ¬P∨¬Q and concluding, on that basis alone, ¬Q is an error in reasoning, and it’s one that strongly resembles a form of overconfidence — or perhaps expecting short inferential distances. 

That’s where the real fierceness lies.  There’s the naked sword.  There’s the solar plasma: in recognizing the limitations of your arguments, the point where the road — or "The Way" — stops. 

Yet he, too, is not immune from motivated error.  A friend and colleague recently forwarded Kahneman’s Edge answer to me.  Apparently, Kahneman himself was so captivated by the lure of a neat theory to handle some difficulties in hedonic experience that he managed to misinterpret the first set of results!

Our hypothesis was that differences in life circumstances would have more impact on this measure than on life satisfaction.  We were so convinced that when we got our first batch of data, comparing teachers in top-rated schools to teachers in inferior schools, we actually misread the results as confirming our hypothesis.  In fact, they showed the opposite: the groups of teachers differed more in their work satisfaction than in their affective experience at work. This was the first of many such findings: income, marital status and education all influence experienced happiness less than satisfaction, and we could show that the difference is not a statistical artifact.  Measuring experienced happiness turned out to be interesting and useful, but not in the way we had expected.  We had simply been wrong. (Emphasis added)

Social scientists, beware.  If this can happen to Daniel Kahneman, it can happen to anyone.

That’s the wonderful thing about the kinds of utilitarian intuitions that tell us, deep in our brains, that we can aggregate a lot of pain and pleasure of different kinds among different people and come up with some kind of scalar representing the net sum of "utility" to be compared to some other scalar for some other pattern of events in some possible world; the scalars to be compared to determine which world is morally better, and to which world our efforts should be directed. Those intuitions always generate a rationalizable answer.

If we demand that our moral questions have answers of that type, comments like Eliezer’s start to look very appealing. Eliezer says that it’s irrational to "impose and rationalize comfortable moral absolutes in defiance of expected utility." But if that’s so, Eliezer owes us an argument for why moral judgments make sense in terms of expected utility. Or why they make sense in terms of any decision theoretic calculation at all. Or why they have to make sense in terms of any overall algorithmic procedure of any kind. To simply assume that decision theory applies to moral questions, that there’s something — utility, goodness, moral worth, whatever — to maximize is to beg the question right from the start. And that’s bias if anything is.

One can easily argue that making decision theoretic arguments about moral questions is a massive category error. On that view, it’s not irrational to blink away the dust speck even if you believe that no quantity of dispersed dust equals even a moment of torture, and even if there’s some action you could take instead with a 1/3^^^3 chance of saving someone from torture. (We don’t need to mess around with hacks about hyperreal numbers and the like.) Likewise, it’s not irrational to spend 3^^^3 dollars to save a single human life, or to decline to spend that money, because, yup, the sort of rationality that compares values in this way just doesn’t apply to moral questions.

But broader than that, there simply might not be a decision procedure at all. I think the sort of people who are drawn to the sorts of Bentham-by-pocket-calculator reasoning that we see here are revealing a serious discomfort with the idea that there might be moral questions that are not easily resolved by some kind of algorithm. Or that might not be easily resolved period, or resolved at all. There might be moral paradoxes. There might be irreconcilable moral conflicts. Normative truth might not follow the same laws that descriptive truth follows.

There’s a classic example in moral philosophy, thanks to Sartre. A young man in Vichy France has to choose between caring for his mother and fighting for the Resistance. Those who drink the decision theoretic kool-aid are committed to the notion that there’s some kind of calculation that’s possible in principle to determine which duty has a higher value. But it’s really hard to swallow that claim. Many people feel very strongly, when confronted by that example, that the poor young man is blamable for any decision he makes, for any decision must neglect some duty. He has had bad moral luck.

Incidentally, that example also shows that anyone who thinks moral absolutes are "comfortable" is seriously misinformed. Consider torture again. It’s hardly comfortable to stick to the morally absolute position that one can’t (in the classic hypothetical case) torture a terrorist to find out the location of the nuclear bomb in New York City. That’s a hard position. Anyone taking that position suffers, internally and externally, and suffers a lot. It’s a lot easier to fall into "expected utility" and justify the torture. So please don’t insult people who accept deontological moral theories by suggesting that they’re (we’re) hiding in a "comfortable" position.

Critics of the measures usually invoke what Robin has been calling an “unseen bias used to justify a seen bias.” They claim that people in a community are likely to discriminate against known convicted sex offenders along most or all dimensions of life, and that this discrimination, apart from being unjust on its own, may actually make it harder for those criminals to rehabilitate and become productive and law-abiding members of society. These problems, say critics, justify creating a “bias” by keeping the costs of acquiring this information high for the public.

These laws may offer an angle on the unseen bias/seen bias question more generally. It seems to me that the critics of Megan’s Law are right, for at least the following reasons:

- People economize on information. Those who view information about a neighbor’s sex crimes as more valuable than foregone information can be expected to visit the website, look through the newspaper listings, etc. But copious psychological research has documented that people greatly overestimate dangers from uncontrolled sources and from unnatural sources. (This white paper sums much of it up.) Accordingly, people can be expected to overestimate the safety benefits of knowing who is a sex offender, and accordingly over-acquire sex offender information. The opportunity cost for this information is real, fine-grained information about individuals, and the likely result is that convicted sex offenders will be subject to a level of social exclusion greater than would be justified by a rational estimate of their risk against the possible benefits from interacting with them.

- There has in fact been some astonishing discrimination against convicted sex offenders. One particularly dramatic example will suffice: residency restrictions have been enacted prohibiting offenders from living within a given (large) number of feet of schools, day care facilities, video arcades, and numerous other places. One court found that Iowa’s law forbade offenders from living in 77% of the state’s housing units — and pretty much everywhere except on farms. (Source: Prawfs.) I think most reasonable people would agree that barring someone from 77% of the residences in a state is prima facie an irrationally over-fearful response to these dangers. (If nothing else, it seems pretty clear that anyone who is dangerous enough to be kept out of three-fourths of the state is also dangerous enough to be kept in prison, or at least civilly committed.) But there’s a lot of this — in addition to legislative disabilities, these offenders are evicted, fired from jobs, and so forth, and not just when their housing or their job involves work with children or other vulnerable populations.

Here, as in (I strongly suspect) many other cases, the “unseen bias” isn’t so hard to see after all, if we bother to open our eyes.

Richard and Jerry are Bayesians with common priors. Richard is an atheist. Jerry was an atheist, but then he had an experience which he believes gives him certain knowledge of the following proposition (LGE): “There is a God, and he loves me.” Jerry’s experiences his knowledge as gnosis: a direct experience of divine grace that bestowed certain knowledge, period, and not conditioned on anything else at all. (Some flavors of Christianity and many other religions claim experiences like this, including prominently at least some forms of Buddhism.) In addition to believing certain knowledge of LGE, Jerry’s gnosis greatly modifies his probability estmates of almost every proposition in his life. For example, before the gnosis, the Christian Bible didn’t significantly impact the subjective probabilities of the propositions it is concerned with. Now it counts very heavily.

Richard and Jerry are aware of a disagreement as to the probability of LGE, and also the truth of the various things in the Bible. They sit down to work it out.

It seems like the first step for Richard and Jerry is to merge their data. Otherwise, Jerry has to violate one rule of rationality or another: since his gnosis is only consistent with the certainty of LGE, he can either discard plainly relevant data (irrational) or fail to reach agreement (irrational). Richard does his best to replicate the actions that got the gnosis into Jerry’s head: he fasts, he meditates on the koans, he gives money to the televangelist. But no matter what he does, Richard can not get the experience that Jerry had. He can get Jerry’s description of the experience, but Jerry insists that the description falls woefully short of the reality — it misses a qualitative aspect, the feeling of being “touched,” the bestowal of certain knowledge of the existence of a loving God.

Is it in principle possible for Richard and Jerry to reach agreement on their disputed probabilities given a non-transmissible experience suggesting to Jerry that P(LGE)=1?