Belief in Belief

Followup to: Making Beliefs Pay Rent (in Anticipated Experiences)

Carl Sagan once told a parable of a man who comes to us and claims: "There is a dragon in my garage." Fascinating! We reply that we wish to see this dragon - let us set out at once for the garage! "But wait," the claimant says to us, "it is an invisible dragon."

Now as Sagan points out, this doesn't make the hypothesis unfalsifiable. Perhaps we go to the claimant's garage, and although we see no dragon, we hear heavy breathing from no visible source; footprints mysteriously appear on the ground; and instruments show that something in the garage is consuming oxygen and breathing out carbon dioxide.

But now suppose that we say to the claimant, "Okay, we'll visit the garage and see if we can hear heavy breathing," and the claimant quickly says no, it's an inaudible dragon. We propose to measure carbon dioxide in the air, and the claimant says the dragon does not breathe. We propose to toss a bag of flour into the air to see if it outlines an invisible dragon, and the claimant immediately says, "The dragon is permeable to flour."

Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification. But I tell this parable to make a different point: The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse.

Some philosophers have been much confused by such scenarios, asking, "Does the claimant really believe there's a dragon present, or not?" As if the human brain only had enough disk space to represent one belief at a time! Real minds are more tangled than that. As discussed in yesterday's post, there are different types of belief; not all beliefs are direct anticipations. The claimant clearly does not anticipate seeing anything unusual upon opening the garage door; otherwise he wouldn't make advance excuses. It may also be that the claimant's pool of propositional beliefs contains There is a dragon in my garage. It may seem, to a rationalist, that these two beliefs should collide and conflict even though they are of different types. Yet it is a physical fact that you can write "The sky is green!" next to a picture of a blue sky without the paper bursting into flames.

The rationalist virtue of empiricism is supposed to prevent us from this class of mistake. We're supposed to constantly ask our beliefs which experiences they predict, make them pay rent in anticipation. But the dragon-claimant's problem runs deeper, and cannot be cured with such simple advice. It's not exactly difficult to connect belief in a dragon to anticipated experience of the garage. If you believe there's a dragon in your garage, then you can expect to open up the door and see a dragon. If you don't see a dragon, then that means there's no dragon in your garage. This is pretty straightforward. You can even try it with your own garage.

No, this invisibility business is a symptom of something much worse.

Depending on how your childhood went, you may remember a time period when you first began to doubt Santa Claus's existence, but you still believed that you were supposed to believe in Santa Claus, so you tried to deny the doubts. As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green.  Dennett calls this "belief in belief".

And here things become complicated, as human minds are wont to do - I think even Dennett oversimplifies how this psychology works in practice. For one thing, if you believe in belief, you cannot admit to yourself that you only believe in belief, because it is virtuous to believe, not to believe in belief, and so if you only believe in belief, instead of believing, you are not virtuous. Nobody will admit to themselves, "I don't believe the Ultimate Cosmic Sky is blue and green, but I believe I ought to believe it" - not unless they are unusually capable of acknowledging their own lack of virtue. People don't believe in belief in belief, they just believe in belief.

(Those who find this confusing may find it helpful to study mathematical logic, which trains one to make very sharp distinctions between the proposition P, a proof of P, and a proof that P is provable.  There are similarly sharp distinctions between P, wanting P, believing P, wanting to believe P, and believing that you believe P.)

There's different kinds of belief in belief. You may believe in belief explicitly; you may recite in your deliberate stream of consciousness the verbal sentence "It is virtuous to believe that the Ultimate Cosmic Sky is perfectly blue and perfectly green." (While also believing that you believe this, unless you are unusually capable of acknowledging your own lack of virtue.) But there's also less explicit forms of belief in belief. Maybe the dragon-claimant fears the public ridicule that he imagines will result if he publicly confesses he was wrong (although, in fact, a rationalist would congratulate him, and others are more likely to ridicule him if he goes on claiming there's a dragon in his garage). Maybe the dragon-claimant flinches away from the prospect of admitting to himself that there is no dragon, because it conflicts with his self-image as the glorious discoverer of the dragon, who saw in his garage what all others had failed to see.

If all our thoughts were deliberate verbal sentences like philosophers manipulate, the human mind would be a great deal easier for humans to understand. Fleeting mental images, unspoken flinches, desires acted upon without acknowledgement - these account for as much of ourselves as words.

While I disagree with Dennett on some details and complications, I still think that Dennett's notion of belief in belief is the key insight necessary to understand the dragon-claimant. But we need a wider concept of belief, not limited to verbal sentences. "Belief" should include unspoken anticipation-controllers.  "Belief in belief" should include unspoken cognitive-behavior-guiders. It is not psychologically realistic to say "The dragon-claimant does not believe there is a dragon in his garage; he believes it is beneficial to believe there is a dragon in his garage."  But it is realistic to say the dragon-claimant anticipates as if there is no dragon in his garage, and makes excuses as if he believed in the belief.

You can possess an ordinary mental picture of your garage, with no dragons in it, which correctly predicts your experiences on opening the door, and never once think the verbal phrase There is no dragon in my garage. I even bet it's happened to you - that when you open your garage door or bedroom door or whatever, and expect to see no dragons, no such verbal phrase runs through your mind.

And to flinch away from giving up your belief in the dragon - or flinch away from giving up your self-image as a person who believes in the dragon - it is not necessary to explicitly think I want to believe there's a dragon in my garage. It is only necessary to flinch away from the prospect of admitting you don't believe.

To correctly anticipate, in advance, which experimental results shall need to be excused, the dragon-claimant must (a) possess an accurate anticipation-controlling model somewhere in his mind, and (b) act cognitively to protect either (b1) his free-floating propositional belief in the dragon or (b2) his self-image of believing in the dragon.

If someone believes in their belief in the dragon, and also believes in the dragon, the problem is much less severe.  They will be willing to stick their neck out on experimental predictions, and perhaps even agree to give up the belief if the experimental prediction is wrong - although belief in belief can still interfere with this, if the belief itself is not absolutely confident.  When someone makes up excuses in advance, it would seem to require that belief, and belief in belief, have become unsynchronized.

Comments

[Anna]

Eliezer, my understanding is: "belief is to believe in something".

Whether you call it science fiction, heuristics, overcoming bias, history, a belief is a belief.
You can't prove belief as it's self-subjective.
You can't tell someone what they feel is wrong.
Each individual has there equation when it comes to understanding the "dragon" within themself.
If dragons can't be verified as they have never been verified based on history, why do people still feel the need to believe in dragons and continue to discuss the subject and be fascinated by it?

Just Curious
Anna

[Eliezer Yudkowsky]

Anna, this blog is too advanced for you and you should not be commenting on it. Go read The Simple Truth until you understand the relation between a map and the territory.

[EDIT: I deleted an additional comment from Anna in this thread.]

[Jan Bussey]

Oh great, now I'm going to think "There's no dragon in my garage" every time I open my garage door for the next week...

[Jan Bussey]

I enjoyed The Simple Truth, thanks for linking it.

[["If the pebbles didn't do anything," says Autrey, "our ISO 9000 process efficiency auditor would eliminate the procedure from our daily work."]]

This "ISO 9000" hypothesis has not been supported by direct observation, unfortunately...

[Pete Carlton]

From the post:

While I disagree with Dennett on some details and complications, I still think that Dennett's notion of belief in belief is the key insight necessary to understand the dragon-claimant. But we need a wider concept of belief, not limited to verbal sentences.

If you've read Dennett on beliefs, you'll appreciate that this "wider concept" based on behavior and predictability is really at the heart of things.

I think it is very difficult to attribute a belief in dragons to this "dragon-believer". Only a small subset of his actions - those involving verbal avowals - make sense if you attribute a belief in dragons to him. There is a conflict with the remainder of his beliefs, as can be seen when he nonchalantly enters his garage, or confabulates all sorts of reasons why his dragon can't be demonstrated.

But as you have shown, everything makes sense if you attribute a related, but slightly different belief, namely "I should avow a genuine, heartfelt belief in dragons". Perhaps we can say that this man (and the religious man, since this is the real point) doesn't just believe in belief, but they believe that they believe. He tries to make a second-order belief do the work of a first-order belief.

[Senthil]

How does this compare with Popper's theory? In the instance above, it's clear that belief in belief doesn't make sense. But things may not be as clear. Won't an event with low probability look like the invisible dragon before it happens?

[Hopefully Anonymous]

Anna,
If you're talking about real dragons, the theory that made the most intuitive sense to me (I think I read it in an E.O. Wilson writing?) is that dragons are an amalgamation of things we've been naturally selected to biologically fear: snakes and birds of prey (I think rats may have also been part of the list). Dragons don't incorporate an element of them that looks like a handgun or a piping hot electric stove, probably because they're too new as threats for us to be naturally selected to fear things with those properties.

[Hopefully Anonymous]

Eliezer,
Very interesting post. I'll try to respond when I've had time to read it more closely and to digest it.

[Rick Smith]

I want my wife to read this, but I don't think she'd believe it.

[Nick Tarleton]

This post helps me understand some of the most infuriating phrases I ever hear (which the title immediately reminded me of): "it doesn't matter what you believe as long as you believe something", "everyone has to believe in something", "faith is a virtue", &c. It makes sense that if a person's second-order belief is stronger than their first-order belief, they would say things like that.

[Shocked]

Was the reply to Anna serious? That's outrageous.

[Eliezer Yudkowsky]

Shocked, it wasn't my first interaction with her.

[John Mark Rozendaal]

I like Eliezer's essay on belief very much.
I've been thinking about the role of belief in religion. (For the sake of full disclosure, my background is Calvinist.) I wonder why Christians say, "We believe in one God," as if that were a particularly strong assertion. Wouldn't it be stronger to say, "We know one God?" What is the difference between belief and knowledge? It seems to me that beliefs are usually based on no data. Most people who believe in a god do so in precisely the same way that they might believe in a dragon in the garage. People are comfortable saying that they know something only when they can refer to supporting data. Believers are valiantly clinging to concepts for which the data is absent. Most people who believe in a god do so in precisely the same way that they might believe in a dragon in the garage.

Regarding the dialogue between the dragon claimant and his challengers, why didn't the challengers simply ask the claimant, "Why do you say that there is an invisible, inaudible, non-respiriating, flour-permeable dragon in your garage?"

[brent]

I've got a double garage... what if the dragon sneaks out one door while I'm coming in through the other door, then comes in behind me through the second door while I look for it outside the first door?? Dragons everywhere now!!

The parable was original with Antony Flew whose Theology and Falsification can be found here http://www.stephenjaygould.org/ctrl/flew_falsification.html

[eclectic]

The parable your refer to by Sagan I think should be attributed to Antony Flew whose Theology and Falsification is available online.
http://www.stephenjaygould.org/ctrl/flew_falsification.html

[Chris]

Trying to work out the biases of the new antispam filter. Frequency of comments from same individual in same thread ?

[Chris]

That lastr one got through, so let's try : Random malfunction ?

[Digital]

Anna:
Whether you call it science fiction, heuristics, overcoming bias, history, a belief is a belief.
You can't prove belief as it's self-subjective.
--That only makes for more of a reason it should only be self affecting, too many people try to influance the actions of others bases on their dragons

You can't tell someone what they feel is wrong.
--Yes I can. "There's a dragon in my bathroom"... (careful examination of the bathroom)... "No, there isn't, you're wrong."

Each individual has there equation when it comes to understanding the "dragon" within themself.
--And it needs to be overcome with logic and reason.

If dragons can't be verified as they have never been verified based on history, why do people still feel the need to believe in dragons and continue to discuss the subject and be fascinated by it?
--simply because they were raised with it, taught to dis-belive any evidence provided, and been shown that those who disagree with it are 'out to get them'

[Richard Hollerith]

If there exists a conventional syntax for quoting another comment, I think a commenter should use that convention rather than invent his own syntax, don't you?

[reasonable robinson]

There is a distinction between Belief and Knowledge. We can believe things that are untru and disbelieve things that are true.