Category Archives: Epistemology

You Provably Can’t Trust Yourself

Followup toWhere Recursive Justification Hits Bottom, Löb’s Theorem

Peano Arithmetic seems pretty trustworthy.  We’ve never found a case where Peano Arithmetic proves a theorem T, and yet T is false in the natural numbers.  That is, we know of no case where []T ("T is provable in PA") and yet ~T ("not T").

We also know of no case where first order logic is invalid:  We know of no case where first-order logic produces false conclusions from true premises. (Whenever first-order statements H are true of a model, and we can syntactically deduce C from H, checking C against the model shows that C is also true.)

Combining these two observations, it seems like we should be able to get away with adding a rule to Peano Arithmetic that says:

All T:  ([]T -> T)

But Löb’s Theorem seems to show that as soon as we do that, everything becomes provable.  What went wrong?  How can we do worse by adding a true premise to a trustworthy theory?  Is the premise not true – does PA prove some theorems that are false?  Is first-order logic not valid – does it sometimes prove false conclusions from true premises?

Continue reading "You Provably Can’t Trust Yourself" »

GD Star Rating
loading...

The Cartoon Guide to Löb’s Theorem

Lo!  A cartoon proof of Löb’s Theorem!

Löb’s Theorem shows that a mathematical system cannot assert its own soundness without becoming inconsistent.  Marcello and I wanted to be able to see the truth of Löb’s Theorem at a glance, so we doodled it out in the form of a cartoon.  (An inability to trust assertions made by a proof system isomorphic to yourself, may be an issue for self-modifying AIs.)

It was while learning mathematical logic that I first learned to rigorously distinguish between X, the truth of X, the quotation of X, a proof of X, and a proof that X’s quotation was provable.

The cartoon guide follows as an embedded Scribd document after the jump, or you can download from yudkowsky.net as a PDF file.  Afterward I offer a medium-hard puzzle to test your skill at drawing logical distinctions.

Continue reading "The Cartoon Guide to Löb’s Theorem" »

GD Star Rating
loading...

Hot Air Doesn’t Disagree

Followup toThe Bedrock of Morality, Abstracted Idealized Dynamics

Tim Tyler comments:

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox’s stomach. They may argue passionately about the rabbit’s fate – and even stoop to violence.

Boy, you know, when you think about it, Nature turns out to be just full of disagreement.

Rocks, for example, fall down – so they agree with us, who also fall when pushed off a cliff – whereas hot air rises into the air, unlike humans.

I wonder why hot air disagrees with us so dramatically.  I wonder what sort of moral justifications it might have for behaving as it does; and how long it will take to argue this out.  So far, hot air has not been forthcoming in terms of moral justifications.

Physical systems that behave differently from you usually do not have factual or moral disagreements with you.  Only a highly specialized subset of systems, when they do something different from you, should lead you to infer their explicit internal representation of moral arguments that could potentially lead you to change your mind about what you should do.

Continue reading "Hot Air Doesn’t Disagree" »

GD Star Rating
loading...

“Arbitrary”

Followup toInseparably Right; or, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

One of the experiences of following the Way is that, from time to time, you notice a new word that you have been using without really understanding.  And you say:  "What does this word, ‘X’, really mean?"

Perhaps ‘X’ is ‘error’, for example.  And those who have not yet realized the importance of this aspect of the Way, may reply:  "Huh? What do you mean?  Everyone knows what an ‘error’ is; it’s when you get something wrong, when you make a mistake."  And you reply, "But those are only synonyms; what can the term ‘error’ mean in a universe where particles only ever do what they do?"

It’s not meant to be a rhetorical question; you’re meant to go out and answer it.  One of the primary tools for doing so is Rationalist’s Taboo, when you try to speak without using the word or its synonyms – to replace the symbol with the substance.

So I ask you therefore, what is this word "arbitrary"?  Is a rock arbitrary?  A leaf?  A human?

How about sorting pebbles into prime-numbered heaps?  How about maximizing inclusive genetic fitness?  How about dragging a child off the train tracks?

How can I tell exactly which things are arbitrary, and which not, in this universe where particles only ever do what they do?  Can you tell me exactly what property is being discriminated, without using the word "arbitrary" or any direct synonyms?  Can you open up the box of "arbitrary", this label that your mind assigns to some things and not others, and tell me what kind of algorithm is at work here?

Continue reading "“Arbitrary”" »

GD Star Rating
loading...

Abstracted Idealized Dynamics

Followup toMorality as Fixed Computation

I keep trying to describe morality as a "computation", but people don’t stand up and say "Aha!"

Pondering the surprising inferential distances that seem to be at work here, it occurs to me that when I say "computation", some of my listeners may not hear the Word of Power that I thought I was emitting; but, rather, may think of some complicated boring unimportant thing like Microsoft Word.

Maybe I should have said that morality is an abstracted idealized dynamic.  This might not have meant anything to start with, but at least it wouldn’t sound like I was describing Microsoft Word.

How, oh how, am I to describe the awesome import of this concept, "computation"?

Perhaps I can display the inner nature of computation, in its most general form, by showing how that inner nature manifests in something that seems very unlike Microsoft Word – namely, morality.

Consider certain features we might wish to ascribe to that-which-we-call "morality", or "should" or "right" or "good":

• It seems that we sometimes think about morality in our armchairs, without further peeking at the state of the outside world, and arrive at some previously unknown conclusion.

Someone sees a slave being whipped, and it doesn’t occur to them right away that slavery is wrong.  But they go home and think about it, and imagine themselves in the slave’s place, and finally think, "No."

Can you think of anywhere else that something like this happens?

Continue reading "Abstracted Idealized Dynamics" »

GD Star Rating
loading...

The Problem at the Heart of Pascal’s Wager

It is a most painful position to a conscientious and cultivated mind to be drawn in contrary directions by the two noblest of all objects of pursuit — truth and the general good.  Such a conflict must inevitably produce a growing indifference to one or other of these objects, most probably to both.

– John Stuart Mill, from Utility of Religion

Much electronic ink has been spilled on this blog about Pascal’s wager.  Yet, I don’t think that the central issue, and one that relates directly to the mission of this blog, has been covered.  That issue is this: there’s a difference between the requirements for good (rational, justified) belief and the requirements for good (rational, prudent — not necessarily moral) action.

Presented most directly: good belief is supposed to be truth and evidence-tracking.  It is not supposed to be consequence-tracking.  We call a belief rational to the extent it is (appropriately) influenced by the evidence available to the believer, and thus maximizes our shot at getting the truth.  We call a belief less rational to the extent it is influenced by other factors, including the consequences of holding that belief.  Thus, an atheist who changed his beliefs in response to the threat of torture from the Spanish Inquisition cannot be said to have followed a correct belief-formation process. 

On the other hand, good action is supposed (modulo deontological moral theories) to be consequence-tracking.  The atheist who professes changed beliefs in response to the threat of torture from the Spanish Inquisition can be said to be acting prudently by making such a profession.

A modern gloss on Pascal’s wager might be understood less as an argument for the belief in God than as a challenge to that separation.  If, Modern-Pascal might say, we’re in an epistemic situation such that our evidence is in equipoise (always keeping in mind Daniel Griffin’s apt point that this is the situation presumed by Pascal’s argument), then we ought to take consequences into account in choosing our beliefs. 

There seem to be arguments for and against that position… 

Continue reading "The Problem at the Heart of Pascal’s Wager" »

GD Star Rating
loading...
Tagged as: , ,

No Logical Positivist I

Followup toMaking Beliefs Pay Rent, Belief in the Implied Invisible

Degrees of Freedom accuses me of reinventing logical positivism, badly:

One post which reads as though it were written in Vienna in the 1920s is this one [Making Beliefs Pay Rent] where Eliezer writes

"We can build up whole networks of beliefs that are connected only to each other – call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens’s ability to build more general and flexible belief networks…  The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict – or better yet, prohibit."

Logical positivists were best known for their verificationism: the idea that a belief is defined in terms of the experimental predictions that it makes.  Not just tested, not just confirmed, not just justified by experiment, but actually defined as a set of allowable experimental results.  An idea unconfirmable by experiment is not just probably wrong, but necessarily meaningless.

I would disagree, and exhibit logical positivism as another case in point of "mistaking the surface of rationality for its substance".

Continue reading "No Logical Positivist I" »

GD Star Rating
loading...