Category Archives: Morality

Augustine’s Paradox of optimal repentance

By Philip Goetz · March 1, 2009 4:09 pm · Comments (72)

Eliezer once wrote this about Newcomb’s problem:

Nonetheless, I would like to present some of my motivations on Newcomb’s Problem – the reasons I felt impelled to seek a new theory – because they illustrate my source-attitudes toward rationality. Even if I can’t present the theory that these motivations motivate…

First, foremost, fundamentally, above all else:

Rational agents should WIN.

As I just commented on another thread, this is faith in rationality, which is an oxymoron.

It isn’t obvious whether there is a rational winning approach to Newcomb’s problem. But here’s a similar, simpler problem that billions of people have believed was real, which I’ll call Augustine’s Paradox (“Lord, make me chaste – but not yet!”)

Most kinds of Christianity teach that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but you sin ten minutes before dying and you don’t repent (Protestantism), or you commit a mortal sin and the priest has already left (Catholicism), you go to Hell. If you’re sinful all your life but repent in your final minute, you go to Heaven.

The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment. But if you repent as part of a plan, it won’t work; you’ll go to Hell anyway. The optimal strategy is to be selfish all your life, without intending to repent, and then repent in your final moments and truly mean it.

I don’t think there’s any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins.

Continue reading "Augustine’s Paradox of optimal repentance" »

GD Star Rating
loading...
Tagged as: , ,

Beware Value Talk

By Robin Hanson · February 16, 2009 7:00 pm · Comments (28)

Last week I wrote:

[For] lawyers supporting clients, engineers presenting designs, accountants presenting financial accounts, or academics presenting analyses, styles are more "no-nonsense."  They avoid colorful flashy emotional visual aids and music, use precise concise technical and unemotional language, make structured and standardized arguments, explicitly summarize and address opposing views, make methods and premises explicit, and warn early of conclusions and structures. … Authors who want to be seen as minimizing the propaganda element of their communications avoid using flashy styles. … [In contrast,] in such propaganda contexts, impressive charismatic leaders tend to speak in simple repetitive eloquent poetic vague emotional language, often with rambling structures, engaging stories, vivid colorful flashy emotional music and visual aids.

Communities of conversation "serious" about working together to make progress in understanding things tend not only to follow the above style conventions, they also tend to follow a key content rule:  avoid arguing basic values. 

Communities that mostly agree on how to evaluate claims can make a lot of progress, and such agreement comes naturally enough when discussion is restricted to "facts" connected to frequent observations and actions.  Such communities can also discuss how to achieve a few commonly accepted values. 

For example, athletes can talk about how to win games, lawyers about winning cases, salesmen about increasing sales, business managers about increasing profits, scientists about accelerating scientific progress, engineers about improving design efficiencies, medical experts about increasing health while decreasing costs, and economists about increasing policy "economic-efficiency." 

In all of these cases, explicit agreement on simply-expressed values allows group conversations to progress effectively toward achieving such values.  Members can specialize, develop and use specialized language and techniques, and evaluate others' contributions to the common cause.  In contrast, groups that freely argue about basic values tend to fragment into like-thinking-cliques focused more on clique loyalty than on fairly crediting informative contributions.

Continue reading "Beware Value Talk" »

GD Star Rating
loading...
Tagged as:

The Thing That I Protect

By Eliezer Yudkowsky · February 7, 2009 2:18 pm · Comments (19)

Followup to:  Something to Protect, Value is Fragile

"Something to Protect" discursed on the idea of wielding rationality in the service of something other than "rationality".  Not just that rationalists ought to pick out a Noble Cause as a hobby to keep them busy; but rather, that rationality itself is generated by having something that you care about more than your current ritual of cognition.

So what is it, then, that I protect?

I quite deliberately did not discuss that in "Something to Protect", leaving it only as a hanging implication.  In the unlikely event that we ever run into aliens, I don't expect their version of Bayes's Theorem to be mathematically different from ours, even if they generated it in the course of protecting different and incompatible values.  Among humans, the idiom of having "something to protect" is not bound to any one cause, and therefore, to mention my own cause in that post would have harmed its integrity.  Causes are dangerous things, whatever their true importance; I have written somewhat on this, and will write more about it.

But still – what is it, then, the thing that I protect?

Friendly AI?  No – a thousand times no – a thousand times not anymore.  It's not thinking of the AI that gives me strength to carry on even in the face of inconvenience.

Continue reading "The Thing That I Protect" »

GD Star Rating
loading...

Interlude with the Confessor (4/8)

By Eliezer Yudkowsky · February 2, 2009 4:11 am · Comments (39)

(Part 4 of 8 in "Three Worlds Collide")

The two of them were alone now, in the Conference Chair's Privilege, the huge private room of luxury more suited to a planet than to space.  The Privilege was tiled wall-to-wall and floor-to-ceiling with a most excellent holo of the space surrounding them: the distant stars, the system's sun, the fleeing nova ashes, and the glowing ember of the dwarf star that had siphoned off hydrogen from the main sun until its surface had briefly ignited in a nova flash.  It was like falling through the void.

Akon sat on the edge of the four-poster bed in the center of the room, resting his head in his hands.  Weariness dulled him at the moment when he most needed his wits; it was always like that in crisis, but this was unusually bad.  Under the circumstances, he didn't dare snort a hit of caffeine – it might reorder his priorities.  Humanity had yet to discover the drug that was pure energy, that would improve your thinking without the slightest touch on your emotions and values.

"I don't know what to think," Akon said.

The Ship's Confessor was standing stately nearby, in full robes and hood of silver.  From beneath the hood came the formal response:  "What seems to be confusing you, my friend?"

"Did we go wrong?" Akon said.  No matter how hard he tried, he couldn't keep the despair out of his voice.  "Did humanity go down the wrong path?"

Continue reading "Interlude with the Confessor (4/8)" »

GD Star Rating
loading...

The Baby-Eating Aliens (1/8)

By Eliezer Yudkowsky · January 30, 2009 7:07 am · Comments (67)

(Part 1 of 8 in "Three Worlds Collide")

This is a story of an impossible outcome, where AI never worked, molecular nanotechnology never worked, biotechnology only sort-of worked; and yet somehow humanity not only survived, but discovered a way to travel Faster-Than-Light:  The past's Future.

Ships travel through the Alderson starlines, wormholes that appear near stars.  The starline network is dense and unpredictable: more than a billion starlines lead away from Sol, but every world explored is so far away as to be outside the range of Earth's telescopes.  Most colony worlds are located only a single jump away from Earth, which remains the center of the human universe.

From the colony system Huygens, the crew of the Giant Science Vessel Impossible Possible World have set out to investigate a starline that flared up with an unprecedented flux of Alderson force before subsiding.  Arriving, the Impossible discovers the sparkling debris of a recent nova – and –

"ALIENS!"

Every head swung toward the Sensory console.  But after that one cryptic outburst, the Lady Sensory didn't even look up from her console: her fingers were frantically twitching commands.

There was a strange moment of silence in the Command Conference while every listener thought the same two thoughts in rapid succession:

Is she nuts?  You can't just say "Aliens!", leave it at that, and expect everyone to believe you.  Extraordinary claims require extraordinary evidence –

And then,

They came to look at the nova too!

Continue reading "The Baby-Eating Aliens (1/8)" »

GD Star Rating
loading...

Three Worlds Collide (0/8)

By Eliezer Yudkowsky · January 30, 2009 7:07 am · Comments (15)

"The kind of classic fifties-era first-contact story that Jonathan Swift might have written, if Jonathan Swift had had a background in game theory."
        — (Hugo nominee) Peter Watts, "In Praise of Baby-Eating"

Three Worlds Collide is a story I wrote to illustrate some points on naturalistic metaethics and diverse other issues of rational conduct.  It grew, as such things do, into a small novella.  On publication, it proved widely popular and widely criticized.  Be warned that the story, as it wrote itself, ended up containing some profanity and PG-13 content.

  1. The Baby-Eating Aliens
  2. War and/or Peace
  3. The Super Happy People
  4. Interlude with the Confessor
  5. Three Worlds Decide
  6. Normal Ending
  7. True Ending
  8. Atonement

PDF version here.

GD Star Rating
loading...

Value is Fragile

By Eliezer Yudkowsky · January 29, 2009 3:46 am · Comments (51)

Followup to:  The Fun Theory Sequence, Fake Fake Utility Functions, Joy in the Merely Good, The Hidden Complexity of Wishes, The Gift We Give To Tomorrow, No Universally Compelling Arguments, Anthropomorphic Optimism, Magical Categories, …

If I had to pick a single statement that relies on more Overcoming Bias content I've written than any other, that statement would be:

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

"Well," says the one, "maybe according to your provincial human values, you wouldn't like it.  But I can easily imagine a galactic civilization full of agents who are nothing like you, yet find great value and interest in their own goals.  And that's fine by me.  I'm not so bigoted as you are.  Let the Future go its own way, without trying to bind it forever to the laughably primitive prejudices of a pack of four-limbed Squishy Things -"

My friend, I have no problem with the thought of a galactic civilization vastly unlike our own… full of strange beings who look nothing like me even in their own imaginations… pursuing pleasures and experiences I can't begin to empathize with… trading in a marketplace of unimaginable goods… allying to pursue incomprehensible objectives… people whose life-stories I could never understand.

That's what the Future looks like if things go right.

If the chain of inheritance from human (meta)morals is broken, the Future does not look like this.  It does not end up magically, delightfully incomprehensible.

With very high probability, it ends up looking dull.  Pointless.  Something whose loss you wouldn't mourn.

Seeing this as obvious, is what requires that immense amount of background explanation.

Continue reading "Value is Fragile" »

GD Star Rating
loading...

31 Laws of Fun

By Eliezer Yudkowsky · January 26, 2009 5:13 am · Comments (15)

So this is Utopia, is it?  Well
I beg your pardon, I thought it was Hell.
        — Sir Max Beerholm, verse entitled
        In a Copy of More's (or Shaw's or Wells's or Plato's or Anybody's) Utopia

This is a shorter summary of the Fun Theory Sequence with all the background theory left out – just the compressed advice to the would-be author or futurist who wishes to imagine a world where people might actually want to live:

  1. Think of a typical day in the life of someone who's been adapting to Utopia for a while.  Don't anchor on the first moment of "hearing the good news".  Heaven's "You'll never have to work again, and the streets are paved with gold!" sounds like good news to a tired and poverty-stricken peasant, but two months later it might not be so much fun.  (Prolegomena to a Theory of Fun.)
  2. Beware of packing your Utopia with things you think people should do that aren't actually fun.  Again, consider Christian Heaven: singing hymns doesn't sound like loads of endless fun, but you're supposed to enjoy praying, so no one can point this out.  (Prolegomena to a Theory of Fun.)
  3. Making a video game easier doesn't always improve it.  The same holds true of a life.  Think in terms of clearing out low-quality drudgery to make way for high-quality challenge, rather than eliminating work.  (High Challenge.)
  4. Life should contain novelty – experiences you haven't encountered before, preferably teaching you something you didn't already know.  If there isn't a sufficient supply of novelty (relative to the speed at which you generalize), you'll get bored.  (Complex Novelty.)

Continue reading "31 Laws of Fun" »

GD Star Rating
loading...

The Fun Theory Sequence

By Eliezer Yudkowsky · January 25, 2009 6:18 am · Comments (16)

(A shorter gloss of Fun Theory is "31 Laws of Fun", which summarizes the advice of Fun Theory to would-be Eutopian authors and futurists.)

Fun Theory is the field of knowledge that deals in questions such as "How much fun is there in the universe?", "Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having more fun?"

Fun Theory is serious business.  The prospect of endless boredom is routinely fielded by conservatives as a knockdown argument against research on lifespan extension, against cryonics, against all transhumanism, and occasionally against the entire Enlightenment ideal of a better future.

Many critics (including George Orwell) have commented on the inability of authors to imagine Utopias where anyone would actually want to live.  If no one can imagine a Future where anyone would want to live, that may drain off motivation to work on the project.  But there are some quite understandable biases that get in the way of such visualization.

Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil).  Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance.  Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization – there is room for improvement.  Fun Theory also highlights the flaws of any particular religion's perfect afterlife – you wouldn't want to go to their Heaven.

Continue reading "The Fun Theory Sequence" »

GD Star Rating
loading...

Higher Purpose

By Eliezer Yudkowsky · January 23, 2009 4:58 am · Comments (19)

Previously in series: Interpersonal Entanglement
Followup to:  Something to Protect, Superhero Bias

Long-time readers will recall that I've long been uncomfortable with the idea that you can adopt a Cause as a hedonic accessory:

"Unhappy people are told that they need a 'purpose in life', so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes."

But conversely it's also a fact that having a Purpose In Life consistently shows up as something that increases happiness, as measured by reported subjective well-being.

One presumes that this works equally well hedonically no matter how misguided that Purpose In Life may be – no matter if it is actually doing harm – no matter if the means are as cheap as prayer.  Presumably, all that matters for your happiness is that you believe in it.  So you had better not question overmuch whether you're really being effective; that would disturb the warm glow of satisfaction you paid for.

And here we verge on Zen, because you can't deliberately pursue "a purpose that takes you outside yourself", in order to take yourself outside yourself.  That's still all about you.

Which is the whole Western concept of "spirituality" that I despise:  You need a higher purpose so that you can be emotionally healthy.  The external world is just a stream of victims for you to rescue.

Continue reading "Higher Purpose" »

GD Star Rating
loading...