That does not make much sense. Death doesn’t have much to do with diversity, if there are backups – and information-theoretic death occurs in both scenarios.

“You’ve probably read “The Lord of the Rings”, right? Don’t think of this as a programming project. Think of it as being something like the Fellowship of the Ring – the Fellowship of the AI, as it were. We’re not searching for programmers for some random corporate inventory-tracking project; we’re searching for people to fill out the Fellowship of the AI.”

– from “BECOMING A SEED AI PROGRAMMER”.

Secondly, the problem with circular preferences is not that they lead you in a circle. The problem arises when you move in a circle and you would have been better off if you had stayed still.

Your example doesn’t result in circular travel. It doesn’t even lead to different behaviour to that which would be produced by a utility maximiser.

In summary, I don’t see that your picking at the edges of this concept and hoping it will unravel is getting you very far.

To sum up why Steve Omohundro’s derivation doesn’t work: being an expected utility maximizer is sufficient, but it’s not necessary, to avoid being exploited.

I suppose that if I could know that my reflective equilibrium left out all aspects of morality except X, Y, Z which were simple enough to program into an AI, that this would simplify the Friendly AI problem. I’m not assuming any such simplification exists; my current morality, extended outward, neither simplifies in this fashion nor looks like it might do so. The utility function is not up for grabs – you just have to come to terms with that fact; if you can’t do complicated things like utility function transfers, you probably shouldn’t be running AIs. Utility function transfers are simpler than all of human morality, though – or you can bootstrap them – that’s the key hope here.

What do you mean? If your preferences change over time, then that can be represented by a more complex preference system that explains such temporal variation in values – and then it is the utility associated with those preferences which is being maximised.

History provides a useful guide. People like Hippocrates, Qin Shi Huang, Charles Darwin, or more recently, Linus Torvalds or Bram Cohen, made significant alterations to the course of human history because of their personal views and technical expertise – be it in developing methodologies, standardising armies and languages, or providing computational tools with the power to effect how societies work. (international communications, education to some extent now depend on linux; bittorrent, within a year or two, came to represent 1/3 of all digital international communications – not a FOOM, but not entirely dissimilar either). These were not social changes that were ‘bound to happen’… look at Galileo. Rather, a single person imposed a new reality upon humanity, generally, through their singlemindedness and skills.

–

“Paperclipping the solar system is an evil beyond the understanding of most human minds. But for a paperclipper it’s completely natural – virtuous even.”

Heaven forbid an AI notices that humans are busy turning planet earth into more humans at a terrifying rate – and the rest of the universe, if we get the chance.

–

Also, a note for anyone interested in FOOM. There’s a scifi book series by Jack Chalker called Rings of the Master. It’s about an AI that FOOMs and takes over humanity, followed by the galaxy. It proceeds to take steps to ensure humanity cannot regain control again. However, the people who created the AI took a precaution against this event; its value system includes an overriding imperative that there must always be SOME way for humans to regain direct control over the AIs actions – perhaps incredibly difficult, but within the realms of possibility.