Robin Hansen: . . .a hand-coded AI foom remains possible after ems, but the context would be different in important ways.

Is it a context that makes Friendliness of the hand-coded AI less of a concern? If so, how?

Like I said before: This scenario already makes me want to look into friendliness, even without a singleton, because it is what I consider an unfriendly outcome. That the people who would exist in this outcome would be ok with it is moot, just as I don’t weigh the values of the paperclip maximizer as relevant to what I want.

it is hard to imagine [...] that ems wouldn’t consider running 1000x slower as something akin to death

I’m a bit suspicious of statements that begin with “It is hard to imagine” or “I can’t see how” when speculating on the non-immediate future. They convey a sense of misplaced confidence in a huge space of potential counter-examples. Whether or not it is probable, it is certainly not hard to imagine, particularly when talking about something as unconstrained as a future em’s philosophical intuitions.

Anyway, it is what it is regardless of what the ems consider it to be. It’s not total suspension, and it’s not information-theoretic death. Put yourself in the role of the evictee, with the options of termination, archival, or massive slowdown. I believe very few would choose archival, and fewer yet termination, especially once there’s any established “slow culture” to participate in.

Lightwave, can my scenario really be more fun to argue than the basement AI that suddenly takes over the world?

Hal, even the evolved morality we inherited does not entirely approve of the morality that ems would evolve, the question is how hard we’d be willing to work to change their world/morality to match ours.

frelkins, I’m pretty sure I have no famous ethical axioms.

Virge, bambi is right; I’ll assume crude trends continue until I see reasons to think otherwise.

Tyrrell and Roko, a hand-coded AI foom remains possible after ems, but the context would be different in important ways.

2. The whole ems economy scenario strikes me as very unlikely. Eliezer somewhere said to be wary of things that are fun to argue, and I think that’s what everyone here is doing.

It’s really not surprising that one can find paradoxes in a welfare function when the mathematics is obviously not modeling what they want it to model over the whole domain of interest. I’ll have to think about it a little more. The only paradoxes I’m seeing so far come from unrealistic modeling.

Well me too, but my personal preference is irrelevant if my offspring are going to compete with more productive but less human ems.

“But I doubt creating a productive zombie can be done quickly.”

(”Zombie” has the wrong connotation – think more of an bright, enthusiastic cult member. But you did say “productive zombie”, so that’s ok)

I’m not so sure about the long time frame. We’re not talking about understanding how memory encoding or reverse-engineering anything “deep” about human intelligence, we’re talking about psychological conditioning. If you can stomach it stretch your imagination …

Think about removing all ethical limits from experimentation into psychological conditioning, having ability to perform perfectly repeatable experiments.

Imagine if you could get hold of a pre-adolescent em (f**k that’s a horrible thought) – the extra plasticity might be worth the longer training time.

You hear about new discoveries in neuroanatomy just about every month from fMRI. Imagine what will be known by the time uploading is possible. Even if you don’t know exactly where all the neurons go and why, you might be able to engineer gross personality changes. You can experiment as much as you want.

Virtual psychopharmachology.

If I can think of that off the top of my head, think what would be achievable given the enormous commercial pressure to produce a willing slave. Yes, the development might be f**king horrible, but if you’re going to assume that ems can be involuntarily KILLED, I don’t think you can assume there will be any ethical restrictions on em development.

But why go straight from 1 to 0? An em can be slowed down to a near-infinite degree. A 6502 pried out of a NES given access to sufficient storage could run an entire civilization, albeit at a tremendous slowdown.

Continual genocide certainly seems possible, but as far as I can tell, you’d need to be confident that storage demands will keep pace with computing demands to put much weight into such a belief.

However I admit that it is hard to come up with arguments to choose one morality over another. Consistency would be desirable at a minimum. You might review the discussion around Parfit’s “Repugnant Conclusion” which to me suggests an inconsistency in failing to value new life sufficiently.

Virge, in answer to your question, although I think Robin has more to offer the world than I do, if you were to balance enough copies of me against him dying, then yes, at some point I think it would be moral to favor the copies. Whether five is enough is hard to say. But as I was trying to indicate, these kinds of dilemmas are not specific to the issue of copying. Would you save Robin’s life or that of five random people, if you had to choose one or the other? How about two people? How about ten? What if they are old and about to die? You can come up with a million variants. It is always hard for us to balance life against death. And see what you think about the Repugnant Conclusion linked in the previous paragraph.