Discussion about this post

User's avatar
Sergey Alexashenko's avatar

Very similar to my thoughts on the subject. I like to start the argument with Orson Scott Card's Hierarchy of Foreignness https://sergey.substack.com/p/what-is-to-be-done

Expand full comment
Dave Orr's avatar

I wonder if there's a fundamental disagreement on what you are even talking about, relative to the people you've been discussing this with. (Kudos, btw, for public discussions on a contentious topic; very few try to drive forward shared understanding like you do!)

In particular, I think that your conception of AI and theirs is so different as to be talking about entirely different categories of things. For instance, you say "Future AIs can also have different beliefs, attitudes, values, and mental styles." But I think many of your interlocutors would deny that current AIs even have any of those things, that the analogues they have are so impoverished that there is nothing meaningfully there that could be considered a value. It's less that AIs will have different values, it's that they may not have values at all.

Would you say that grey goo[1] has values of growth and competition? I think Zvi would say it's mindlessly replicating, it doesn't have value. If the future was automated factories building self driving Teslas but no humans, would that be a future that seems good with entities that have very different values? Or would the loss of humanity be something to mourn?

I think you think that our AI descendants will have values and beliefs and thought styles, and there's no particular reason to think that ours are much more valuable than theirs, for the same reason that we don't think our distant ancestors' very different values are much better than ours. But I think the AI risk camp thinks that AI will likely not have values at all, just a thoughtless optimization function.

It's true that in many times in history humans have acted and believed that other tribes were not thinking beings or didn't have meaningful value, and were wrong. But I think there is a very large class of things that everyone agrees don't have values and we're all right about it, and I think you think that too.

So maybe the main difference is that when you and your discussants are drawing lines around things that think and have values versus don't, AI is on one side for you and the other for them. And then many further ideas cascade from there.

[1] https://en.wikipedia.org/wiki/Gray_goo

Expand full comment
45 more comments...

No posts