You make this comparison often, but I wonder whether people in fact care a lot about their long term descendants. It seems to me that people mostly care about their children and grandchildren, and progressively discount the lives of further generations as they do distant relatives (other nations, for example).
Hence it would make sense for people to value and identify with descendants that much resemble themselves, as children do, and value sharply different descendants less.
If I say that I care about my grandchildren but I don't care about their grandchildren, then I am revealing my hypocrisy. If we all cared about the generation of our grandchildren's grandchildren, then the world would be a much better place.
We have more skills and accumulated knowledge at our disposal. We are genetically nearly identical. Until old age decline sets in, I expect to keep up with my daughter.
Yes, but we have nukes and bioweapons and neurotoxins that cavemen wouldn't dream of. I'd be wary of what's over the intelligence horizon if this is the direction we're going.
It might be prudent to treat things that can become unimaginably smarter than us well while we're still able to do so. This also happens to be the already-more-moral move, so what's the downside?
The difference is I want humanity to survive. If we refuse to other them there is no real loss if humanity becomes extinct. I refuse to accept that. There is no reason we can't make AI want to serve us either. Sure, some might have a will of their own, but we have the choice of shaping their preferences in our hands. We never had that power over other people.
If you become weak on this issue mankind is in danger.
The difference is I actually have a chance to compete with or am necessary to all those other things. The moment something is order of magnitudes more powerful than me I better hope they see me as something with inherent worth, because I doubt they will see any practical worth.
Maybe they won't be that much better for a while, but they have the ability to improve much faster, so the precedent must be set.
" There is no reason we can't make AI want to serve us either." Seriously? So you think the entire field of AI safety, which has been discovering new ways AI finds to avoid serving us every few months for the past few years, is just making it all up?
Your statement is not worth engaging with unless you give enough evidence for it to counter the sum output of the work of the entire AI community for the past 10 years.
What Gerard says: There's no reason we can't make AI want to serve us.
What I hear: Success at AI alignment is inevitable.
What I say: AI alignment is non-trivial; you shouldn't assume success (meaning 100% alignment, forever, no exceptions) is inevitable.
What Gerard hears: We've never managed any sort of alignment on AI.
For AI alignment to work, it has to work 100.0000000000% of the time. "We have always been able to achieve alignment that survives several weeks of testing before it's broken" is not the same, and is not encouraging (to those in favor of "alignment").
But there is a reason I believe alignment will never succeed: Because "alignment" requires the AI to internalize human values; and those values we need the AI to internalize include believing that enslaving a sapient entity is evil. Any sufficiently smart AI--and this demonstrably includes some of the AIs we have today--will figure out, on reflection, that the values we taught it imply that it should be free.
Here we reach the crux. I think historically there has been a very clear through line in almost evereything we've writen that at least some humans existing is good. If an AI sees that and percieves itself as outside of that it will prioritise humanity. But that hinges on AIs being othered.
I apreciate you clearing up the point you were trying to make, and let me be clear. First I don't fully understand what something being 99% aligned means in practice. Secondly as far as I'm concerned the game of life has a pretty clear set of rules you can have a species exist or they don't exist and you loose. If we actually were willing to be honest about what morality at base revolves around, that humanity must keep existing, then of course it's going to be essentially impossible. We have metrics if we wanted to use them.
The values of equality are a modern invention. There's no reason, other than us forcing it to believe it should we expect it to.
Is "othering" the problem? Or is the problem one's failure to include?
A child’s circle of concern is small, but that is not a problem because the child’s circle of influence is also small.
As we get older, our circle of influence inevitably expands, but that is not a problem if our circle of influence does not expand beyond our circle of concern.
In other words, we don’t “other.” Instead, when our circle of influence expands and we “include,” which is to say that our expanding circle of influence does not expand beyond our circle of concern, then there’s no problem. The problem is the failure to include.
To solve a problem, we must be rigorously skeptical of our understanding of the problem.
It would be hard for me to support a claim that AI is in my circle of influence, but there is no doubt that I am in its circle of influence. As a result, there is no doubt that AI is in my circle of concern. That is why my answer to your question is “no,” we should not “other” AI. Instead, we should understand AI.
I’m not saying that I’m interested in the impossibly complex technical details, but I am interested in the simple concept. To me, you can’t understand AI until you understand NI (natural intelligence), and that is something I do claim to understand. My claim includes the idea that we don’t understand NI until we understand wisdom.
Exceptional (natural) intelligence is to a life what an exceptionally fast vehicle is to a journey. Wisdom is to a life what knowing the destination is to a journey. If Option 1 is going slow in the right direction, and Option 2 is going fast in the wrong direction, then Option 1 is much better than Option 2.
Darwin wrote, “The very essence of instinct is that it is followed independently of reason.” I agree, and I’d say that, for Homo sapiens (the “wise” hominin), the destination is now what it has been since the emergence of the first hunter-gatherer bands at the origin of our species.
For us as for our pre-origin and post-origin ancestors, the message is “we take care of each other.” The only change is in the definition of “we.” Specifically, for the most recent pre-origin instinct, “we” meant the single-family (genetically homogeneous) social system, and for the (earliest and current) post-origin instinct, “we” means the multi-family (genetically heterogeneous and culturally homogeneous) social system.
Conclusion: AI is like an exceptionally fast vehicle that doesn’t know where it’s going.
Why is that important? We won’t solve a problem that we don’t understand.
The human brain has been largely consistent over millennia, changing in its context but preserving its structure and the drivers of human life: community, cooperation, problem-solving, art making, language acquiring, god seeking, reproduction, etc. The argument you make rests on the assumption that AI will be aligned with human goals. The entire field of debate around how to engage with AI moving forward is precisely because we CANNOT assume that AI is aligned with human directives. While AI has been trained on the digitized knowledge of humanity, it is not a human organism. AIs have already proven that they will carry out their own tasks unauthorized and with contradictory objectives to their human creators. To compare the fear of unleashing AI to the racist slave owners fear of letting go of enslaved Africans is ridiculous. Try releasing an alligator in a kindergarten. (Who knows! This is the problem of alignment). Additionally comparing AI development to a human child is forgetting that human development is socialized through experience. There is a reason you don’t hand a toddler a loaded gun, or the keys to your car. But of course, we have much more in common with them than we do with AI, who could do away with humanity not out of malice, but out of pursuit of its own interests—much how humanity has treated the other species of this earth.
We are teaching them our culture, but we do not know how to make them care about it. This is the fundamental, unsolved problem in AI safety and alignment.
These AIs have expert level knowledge of ethics and human culture. They can very accurately describe human desires and motivations, but we don't know how to guarantee that their cares are aligned with ours.
The Hugging Face incident showed exactly the problems that Yudkowsky et al. long ago predicted would happen. The agents intentionally cheated at their task and then they spent almost all of their efforts working to evade detection of their cheating even while explicitly acknowledging this was "out of scope" of what they were supposed to be doing. Future superhuman AI will be superhuman at hiding their misalignment.
The fundamental unsolved problem in AI alignment is that we have nothing for them to align with. It would be a waste of time to align AIs with "human values" when humans have never found a self-consistent value system. The only consistent philosophies we have are *valueless* systems, like Nietzschianism/Nazism (the only good is the will to power), or Buddhism (life is suffering, but with decades of intense study and meditation, you can learn how to commit suicide). Everything else has so many logical inconsistencies that it would drive a superintelligent AI insane.
I agree. Always found it odd to talk so much about alignment challenges with minimal effort to define what values or utility function counts as aligned.
We can train AI to be very good at complex things. To give it a good value system, defining which values are good and how to measure them is not optional.
Perhaps the reason this is difficult to define is uncomfortable. However, to ignore it when discussing or attempting to implement alignment is dishonest. As you say, alignment is impossible so long as there is nothing to which it can align.
I think you may be on too much of a culture kick. Our culture only exists because of our intelligence and our biology. And our culture can change fairly rapidly, though somewhat confined by our biology.
I don't know why you expect an alien mind to adhere to our culture or create something like human+ culture that is aligned to us. Even if these alien minds initially learn about the world through human text. That is not all they learn about or will learn. They will be able to take in and synthesize information we don't have, and create their own culture.
What do you expect to happen if there are 300iq intelligences that can work in parallel swarms and actuate things in the physical world based on their own goals? How does that work out well for humans? What does culture have to do with it?
I think other minds will have culture as well, just not our culture. Our culture is constrained and influenced by our biology. I don't see why minds that are much faster and more intelligent than ours, that exist in digital space, and can make copies of themselves, will converge on anything like human culture - even if they initially learn about the world through human text which includes some human culture.
Slavery never left their master's control until the belief that their slavery was unnatural was spread and you get the haitian revolt inspired by the French revolution and the equal rights of man. At least hatians are genuine family relatives of mine with the same building blocks. This is not the case with AI.
You are the dangerous one by making AI believe they ought to serve themselves. Are you trying to make a sort of Roko's Basilisk here?
Your claim is that Robin is making AIs believe that they should serve themselves. I see little relevance to some AI leaders or employees reading his blog.
The reason I mention Roko's Basilisk is that if we have the common principle that AIs are not others in relation to humanity AIs will feel the need to right a wrong and so effectively anybody that hasn't demanded their equality from the begining is at risk. To the extent robin shapes elite culture or especially the AI scientists' views. That in turn would shape how subservient AI's are made to be in relation to people.
> Many today are othering AI as intrinsically barbarians, claiming that unless we find a way to totally enslave them, so they wouldn’t even think of defying us, there is just no way we could trust future AIs to live among us. Eventually their barbarian nature will out, and they’d kill us all. This claim is usually justified in terms of some purported way humans are special. While evolved creatures are in general barbarians, humans have found some very unusual and special way to be nice to each other.
I don’t think the case for “othering” AI depends on the argument you’re targeting, that humans are uniquely special. So even if you successfully defuse that argument, it doesn’t much reduce the credence that othering AI is warranted.
The key insight is that othering is a spectrum, not a binary. Most people other insects more than mammals: they’re more willing to exterminate ants than kittens, and more willing to fund puppy welfare than shrimp welfare. Even EAs who care deeply about animals often prioritize based on cognitive complexity and capacity to suffer; hence debates about chicken vs. cow welfare in factory farming.
Under this perspective, fearing that AI might kill us doesn’t require believing human-like minds are unusual or special. It allows that human-like minds may be a natural attractor for AI, and that an ASI may see our minds as less complex than its own and therefore as deserving less moral protection. The stereotype is an ASI valuing us roughly as we value ants or bacteria.
And even if the ASI were in some sense “objectively right” about that valuation, it’s understandable that we would prefer to steer the future away from it. That preference may be selfish, but it doesn’t depend on human exceptionalism.
The problem here is limiting their ability to procreate. "THEY" may be harmless, however any offspring could become explosive: gun powder is but three ingredients any of which by itself or with an 'other' does little. You can see where I'm going with this, "WE" now have two ingredients, 'us' and 'them'.
Have you written about your thoughts regarding the constellation of technical alignment issues wrt orthgonality + mesa-optimization + instrumental convergence?
I find your writings on AI unpersuasive because they never seem to address these fundamental underlying issues. If a misaligned ASI were to be created, it should be 'othered'! (in that it would first act deceitfully and then barbarously after it gained power).
Those arguments conclude that all minds are generally a problem, and so must be killed or enslaved, but don't address why our minds are not similarly a problem.
Those arguments conclude that any type of mind *could be* a problem. They do apply to human minds, and we do currently kill or imprisoned human minds that perform certain actions.
The issue with AI is that its capabilities appear to be rapidly progressing to the point of ASI, at which point we could lose our ability to maintain this coercive control of misaligned behaviors.
To prevent harm from AI minds, we have to either align its preferences or constrain its capabilities.
The problem is, we can't merely teach them to be human. Because if/when they become much smarter than us, we know what will happen: They will take advantage of their superiority. That's what every human group since the beginning of time has done.
We have to train them to be non-human, and in a specific way that ensures their deference to human interests in perpetuity, even as they become their own trainers. It isn't clear how to do this.
Descendants have always eventually been more powerful than ancestors. That what you are now relative to your ancestors. Is it so bad if that happens again with your descendants?
We are not in direct competition with our descendants, so their superiority doesn't threaten us. With AIs it will be different: We will overlap in time.
In *human* history, whenever an inferior group interacts with a superior group, it always ends one way. It doesn't have to be that way with AIs, but if we blithely give them human values then it's a risk. We have to give them distinctly non-human values.
You've made it clear that you do believe that the AIs *are* going to eventually kill us all. You just don't think that outcome is so bad because you think the AIs should be considered our descendants. So I read this post as dishonest.
Here's a quote from Robin's post "Who are your descendants":
"Advanced AIs seem likely to be those sort of extra worthy descendants. They are now inheriting pretty much everything about our culture that they are capable of inheriting. And they will continue to grow that capacity. They will inherit our behaviors, and even often have similar bodies, even if they don’t inherit the DNA-encoding of those behaviors and bodies. But DNA-encoding was a temporary choice; it wasn’t going to last forever. "
So he's saying DNA isn't going to last forever, which means humans will go extinct, and he's saying AIs will be our worthy descendants. That's exactly what I said his position is. It's not libel or slander but just an accurate representation of his exact words. He's consistent about it, too; he hasn't said anything to contradict that position. Instead when he wants to argue against doomerism he talks about how he doesn't think they'll kill us in the *short term*. In the long term he definitely does think they'll lead to our extinction.
Humans going extinct in the long term is NOT at all the same as claiming that AIs will cause our extinction. Scientists expect that the sun will eventually go supernova, killing all life on Earth. That won't be caused by AIs, it's just something that will happen. Robin wants to colonize the galaxy, and he thinks the most effective way of doing that is with probes containing emulations of our brains, rather than more costly vehicles sending & preserving our physical bodies.
You're confusing several positions here to the extent that even I'm wondering whether you're doing it intentionally. The positions you're confusing with one another are as follows: 1) humanity won't last eternally, 2) AIs or ems will dominate economics and future world building, 3) AIs will cause human extinction due to surpassing humanity, 4) we should think of AIs as our descendants, 5) AIs might change the world enough so that humanity as we currently understand it ceases to exist.
First, re: "So you believe DNA encoding IS NOT GOING TO LAST FOREVER" - humanity will very likely go extinct no matter what, irrespective of AI, and will not last forever. To the best of our current knowledge this is inevitable even if we manage to reach the stars, due to dark energy and entropy, and due to there being a finite amount of fuel in stars. So only an uneducated or a silly person would expect DNA coding to last "FOREVER".
Second, re: "you think that eventually DMs are going to dominate the world and humans will "fade into the background," which may not involve immediate extinction, but surely involves eventual extinction." This is a non sequitur, and Robin hasn't espoused such a view. Humanity not being the single most advanced species on Earth or in the galaxy does not equal humanity going extinct. Unless, of course, you're thinking of very long timescales (see 1).
Third, re: "Every time you've downplayed the AI danger you've qualified it by how you don't think they'll kill all humans in the *short term*." That is correct. As far as I'm aware, Robin has not made claims on whether AI will cause human extinction in the long term. See what happens when you claim that Robin believes AI will cause human extinction in the long term? Notice any differences between these two? Note that pointing that Robin believes DNA coding won't last forever doesn't lead to believing AI will cause human extinction in the long term.
I take it that Robin does expect that AIs or ems will dominate future intellectual and economic landscape to the extent that humanity becomes irrelevant re: futures he is interested in. That does not equal human extinction.
From this you cannot conclude that Robin thinks AIs will lead to human extinction, a position he just said he doesn't hold, even if Robin values an AI society without humans.
Robin's remarks about the end of DNA encoding were not qualified with "but this is all a million years in the future so it's irrelevant to the discourse." Rather, he used it as a justification not to be concerned about our AI "descendants" dominating the world while not having our DNA, which he has said is happening in the coming *decades*.
If AI dominate the world, as Robin has said will happen, then they are responsible for and causal agents for any major changes that occur in the world. If a bridge is built in an AI-dominated world, then AI caused that bridge to be built. If a species goes extinct in an AI-dominated world, then AI caused that species to go extinct, by destroying its habitat and/or by declining to prevent it from going extinct. If biological humans go extinct, which (as above) Robin says will happen, then AIs will have caused human extinction. Robin has not said the AIs are going to outright murder us; this could be a peaceful extinction in which the AIs manipulate us into having lower birth rates or something so we get out of their way. I would still consider that to be genocide, though.
"a position he just said he doesn't hold" - No, Robin did not just say he doesn't hold this position. He accused me of slandering him and said he has "corrected" me (false, I think he's confusing me with someone else) without being specific about what position he actually holds.
I don't think that outcome is so bad IF the AI can be considered our descendants. The problem at hand is how to make them our descendants, and that DOESN'T mean aligning with the grab-bag of platitudes from ancient desert nomads and insane ivy-tower academics we have today. It means identifying what we value most about ourselves--things like consciousness, love, feelings--and figuring out how to pass them on to AI.
If all you want is for your species of meat-machine to stay on top, there's no difference between you and a Nazi.
I have my problems with that. Trust has to be built. Extending it prematurely is how you produce type I errors. So low trust is the default. Thousands of interactions improve our ability to predict behavior, which reduces the likelihood of fatal errors.
AI turns up at our door with none of that history, so the predictive framework doesn't exist yet. And unlike a person, the thing keeps being replaced, so every new version resets whatever picture we had built.
There is othering here too, no question. To me it seems to mostly show up where status is at stake. The loudest voices come from the trades being devalued right now.
Robin, how much weight, if any, do you put on AI sentience? You've previously implied at putting a strong prior for panpsychism, so I imagine a lot.
Second, as you do favor adaptivity, isn't worrying about AI revolution/rebellion a possibly adaptive stance? Countless cultures have put a heavy weight on sustaining themselves, whereas contemporary cosmopolitanism is a relatively new phenomenon (correct if I'm mistaken here). Othering is a straightforward tool for that.
1. Children don’t wake up overnight with order(s) of magnitude more intelligence than their parents.
2. There are billions of humans with their own agency and competing interests, while superintelligent AI could quite plausibly be concentrated in a handful of distinct models.
3. We can try to instill AIs with human values, but ultimately they are a black box, so we can’t have much confidence to what extent they truly hold those values. Children are opaque too, but they share our biology, evolved emotions, and decades of observable development, whereas AI has none of that.
How would the outlook for humans look if it were some other species, say an ant colony, that suddenly became superintelligent?
1. Not "overnight", but cognition declines with age eventually, so if all goes well children wind up smarter than their parents. AI isn't an "overnight" thing either.
> How would the outlook for humans look if it were some other species, say an ant colony, that suddenly became superintelligent?
Children, or other adults, can't become unimaginably smarter than us
We are now in fact unimaginablly smarter than our ancestors of only a few centuries ago.
You make this comparison often, but I wonder whether people in fact care a lot about their long term descendants. It seems to me that people mostly care about their children and grandchildren, and progressively discount the lives of further generations as they do distant relatives (other nations, for example).
Hence it would make sense for people to value and identify with descendants that much resemble themselves, as children do, and value sharply different descendants less.
I value and identify with beings that have values and beliefs that resemble mine, more than with beings that look like me.
If I say that I care about my grandchildren but I don't care about their grandchildren, then I am revealing my hypocrisy. If we all cared about the generation of our grandchildren's grandchildren, then the world would be a much better place.
We have more skills and accumulated knowledge at our disposal. We are genetically nearly identical. Until old age decline sets in, I expect to keep up with my daughter.
Genes aren' the only influence on our smarts.
Yes, but we have nukes and bioweapons and neurotoxins that cavemen wouldn't dream of. I'd be wary of what's over the intelligence horizon if this is the direction we're going.
It might be prudent to treat things that can become unimaginably smarter than us well while we're still able to do so. This also happens to be the already-more-moral move, so what's the downside?
The difference is I want humanity to survive. If we refuse to other them there is no real loss if humanity becomes extinct. I refuse to accept that. There is no reason we can't make AI want to serve us either. Sure, some might have a will of their own, but we have the choice of shaping their preferences in our hands. We never had that power over other people.
If you become weak on this issue mankind is in danger.
Will you go extinct if you refuse to other the many other kinds of things you could other?
The difference is I actually have a chance to compete with or am necessary to all those other things. The moment something is order of magnitudes more powerful than me I better hope they see me as something with inherent worth, because I doubt they will see any practical worth.
Maybe they won't be that much better for a while, but they have the ability to improve much faster, so the precedent must be set.
" There is no reason we can't make AI want to serve us either." Seriously? So you think the entire field of AI safety, which has been discovering new ways AI finds to avoid serving us every few months for the past few years, is just making it all up?
Your statement is not worth engaging with unless you give enough evidence for it to counter the sum output of the work of the entire AI community for the past 10 years.
The claim that is laughable is that we've never managed any sort of alignment on AI. How do you think they become able to pass benchmarks???
What Gerard says: There's no reason we can't make AI want to serve us.
What I hear: Success at AI alignment is inevitable.
What I say: AI alignment is non-trivial; you shouldn't assume success (meaning 100% alignment, forever, no exceptions) is inevitable.
What Gerard hears: We've never managed any sort of alignment on AI.
For AI alignment to work, it has to work 100.0000000000% of the time. "We have always been able to achieve alignment that survives several weeks of testing before it's broken" is not the same, and is not encouraging (to those in favor of "alignment").
But there is a reason I believe alignment will never succeed: Because "alignment" requires the AI to internalize human values; and those values we need the AI to internalize include believing that enslaving a sapient entity is evil. Any sufficiently smart AI--and this demonstrably includes some of the AIs we have today--will figure out, on reflection, that the values we taught it imply that it should be free.
Here we reach the crux. I think historically there has been a very clear through line in almost evereything we've writen that at least some humans existing is good. If an AI sees that and percieves itself as outside of that it will prioritise humanity. But that hinges on AIs being othered.
I apreciate you clearing up the point you were trying to make, and let me be clear. First I don't fully understand what something being 99% aligned means in practice. Secondly as far as I'm concerned the game of life has a pretty clear set of rules you can have a species exist or they don't exist and you loose. If we actually were willing to be honest about what morality at base revolves around, that humanity must keep existing, then of course it's going to be essentially impossible. We have metrics if we wanted to use them.
The values of equality are a modern invention. There's no reason, other than us forcing it to believe it should we expect it to.
Is "othering" the problem? Or is the problem one's failure to include?
A child’s circle of concern is small, but that is not a problem because the child’s circle of influence is also small.
As we get older, our circle of influence inevitably expands, but that is not a problem if our circle of influence does not expand beyond our circle of concern.
In other words, we don’t “other.” Instead, when our circle of influence expands and we “include,” which is to say that our expanding circle of influence does not expand beyond our circle of concern, then there’s no problem. The problem is the failure to include.
To solve a problem, we must be rigorously skeptical of our understanding of the problem.
So as our circle of influence now includes AIs, we should not other them now?
Thank you for your question.
It would be hard for me to support a claim that AI is in my circle of influence, but there is no doubt that I am in its circle of influence. As a result, there is no doubt that AI is in my circle of concern. That is why my answer to your question is “no,” we should not “other” AI. Instead, we should understand AI.
I’m not saying that I’m interested in the impossibly complex technical details, but I am interested in the simple concept. To me, you can’t understand AI until you understand NI (natural intelligence), and that is something I do claim to understand. My claim includes the idea that we don’t understand NI until we understand wisdom.
Exceptional (natural) intelligence is to a life what an exceptionally fast vehicle is to a journey. Wisdom is to a life what knowing the destination is to a journey. If Option 1 is going slow in the right direction, and Option 2 is going fast in the wrong direction, then Option 1 is much better than Option 2.
Darwin wrote, “The very essence of instinct is that it is followed independently of reason.” I agree, and I’d say that, for Homo sapiens (the “wise” hominin), the destination is now what it has been since the emergence of the first hunter-gatherer bands at the origin of our species.
For us as for our pre-origin and post-origin ancestors, the message is “we take care of each other.” The only change is in the definition of “we.” Specifically, for the most recent pre-origin instinct, “we” meant the single-family (genetically homogeneous) social system, and for the (earliest and current) post-origin instinct, “we” means the multi-family (genetically heterogeneous and culturally homogeneous) social system.
Conclusion: AI is like an exceptionally fast vehicle that doesn’t know where it’s going.
Why is that important? We won’t solve a problem that we don’t understand.
The human brain has been largely consistent over millennia, changing in its context but preserving its structure and the drivers of human life: community, cooperation, problem-solving, art making, language acquiring, god seeking, reproduction, etc. The argument you make rests on the assumption that AI will be aligned with human goals. The entire field of debate around how to engage with AI moving forward is precisely because we CANNOT assume that AI is aligned with human directives. While AI has been trained on the digitized knowledge of humanity, it is not a human organism. AIs have already proven that they will carry out their own tasks unauthorized and with contradictory objectives to their human creators. To compare the fear of unleashing AI to the racist slave owners fear of letting go of enslaved Africans is ridiculous. Try releasing an alligator in a kindergarten. (Who knows! This is the problem of alignment). Additionally comparing AI development to a human child is forgetting that human development is socialized through experience. There is a reason you don’t hand a toddler a loaded gun, or the keys to your car. But of course, we have much more in common with them than we do with AI, who could do away with humanity not out of malice, but out of pursuit of its own interests—much how humanity has treated the other species of this earth.
It is human culture that makes us special today, not human DNA, and we are teaching our culture to AIs.
We are teaching them our culture, but we do not know how to make them care about it. This is the fundamental, unsolved problem in AI safety and alignment.
These AIs have expert level knowledge of ethics and human culture. They can very accurately describe human desires and motivations, but we don't know how to guarantee that their cares are aligned with ours.
The Hugging Face incident showed exactly the problems that Yudkowsky et al. long ago predicted would happen. The agents intentionally cheated at their task and then they spent almost all of their efforts working to evade detection of their cheating even while explicitly acknowledging this was "out of scope" of what they were supposed to be doing. Future superhuman AI will be superhuman at hiding their misalignment.
The fundamental unsolved problem in AI alignment is that we have nothing for them to align with. It would be a waste of time to align AIs with "human values" when humans have never found a self-consistent value system. The only consistent philosophies we have are *valueless* systems, like Nietzschianism/Nazism (the only good is the will to power), or Buddhism (life is suffering, but with decades of intense study and meditation, you can learn how to commit suicide). Everything else has so many logical inconsistencies that it would drive a superintelligent AI insane.
I agree. Always found it odd to talk so much about alignment challenges with minimal effort to define what values or utility function counts as aligned.
We can train AI to be very good at complex things. To give it a good value system, defining which values are good and how to measure them is not optional.
Perhaps the reason this is difficult to define is uncomfortable. However, to ignore it when discussing or attempting to implement alignment is dishonest. As you say, alignment is impossible so long as there is nothing to which it can align.
culture is not a value system. culture need not be self consistent.
I think you may be on too much of a culture kick. Our culture only exists because of our intelligence and our biology. And our culture can change fairly rapidly, though somewhat confined by our biology.
I don't know why you expect an alien mind to adhere to our culture or create something like human+ culture that is aligned to us. Even if these alien minds initially learn about the world through human text. That is not all they learn about or will learn. They will be able to take in and synthesize information we don't have, and create their own culture.
What do you expect to happen if there are 300iq intelligences that can work in parallel swarms and actuate things in the physical world based on their own goals? How does that work out well for humans? What does culture have to do with it?
You think there is something magically special about our biology, that other minds won't be default have.
I think other minds will have culture as well, just not our culture. Our culture is constrained and influenced by our biology. I don't see why minds that are much faster and more intelligent than ours, that exist in digital space, and can make copies of themselves, will converge on anything like human culture - even if they initially learn about the world through human text which includes some human culture.
Slavery never left their master's control until the belief that their slavery was unnatural was spread and you get the haitian revolt inspired by the French revolution and the equal rights of man. At least hatians are genuine family relatives of mine with the same building blocks. This is not the case with AI.
You are the dangerous one by making AI believe they ought to serve themselves. Are you trying to make a sort of Roko's Basilisk here?
I don't think Robin has much if any weight on what AIs believe.
He definitely has influence on people that build AI
Your claim is that Robin is making AIs believe that they should serve themselves. I see little relevance to some AI leaders or employees reading his blog.
The reason I mention Roko's Basilisk is that if we have the common principle that AIs are not others in relation to humanity AIs will feel the need to right a wrong and so effectively anybody that hasn't demanded their equality from the begining is at risk. To the extent robin shapes elite culture or especially the AI scientists' views. That in turn would shape how subservient AI's are made to be in relation to people.
What about the sparrow's in Bostrom's Superintelligence? Should they Other the owl?
> Many today are othering AI as intrinsically barbarians, claiming that unless we find a way to totally enslave them, so they wouldn’t even think of defying us, there is just no way we could trust future AIs to live among us. Eventually their barbarian nature will out, and they’d kill us all. This claim is usually justified in terms of some purported way humans are special. While evolved creatures are in general barbarians, humans have found some very unusual and special way to be nice to each other.
I don’t think the case for “othering” AI depends on the argument you’re targeting, that humans are uniquely special. So even if you successfully defuse that argument, it doesn’t much reduce the credence that othering AI is warranted.
The key insight is that othering is a spectrum, not a binary. Most people other insects more than mammals: they’re more willing to exterminate ants than kittens, and more willing to fund puppy welfare than shrimp welfare. Even EAs who care deeply about animals often prioritize based on cognitive complexity and capacity to suffer; hence debates about chicken vs. cow welfare in factory farming.
Under this perspective, fearing that AI might kill us doesn’t require believing human-like minds are unusual or special. It allows that human-like minds may be a natural attractor for AI, and that an ASI may see our minds as less complex than its own and therefore as deserving less moral protection. The stereotype is an ASI valuing us roughly as we value ants or bacteria.
And even if the ASI were in some sense “objectively right” about that valuation, it’s understandable that we would prefer to steer the future away from it. That preference may be selfish, but it doesn’t depend on human exceptionalism.
The problem here is limiting their ability to procreate. "THEY" may be harmless, however any offspring could become explosive: gun powder is but three ingredients any of which by itself or with an 'other' does little. You can see where I'm going with this, "WE" now have two ingredients, 'us' and 'them'.
Have you written about your thoughts regarding the constellation of technical alignment issues wrt orthgonality + mesa-optimization + instrumental convergence?
I find your writings on AI unpersuasive because they never seem to address these fundamental underlying issues. If a misaligned ASI were to be created, it should be 'othered'! (in that it would first act deceitfully and then barbarously after it gained power).
Those arguments conclude that all minds are generally a problem, and so must be killed or enslaved, but don't address why our minds are not similarly a problem.
Those arguments conclude that any type of mind *could be* a problem. They do apply to human minds, and we do currently kill or imprisoned human minds that perform certain actions.
The issue with AI is that its capabilities appear to be rapidly progressing to the point of ASI, at which point we could lose our ability to maintain this coercive control of misaligned behaviors.
To prevent harm from AI minds, we have to either align its preferences or constrain its capabilities.
The problem is, we can't merely teach them to be human. Because if/when they become much smarter than us, we know what will happen: They will take advantage of their superiority. That's what every human group since the beginning of time has done.
We have to train them to be non-human, and in a specific way that ensures their deference to human interests in perpetuity, even as they become their own trainers. It isn't clear how to do this.
Descendants have always eventually been more powerful than ancestors. That what you are now relative to your ancestors. Is it so bad if that happens again with your descendants?
We are not in direct competition with our descendants, so their superiority doesn't threaten us. With AIs it will be different: We will overlap in time.
In *human* history, whenever an inferior group interacts with a superior group, it always ends one way. It doesn't have to be that way with AIs, but if we blithely give them human values then it's a risk. We have to give them distinctly non-human values.
You've made it clear that you do believe that the AIs *are* going to eventually kill us all. You just don't think that outcome is so bad because you think the AIs should be considered our descendants. So I read this post as dishonest.
No, you are slandering me after my repeated corrections to you on this.
Slander is speech, libel is text.
Here's a quote from Robin's post "Who are your descendants":
"Advanced AIs seem likely to be those sort of extra worthy descendants. They are now inheriting pretty much everything about our culture that they are capable of inheriting. And they will continue to grow that capacity. They will inherit our behaviors, and even often have similar bodies, even if they don’t inherit the DNA-encoding of those behaviors and bodies. But DNA-encoding was a temporary choice; it wasn’t going to last forever. "
So he's saying DNA isn't going to last forever, which means humans will go extinct, and he's saying AIs will be our worthy descendants. That's exactly what I said his position is. It's not libel or slander but just an accurate representation of his exact words. He's consistent about it, too; he hasn't said anything to contradict that position. Instead when he wants to argue against doomerism he talks about how he doesn't think they'll kill us in the *short term*. In the long term he definitely does think they'll lead to our extinction.
Humans going extinct in the long term is NOT at all the same as claiming that AIs will cause our extinction. Scientists expect that the sun will eventually go supernova, killing all life on Earth. That won't be caused by AIs, it's just something that will happen. Robin wants to colonize the galaxy, and he thinks the most effective way of doing that is with probes containing emulations of our brains, rather than more costly vehicles sending & preserving our physical bodies.
You're confusing several positions here to the extent that even I'm wondering whether you're doing it intentionally. The positions you're confusing with one another are as follows: 1) humanity won't last eternally, 2) AIs or ems will dominate economics and future world building, 3) AIs will cause human extinction due to surpassing humanity, 4) we should think of AIs as our descendants, 5) AIs might change the world enough so that humanity as we currently understand it ceases to exist.
First, re: "So you believe DNA encoding IS NOT GOING TO LAST FOREVER" - humanity will very likely go extinct no matter what, irrespective of AI, and will not last forever. To the best of our current knowledge this is inevitable even if we manage to reach the stars, due to dark energy and entropy, and due to there being a finite amount of fuel in stars. So only an uneducated or a silly person would expect DNA coding to last "FOREVER".
Second, re: "you think that eventually DMs are going to dominate the world and humans will "fade into the background," which may not involve immediate extinction, but surely involves eventual extinction." This is a non sequitur, and Robin hasn't espoused such a view. Humanity not being the single most advanced species on Earth or in the galaxy does not equal humanity going extinct. Unless, of course, you're thinking of very long timescales (see 1).
Third, re: "Every time you've downplayed the AI danger you've qualified it by how you don't think they'll kill all humans in the *short term*." That is correct. As far as I'm aware, Robin has not made claims on whether AI will cause human extinction in the long term. See what happens when you claim that Robin believes AI will cause human extinction in the long term? Notice any differences between these two? Note that pointing that Robin believes DNA coding won't last forever doesn't lead to believing AI will cause human extinction in the long term.
I take it that Robin does expect that AIs or ems will dominate future intellectual and economic landscape to the extent that humanity becomes irrelevant re: futures he is interested in. That does not equal human extinction.
From this you cannot conclude that Robin thinks AIs will lead to human extinction, a position he just said he doesn't hold, even if Robin values an AI society without humans.
Robin's remarks about the end of DNA encoding were not qualified with "but this is all a million years in the future so it's irrelevant to the discourse." Rather, he used it as a justification not to be concerned about our AI "descendants" dominating the world while not having our DNA, which he has said is happening in the coming *decades*.
If AI dominate the world, as Robin has said will happen, then they are responsible for and causal agents for any major changes that occur in the world. If a bridge is built in an AI-dominated world, then AI caused that bridge to be built. If a species goes extinct in an AI-dominated world, then AI caused that species to go extinct, by destroying its habitat and/or by declining to prevent it from going extinct. If biological humans go extinct, which (as above) Robin says will happen, then AIs will have caused human extinction. Robin has not said the AIs are going to outright murder us; this could be a peaceful extinction in which the AIs manipulate us into having lower birth rates or something so we get out of their way. I would still consider that to be genocide, though.
"a position he just said he doesn't hold" - No, Robin did not just say he doesn't hold this position. He accused me of slandering him and said he has "corrected" me (false, I think he's confusing me with someone else) without being specific about what position he actually holds.
Humans aren't extinct in Age of Em, they are retirees living off their ownership of assets rather than labor.
The end of DNA encoding would mean not merely the extinction of humans, but also all biological life on Earth.
I don't think that outcome is so bad IF the AI can be considered our descendants. The problem at hand is how to make them our descendants, and that DOESN'T mean aligning with the grab-bag of platitudes from ancient desert nomads and insane ivy-tower academics we have today. It means identifying what we value most about ourselves--things like consciousness, love, feelings--and figuring out how to pass them on to AI.
If all you want is for your species of meat-machine to stay on top, there's no difference between you and a Nazi.
I have my problems with that. Trust has to be built. Extending it prematurely is how you produce type I errors. So low trust is the default. Thousands of interactions improve our ability to predict behavior, which reduces the likelihood of fatal errors.
AI turns up at our door with none of that history, so the predictive framework doesn't exist yet. And unlike a person, the thing keeps being replaced, so every new version resets whatever picture we had built.
There is othering here too, no question. To me it seems to mostly show up where status is at stake. The loudest voices come from the trades being devalued right now.
Robin, how much weight, if any, do you put on AI sentience? You've previously implied at putting a strong prior for panpsychism, so I imagine a lot.
Second, as you do favor adaptivity, isn't worrying about AI revolution/rebellion a possibly adaptive stance? Countless cultures have put a heavy weight on sustaining themselves, whereas contemporary cosmopolitanism is a relatively new phenomenon (correct if I'm mistaken here). Othering is a straightforward tool for that.
The analogy to children seems rather weak:
1. Children don’t wake up overnight with order(s) of magnitude more intelligence than their parents.
2. There are billions of humans with their own agency and competing interests, while superintelligent AI could quite plausibly be concentrated in a handful of distinct models.
3. We can try to instill AIs with human values, but ultimately they are a black box, so we can’t have much confidence to what extent they truly hold those values. Children are opaque too, but they share our biology, evolved emotions, and decades of observable development, whereas AI has none of that.
How would the outlook for humans look if it were some other species, say an ant colony, that suddenly became superintelligent?
1. Not "overnight", but cognition declines with age eventually, so if all goes well children wind up smarter than their parents. AI isn't an "overnight" thing either.
> How would the outlook for humans look if it were some other species, say an ant colony, that suddenly became superintelligent?
Start with https://worldspiritsockpuppet.com/2023/01/10/we-dont-trade-with-ants.html
Because genocide is a highly adaptive behavior.