On AI risk, I’ve said that evolved things have two robust habits: wariness of co-existing rivals, even when quite similar to you, and indulgence of descendants, even when quite different from you.
I think the last paragraph will change few minds because people rarely assign moral weight to descendants qua descendants, they have specific attachments to specific human beings they have formed relationships with as well as humanity in general due to shared traits among humans such as the capacity for human-like experience. Maybe this will change when more see AIs as moral patients, but I’m not sure a majority ever will.
Your last line is what people disagree with. People are worried AIs outcompete humans, so there are no humans left and all of our biological descentants who share all the biological human things with us that make us human, including dna, will die off. Most humans don't want this. It seems like you are being very obtuse about this point.
Disingenuous because *you* believe it is likely that AIs will outcompete humans and our biological descendants will die off. You said, "DNA-encoding was a temporary choice; it wasn’t going to last forever. "
I also believe it is likely that AIs will outcompete humans and our biological descendants will die off. But unlike you I'm not convinced this is a good thing.
You're right, just because it's a worry doesn't make it likely. I've read and listened to you a fair bit on this subject and when you talk about AIs supplanting humans, I'm never quite clear on how you think that will go. You will say there may be a lot of AIs around in a few hundred years and not many humans, and that's ok. When people push back against that you say something like if it was over 100 thousand years people wouldn't care, but the shortness matters. There are implications in the shortness. Namely, that humans didn't get to steer that outcome. I must be missing something. Can you fill me in?
"AI will inherent a great many things from us" may or may not be true, depending on what you mean exactly. If it's the earth, then ok I agree. If it's something like remnants of culture because the current llms use our language and data then I have no idea and can see that getting vaporized if RSI leads down a different path.
Grant that AI is our descendant. My worry is what kind the process producing it right now (US-China race, firm-vs-firm sprint, open-weights leakage) systematically rewards shipping speed and raw capability over caution and value constraint. The benign inheritance "indulgence" quietly assumes doesn't survive that kind of selection pressure.
Very few evolutionary biologists would agree with your stretched meaning of the word "descendants." You are using it in the universal-Darwinism way that appeals to some futurists and AI philosophers. But the term is native to evolutionary biology, and there the word "gene" almost always refers to DNA genes, and "descendant" means biological offspring (or biological offspring of biological offspring, etc).
What you're failing to argue is why we should *care* about having AI "descendants." Many people do care about having biological human descendants, because our biological instincts and our traditional cultures have programmed us to care about that. Our biological instincts and our traditional cultures have not programmed us to care about having AI descendants, especially if it means the extinction of biological human descendants. Simply because you can stretch to use the word "descendant" for both cases does not imply that our existing preferences for human descendants would or should transfer from one case to the other.
The AIs we create would be very different from us in ways that human descendants would be similar to us. From this perspective, AI descendants are very inferior for the purpose of spreading creatures like ourselves. They think but not like we do, their brains do not work like our brains, their bodies do not work like our bodies. They might as well be alien invaders.
Darwin didn't know about DNA, so his use of "gene" couldn't have referred to DNA. And since culture evolution dominates human evolution, DNA is much less important for us re descendants.
Darwin certainly didn't know about Universal Darwinism (UD). He was concerned with evolution among living, biological organisms. He would not have considered a kicked stone to be a descendant of your foot, as you asserted in a comment.
It also doesn't matter how Darwin used the word - that's the etymological fallacy.
The definition of the word "gene" also doesn't matter. You can define it in the broad UD sense, that's fine, provided you make clear that's what you're doing. And you can define "descendants" in a UD sense too. But defining it that way does not provide any convincing argument that we should *care* about our UD "descendants" in the same way we care about our biological descendants. And that's the key that you're missing.
It's a fallacy of equivocation: "we want our descendants to prosper. AI is our descendants. Therefore we want AI to prosper." But the first use of "descendants" means "biological human descendants," (or at least that's the sense of the premise that 99% of people would be willing to accept) and the second use means "UD descendants," and you're equivocating between the two.
I think you have to speak for yourself here. Importance is subjective. In practice, people vary considerably in the extent to which they value their memes relative to the value they assign to their genes. Important factors are: how strong their memetic immune system is; the extent to which they are vulnerable to memetic hijacking; how virulent the memes they have been exposed to are; and which other parasites they have been infected with.
I expect we'll see the perceived "alienness" of AI come down in the future. The obvious case would be a brain emulation or mind uploading scenario, but even if those don't happen several things will come with high likelihood: it will start to learn through personal experience like a human does, it will have a physical presence, and it will be tailored by its creators to be appealing/useful to people. People are quick to get emotionally attached to pets of all kinds and I see no reason why AIs must be different.
Perceived alienness may come down when companies come out with friendly-looking helper robots. I dunno that the companies will be really successful. The robots won't look and seem exactly like a human, because that would be costly and risks the uncanny valley. Maybe they'll be plush mascots. But humans are known for discriminating against humans that look a little different from them, so I suspect if there's anger towards robots for taking jobs, humans will be angry and discriminatory towards the plush mascots.
Actual alienness coming down? Much more dubious. The brains of LLMs are nothing like the brains of humans.
Brain emulation and mind uploading are still very far away. The world computing power is insufficient to simulate a single human brain on the level of individual neurons. Plus we have no way of scanning individual neurons. AI will destabilize the humans-work-and-get-paid basis for the world economy long before we get anything like that.
You raise some good points. For my part I think it isn't clear how human attitudes toward AI will evolve. It will likely be very complicated, even more so than our relationship with the internet or the smartphone. These things don't usually boil down to a simple "good" or "bad".
As for computing power needed to simulate a human brain, that depends on the simulation's fidelity. I.e., do all the details of our nervous system need to be faithfully simulated, or is a coarser-grained simulation enough. This has been hotly debated for decades and in the end it will be settled empirically.
"Enough" for what? For functional similarity? We can probably get functional similarity with LLM style transformers. Depending on how low your standards are we already have functional similarity with LLM style transformers. But that wouldn't be an uploaded human because it wouldn't think like a human - its brain structure would be totally different. Only its inputs and outputs would approximately match.
To really upload a human mind into a computer, the causal structure of its thoughts needs to be the same as the causal structure of thoughts within a human brain. For instance, if a certain thought involves a certain amount of activity in the hypothalamus, then the uploaded mind needs to have a corresponding amount of (and kind of) activity in a structure corresponding to the hypothalamus.
And that means we actually do need the simulation to be on a neural level. If it were possible to use substantially fewer neurons to do it, then evolution would have been able to make do with substantially fewer neurons in a substantially smaller and more energy-efficient brain. The fact evolution made this huge and energetically expensive brain with lots of neurons for us, means that it was necessary to do that, and a smaller brain wouldn't have worked.
I don't think anyone is debating the number of neurons. The human brain has a little under 100 billion.
The question is how accurately do we need to simulate the details of each neuron. At a coarse level we abstract a neuron's activity as an average firing rate, in which case a brain can be simulated today on a handful of GPUs. (Their connections being the big unknown.) Or we can get more fine-grained with e.g. Hodgkin–Huxley and model each individual action potential and how it propagates down the membrane. Penrose claims that consciousness relies on quantum superposition in even smaller structures.
The unknown is how much of this low-level detail is important to capture the essence of what's happening in the brain. The computational complexity ranges from "a handful of GPUs" to "more compute than exists in the entire world".
Not just average firing rate. Spike timing is important for synchronizing different parts of the brain. One learning rule (STDP) depends sensitively on when the spikes occur, on about a millisecond time scale. So you do need to simulate the neurons millisecond by millisecond.
100 billion neurons, 100 trillion synapses. We might represent each synapse with one or a few floating point numbers. So that would be around a petabyte of RAM, or perhaps a few petabytes. That's going to take more than a few GPUs. Furthermore, to update them at millisecond resolution, assuming a small number of floating point calculations per synapse per millisecond, would be several exaflops (N * 10^18 flops for some small N).
This might be in reach of some of our top supercomputers, like LineShine. If we were willing to devote one of them entirely to simulating a single human brain. And if we had any way of actually scanning those synapses from a real human brain, which we don't. We can't slice a brain into thin enough slices to see all the neurons.
There is a possibility we could use something like U-net to accelerate the computations, as has been done for physics. But still the problem of scanning the synapses is insurmountable for the foreseeable future. And you can't get around the amount of RAM required.
On top of which, there are things we still don't theoretically understand about how neurons learn.
AIs don't feel like descendants yet because, like your pebble example, they are not self-sustaining. If humans ceased to exist then AIs would quickly die out. Humans still need to push the pebbles around to make them move.
How much longer before the AIs become self-sustaining? Someday we may look on them as we do our children, satisfied in the knowledge that they will carry the torch forward when we're gone.
This argument proves too much. A species can have *many* descendant species, arbitrarily widely divergent from their current forms. This does not mean they would or should feel as though each equally carries on their legacy. A magically-made-sapient Cretaceous dinosaur would, presumably, not feel better about the impending Chicxulub crater based on the fact that billions of chickens now exist. There's a missing step, an unidentified missing piece of the connection that enables caring about descendants in the way you're claiming things will or should be.
Re: "The general Darwinian theory is that a tendency of things to cause similar future things with a similar tendency is selected for. When such selection is possible, it happens, and such things tend to spread everywhere possible. Those selected similar future things are “descendants”."
Do you have a reference for this definition of "descendants"? Or is this something you made up?
Hi Robin. Thanks for the piece. I don't know if you saw my little draft about Darwin, but I tagged you in it.
I had a question: is there any reason to think that these two robust habits are mutually exclusive? That is, doesn't my preference to indulge my decendants involve an aversion to competitors?
This post classifies "selected similar future things" as “descendants”. I don't think that's right. You would not classify your mother as a descendant - despite similarity. The same goes for her children and grand children - even though those may be classified as being "future things".
So: what are descendants, if not what is claimed in this post? I think you need an additional criteria along the lines of: the ways in which they are similar to you were copied from you - and not from some other common ancestor. If your heritable info gets mutated, probably your descendants inherit those mutations. This is not true of mutations in your younger siblings and their offspring - they are not your descendants.
Who Are Your "Descendants"? They are the other ones that come away if you cut the family tree at you. Some evolution is reticulated - so that is a simplification - but that is the general idea.
I was careful to use the word "cause" in my definition of Darwinian selection, and thus also in my definition of "descendant". Mere correlation is not enough.
So: if I pay my sister a million dollars to have a baby then that baby is my "descendant" - according to Hanson's definition - because I caused its birth. It is certainly counter-intuitive.
Is it a consequence of generalized Darwinism? I am not sure about that. Darwinism typically does not get involved in defining what counts as a "descendant" at the level of organisms. An important reason is because some simple creatures reproduce by splitting in two. Some single-celled organisms do this - as do some worms. In which case, there is no obvious asymmetry between parent and offspring - and between "ancestor" and "descendant".
As I understand it, the "ancestor" and "descendant" concepts make more sense at the level of genes.
Re: "for many hundreds of thousands of years human evolution has been driven mostly by a different, non-DNA, kind of Darwinian selection: cultural evolution." - more specifically the theory of meme-gene coevolution. Memes influence the environment in which genes evolve and genes influence the environment in which memes evolve.
This process has been going on for millions of years - not just hundreds of thousands of years - and it is responsible for our large brains, upright gait, babbling speech - and most of the other features that make us human.
So: humans give rise to descendant humans. Bibles give rise to descendant bibles. However these are pretty separate lineages that do not interbreed significantly. You don't get to claim that bibles are the descendants of humans just because once upon a time thousands of humans produced the first bibles. Descendants are creatures substantially similar to ancestors - including in theories of generalized Darwinism. Humans and bibles are not similar enough to qualify. They are more like different species in a symbiotic relationship.
What about intelligent machines? They can talk like humans. They share some knowledge with humans. There is some relationship. Perhaps like a 10th cousin - because of dilution with the many other people who contributed to their development. Notice that standard kin selection theory predicts that we don't give a crap about those.
I think the last paragraph will change few minds because people rarely assign moral weight to descendants qua descendants, they have specific attachments to specific human beings they have formed relationships with as well as humanity in general due to shared traits among humans such as the capacity for human-like experience. Maybe this will change when more see AIs as moral patients, but I’m not sure a majority ever will.
If the relevant AI entities are adaptive and have a dependant beneficial relationship with their creators, this will be selected against, as usual
Your last line is what people disagree with. People are worried AIs outcompete humans, so there are no humans left and all of our biological descentants who share all the biological human things with us that make us human, including dna, will die off. Most humans don't want this. It seems like you are being very obtuse about this point.
Expressing a "worry" isn't at all the same as showing it to be likely. And AI will in fact inherit a great many things from us.
Disingenuous because *you* believe it is likely that AIs will outcompete humans and our biological descendants will die off. You said, "DNA-encoding was a temporary choice; it wasn’t going to last forever. "
I also believe it is likely that AIs will outcompete humans and our biological descendants will die off. But unlike you I'm not convinced this is a good thing.
You're right, just because it's a worry doesn't make it likely. I've read and listened to you a fair bit on this subject and when you talk about AIs supplanting humans, I'm never quite clear on how you think that will go. You will say there may be a lot of AIs around in a few hundred years and not many humans, and that's ok. When people push back against that you say something like if it was over 100 thousand years people wouldn't care, but the shortness matters. There are implications in the shortness. Namely, that humans didn't get to steer that outcome. I must be missing something. Can you fill me in?
"AI will inherent a great many things from us" may or may not be true, depending on what you mean exactly. If it's the earth, then ok I agree. If it's something like remnants of culture because the current llms use our language and data then I have no idea and can see that getting vaporized if RSI leads down a different path.
Grant that AI is our descendant. My worry is what kind the process producing it right now (US-China race, firm-vs-firm sprint, open-weights leakage) systematically rewards shipping speed and raw capability over caution and value constraint. The benign inheritance "indulgence" quietly assumes doesn't survive that kind of selection pressure.
Very few evolutionary biologists would agree with your stretched meaning of the word "descendants." You are using it in the universal-Darwinism way that appeals to some futurists and AI philosophers. But the term is native to evolutionary biology, and there the word "gene" almost always refers to DNA genes, and "descendant" means biological offspring (or biological offspring of biological offspring, etc).
What you're failing to argue is why we should *care* about having AI "descendants." Many people do care about having biological human descendants, because our biological instincts and our traditional cultures have programmed us to care about that. Our biological instincts and our traditional cultures have not programmed us to care about having AI descendants, especially if it means the extinction of biological human descendants. Simply because you can stretch to use the word "descendant" for both cases does not imply that our existing preferences for human descendants would or should transfer from one case to the other.
The AIs we create would be very different from us in ways that human descendants would be similar to us. From this perspective, AI descendants are very inferior for the purpose of spreading creatures like ourselves. They think but not like we do, their brains do not work like our brains, their bodies do not work like our bodies. They might as well be alien invaders.
Darwin didn't know about DNA, so his use of "gene" couldn't have referred to DNA. And since culture evolution dominates human evolution, DNA is much less important for us re descendants.
Darwin certainly didn't know about Universal Darwinism (UD). He was concerned with evolution among living, biological organisms. He would not have considered a kicked stone to be a descendant of your foot, as you asserted in a comment.
It also doesn't matter how Darwin used the word - that's the etymological fallacy.
The definition of the word "gene" also doesn't matter. You can define it in the broad UD sense, that's fine, provided you make clear that's what you're doing. And you can define "descendants" in a UD sense too. But defining it that way does not provide any convincing argument that we should *care* about our UD "descendants" in the same way we care about our biological descendants. And that's the key that you're missing.
It's a fallacy of equivocation: "we want our descendants to prosper. AI is our descendants. Therefore we want AI to prosper." But the first use of "descendants" means "biological human descendants," (or at least that's the sense of the premise that 99% of people would be willing to accept) and the second use means "UD descendants," and you're equivocating between the two.
I think you have to speak for yourself here. Importance is subjective. In practice, people vary considerably in the extent to which they value their memes relative to the value they assign to their genes. Important factors are: how strong their memetic immune system is; the extent to which they are vulnerable to memetic hijacking; how virulent the memes they have been exposed to are; and which other parasites they have been infected with.
> What you're failing to argue is why we should *care* about having AI "descendants"
He's got his fingers in his ears on this issue and he ain't pulling them out
I expect we'll see the perceived "alienness" of AI come down in the future. The obvious case would be a brain emulation or mind uploading scenario, but even if those don't happen several things will come with high likelihood: it will start to learn through personal experience like a human does, it will have a physical presence, and it will be tailored by its creators to be appealing/useful to people. People are quick to get emotionally attached to pets of all kinds and I see no reason why AIs must be different.
Perceived alienness may come down when companies come out with friendly-looking helper robots. I dunno that the companies will be really successful. The robots won't look and seem exactly like a human, because that would be costly and risks the uncanny valley. Maybe they'll be plush mascots. But humans are known for discriminating against humans that look a little different from them, so I suspect if there's anger towards robots for taking jobs, humans will be angry and discriminatory towards the plush mascots.
Actual alienness coming down? Much more dubious. The brains of LLMs are nothing like the brains of humans.
Brain emulation and mind uploading are still very far away. The world computing power is insufficient to simulate a single human brain on the level of individual neurons. Plus we have no way of scanning individual neurons. AI will destabilize the humans-work-and-get-paid basis for the world economy long before we get anything like that.
You raise some good points. For my part I think it isn't clear how human attitudes toward AI will evolve. It will likely be very complicated, even more so than our relationship with the internet or the smartphone. These things don't usually boil down to a simple "good" or "bad".
As for computing power needed to simulate a human brain, that depends on the simulation's fidelity. I.e., do all the details of our nervous system need to be faithfully simulated, or is a coarser-grained simulation enough. This has been hotly debated for decades and in the end it will be settled empirically.
"Enough" for what? For functional similarity? We can probably get functional similarity with LLM style transformers. Depending on how low your standards are we already have functional similarity with LLM style transformers. But that wouldn't be an uploaded human because it wouldn't think like a human - its brain structure would be totally different. Only its inputs and outputs would approximately match.
To really upload a human mind into a computer, the causal structure of its thoughts needs to be the same as the causal structure of thoughts within a human brain. For instance, if a certain thought involves a certain amount of activity in the hypothalamus, then the uploaded mind needs to have a corresponding amount of (and kind of) activity in a structure corresponding to the hypothalamus.
And that means we actually do need the simulation to be on a neural level. If it were possible to use substantially fewer neurons to do it, then evolution would have been able to make do with substantially fewer neurons in a substantially smaller and more energy-efficient brain. The fact evolution made this huge and energetically expensive brain with lots of neurons for us, means that it was necessary to do that, and a smaller brain wouldn't have worked.
I don't think anyone is debating the number of neurons. The human brain has a little under 100 billion.
The question is how accurately do we need to simulate the details of each neuron. At a coarse level we abstract a neuron's activity as an average firing rate, in which case a brain can be simulated today on a handful of GPUs. (Their connections being the big unknown.) Or we can get more fine-grained with e.g. Hodgkin–Huxley and model each individual action potential and how it propagates down the membrane. Penrose claims that consciousness relies on quantum superposition in even smaller structures.
The unknown is how much of this low-level detail is important to capture the essence of what's happening in the brain. The computational complexity ranges from "a handful of GPUs" to "more compute than exists in the entire world".
Not just average firing rate. Spike timing is important for synchronizing different parts of the brain. One learning rule (STDP) depends sensitively on when the spikes occur, on about a millisecond time scale. So you do need to simulate the neurons millisecond by millisecond.
100 billion neurons, 100 trillion synapses. We might represent each synapse with one or a few floating point numbers. So that would be around a petabyte of RAM, or perhaps a few petabytes. That's going to take more than a few GPUs. Furthermore, to update them at millisecond resolution, assuming a small number of floating point calculations per synapse per millisecond, would be several exaflops (N * 10^18 flops for some small N).
This might be in reach of some of our top supercomputers, like LineShine. If we were willing to devote one of them entirely to simulating a single human brain. And if we had any way of actually scanning those synapses from a real human brain, which we don't. We can't slice a brain into thin enough slices to see all the neurons.
There is a possibility we could use something like U-net to accelerate the computations, as has been done for physics. But still the problem of scanning the synapses is insurmountable for the foreseeable future. And you can't get around the amount of RAM required.
On top of which, there are things we still don't theoretically understand about how neurons learn.
AIs don't feel like descendants yet because, like your pebble example, they are not self-sustaining. If humans ceased to exist then AIs would quickly die out. Humans still need to push the pebbles around to make them move.
How much longer before the AIs become self-sustaining? Someday we may look on them as we do our children, satisfied in the knowledge that they will carry the torch forward when we're gone.
Agree they can't exist w/o humans yet, but once they have human level abilities that will come soon after.
In The Selfish Gene, Richard Dawkins wrote explicitly about the descendants of nonliving replicators. I suspect he would agree with your analysis.
This argument proves too much. A species can have *many* descendant species, arbitrarily widely divergent from their current forms. This does not mean they would or should feel as though each equally carries on their legacy. A magically-made-sapient Cretaceous dinosaur would, presumably, not feel better about the impending Chicxulub crater based on the fact that billions of chickens now exist. There's a missing step, an unidentified missing piece of the connection that enables caring about descendants in the way you're claiming things will or should be.
Re: "The general Darwinian theory is that a tendency of things to cause similar future things with a similar tendency is selected for. When such selection is possible, it happens, and such things tend to spread everywhere possible. Those selected similar future things are “descendants”."
Do you have a reference for this definition of "descendants"? Or is this something you made up?
Hi Robin. Thanks for the piece. I don't know if you saw my little draft about Darwin, but I tagged you in it.
I had a question: is there any reason to think that these two robust habits are mutually exclusive? That is, doesn't my preference to indulge my decendants involve an aversion to competitors?
Thanks again, and have a nice night.
Sure the two concepts could sometimes overlap, leading to a mix of contrary tendencies.
In that case could my preference for my descendant be predicated on my defining alternatives as (relatively or absolutely) coexisting rivals?
This post classifies "selected similar future things" as “descendants”. I don't think that's right. You would not classify your mother as a descendant - despite similarity. The same goes for her children and grand children - even though those may be classified as being "future things".
So: what are descendants, if not what is claimed in this post? I think you need an additional criteria along the lines of: the ways in which they are similar to you were copied from you - and not from some other common ancestor. If your heritable info gets mutated, probably your descendants inherit those mutations. This is not true of mutations in your younger siblings and their offspring - they are not your descendants.
Who Are Your "Descendants"? They are the other ones that come away if you cut the family tree at you. Some evolution is reticulated - so that is a simplification - but that is the general idea.
I was careful to use the word "cause" in my definition of Darwinian selection, and thus also in my definition of "descendant". Mere correlation is not enough.
So: if I pay my sister a million dollars to have a baby then that baby is my "descendant" - according to Hanson's definition - because I caused its birth. It is certainly counter-intuitive.
Is it a consequence of generalized Darwinism? I am not sure about that. Darwinism typically does not get involved in defining what counts as a "descendant" at the level of organisms. An important reason is because some simple creatures reproduce by splitting in two. Some single-celled organisms do this - as do some worms. In which case, there is no obvious asymmetry between parent and offspring - and between "ancestor" and "descendant".
As I understand it, the "ancestor" and "descendant" concepts make more sense at the level of genes.
Re: "for many hundreds of thousands of years human evolution has been driven mostly by a different, non-DNA, kind of Darwinian selection: cultural evolution." - more specifically the theory of meme-gene coevolution. Memes influence the environment in which genes evolve and genes influence the environment in which memes evolve.
This process has been going on for millions of years - not just hundreds of thousands of years - and it is responsible for our large brains, upright gait, babbling speech - and most of the other features that make us human.
So: humans give rise to descendant humans. Bibles give rise to descendant bibles. However these are pretty separate lineages that do not interbreed significantly. You don't get to claim that bibles are the descendants of humans just because once upon a time thousands of humans produced the first bibles. Descendants are creatures substantially similar to ancestors - including in theories of generalized Darwinism. Humans and bibles are not similar enough to qualify. They are more like different species in a symbiotic relationship.
What about intelligent machines? They can talk like humans. They share some knowledge with humans. There is some relationship. Perhaps like a 10th cousin - because of dilution with the many other people who contributed to their development. Notice that standard kin selection theory predicts that we don't give a crap about those.
So you're saying we live in a rippling pond...
Each of us, unconsciously affecting the others.
Even thoughts and emotions may have simple but lasting repercussion across the globe.