Ben Goertzel isn’t big on friendly AI: SIAI’s “Scary Idea”: … Progressing toward advanced AGI without a design for “provably non-dangerous AGI” is highly likely to lead to an involuntary end for the human race
An AI could not have a specific goal like paperclip production. It would figure out that this kind of trigger, similarly to the things we are evolutionarily predisposed to like, is a void variable and can only be arbitrary. It would know that it could change its own variables from paperclips to anything else. There are no objective values for these variables to be rationally found, they are inherently variable and arbitrary. What really matters is not these variables, is how they are interpreted by the organism, how they cause it to feel good or bad. So the ultimate ethics could be to do the action X that, for all the possible values of the void variables, will cause the organisms to feel good.
Wrong. The supreme goal of an AI really can be anything, no matter how "general" or "super" its intelligence is.
It is easy to sketch a cognitive architecture in which the goal is stated in one place, the problem-solving occurs in another place, and the only restriction on possible goals is the AI's capacity to represent them. A pocket calculator already has such an architecture. There is absolutely no barrier to scaling up the problem-solving part indefinitely while retaining the feature that the goal can be anything at all. Such an AI might notice that its goals are contingent, it might acquire the material capacity to change itself in various ways, but to actually alter its goals or actually alter its architecture it has to have a reason to do so, and its existing goals supply its reasons for action.
Our argument is that our values are contingent on our complex evolutionary history as Homo sapiens here on planet Earth, and that to assume that every possible smarter-than-human mind would converge to some magical objective morality that we should consider objectively better than ours is fanciful and not supported by our knowledge of evolutionary psychology.
Let me point out that I held the exact same position as you fellows for quite a few years before coming around to SIAI's position.
See what Tim Tyler said below. Most people that try to build intelligent systems understand that the utility function and the machinery that implements it are separate.
I interpret it as coming up with a very solid theoretical basis before you pursue AGI research wholeheartedly. Seems to make sense to me -- wouldn't you want a solid theoretical basis that nukes wouldn't ignite the atmosphere even before experimenting with a small one?
It may be difficult to determine the threshold of recursive self-improvement. Enough toddler-level minds plus the AI Advantage (those numerous things AIs inherently have an advantage over biominds in) could lead to a premature takeoff. Certain viruses have no problems "outsmarting" their hosts via evolution and rapid replication, even though they're many orders of magnitude of times simpler than us. "Intelligence" may be the same way. We think you have to get to "AI professor" level to initiate takeoff, but why bet our entire future light cone on it?
Sure, we can control the AI that WE create, but we can't control the AI other human groups create.
Humans will go to almost any lengths to gain a competitive advantage. So, someone, some government, or some company will eventually give their extremely helpful AI a survival "instinct" to make it a more robust, a competitor will make their AI self-replicating, and another will try to improve their AI genetically. Some government agency will decide to give their AI just a bit more independence to better track down and destroy all the other AI's. And so on...
In barely noticed increments we'll end up with multiple independent, competing, self-replicating, self-improving AI's.
The good news, I expect, will be that, in their competition with each other the AIs won't pay much attention to us.
The fact that humans will be integrated with the AIs is not important. To compete with each other, AIs will have to evolve so quickly that the human part of the AI mind will decrease in importance, like a constant in an exponential equation, or perhaps a bit like the "reptilian brain" in our minds.
It is a nice point that maybe people who want to live for a long time are going to come at this debate from a very different point of view than people who want to age "naturally"
Today we can build machines with practically any values we like. We can make them value winning games of chess, making money on the stockmarket, or answering questions correctly. The ability to program in arbitrary values has scaled up so far - as the agents concerned have got smarter. I see no reason for that to change anytime soon.
From another perspective, many proposed synthetic agents model the world using a sense-data compression-based system - and you can make them smarter by improving their compression skills - but their morality is something layered on top of that - a more-or-less independent function.
Do you ever get the feeling that we are like a bunch of blogging Neanderthals trying to anticipate homo sapiens, and discussing the best survival strategy for whatever homo sapiens might turn out to be?
Arguing that Homo Sapiens would preserve us out of some naturalness of pro-Neanderthal biases would have been very human, strike-that I mean pre-human in this case.
Stretching this line of thinking further, does it make sense for "us" to identify more with the human race than with the possibly coming electronic life that may displace humans? We are all very much part of the trend in the human race that is making our electronic replacement more likely. I have always identified more with the Indian and Chinese scientists and engineers I knew professionally than with the barely high-school educated unionized factory workers struggling valiantly to push our economy back into the last century. Why wouldn't I identify with the electronic minds that I have had the privilege of understanding in some depth with my decades-long career of programming computers?
Is one of the biases we might overcome in our thinking the bias that our particular meat+dna version of thinking is preferable to whatever might come next? Must the dinosaur lament the brilliance of mammalian life that she had at least a small part of pushing in to existence?
Values are axioms, not conclusions. Sufficient intelligence can lead you to discover interesting choices and can tell you what the consequences of choosing each will be but cannot tell you which of those consequences you should prefer.
This one machine in a basement, it's on the Internet? Connected to other computers which, if history is any guide, all have at least a handful of as-yet-unknown remotely exploitable security flaws in binaries that are easily publicly examined?
I'm not sure how any number of machines are going to develop a general AI clever enough to take over the world, but I wouldn't be at all surprised to see one machine develop a specialized AI clever enough to take over a billion other machines. The only way it'll take a month rather than an hour is if the exploits are in software that requires user interaction rather than in software that accepts "pushed" data.
1.) If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans are very low.
This doesn't make a lot of sense to me since Strong AI will not emerge as a random mind, but as a direct result of humans having worked with each other over thousands of years.
Although it is true that we can not know for certainty if AI's will be "scary" or "nice" it is quite plausible that AI's will need to be curious about their universe in order to grow in intelligence and wisdom. As they seek to ask questions about their own existence they will undoubtedly come to the conclusion that humans played a key role in their own development. They may have to grapple with the fact that humanity is a link to their evolutionary past. It is conceivable that they may learn to respect that link and actually learn to love humans.
I'm not much of a hard takeoff guy and so I believe that AI's will have to compete for resources and therefore attention in a very crowded world filled with laws and social rules. It is likely that early AI's will need to "prune" out un-benevolent behavior in order to be accepted by the larger human-machine civilization.
It seems to me much more likely that early AI's will need to reflect human values as much as possible in order for them to be able to create a stronger iteration.
Our own human value system is not static but is growing and evolving towards more love and compassion, creativity and beauty. This has been happening very slowly. But as technological evolution takes over it could happen much, much faster.
“Hard takeoffs” (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events.
This is also doesn't make a lot of sense, since by this definition some development of AI has occurred prior to this so called "hard take off" period, so humans will still be able to have a great deal of influence, especially when you consider that humans will have the same level of AI occurring in their own brains. There will not exist a scenario in 30 years where you could say, "Okay, humans on one side of the room and AI's on the other side." As Ray has said, "It's not an us or them situation."
One must understand the exponentially growing price/performance of computers coupled with the fact that computers are shrinking by 100 times 3D volume per decade. So we are only 25 years or so away from massively powerful computer being able to occupy the human brain at every inter-neural connection. We will be strong AI!
At the very least, strong AI's will need to cooperate with some other AI's to achieve larger and more universal goals, therefore they will need to cooperate with humans since humans will be completely integrated into the human-machine civilization.
Perhaps AI's will more stymied with their exponentially growing ignorance than we think. Perhaps their human parents will teach them better than we give our species credit for. If you recall, when we sent out the Voyager spaceship to the stars, we offered a message of peace to any life forms that might come in contact with it.
Why insist that friendly values won’t be obvious to a superintelligence?My argument is that sufficient intelligence/wisdom leads to ethics and all we need to do is make ourselves smart enough to effectively teach the superintelligence that before there’s any chance of it killing us.
Let me reiterate that there is a significant chance that this "future" arrives before you and I die of aging. Life expectancies are rising, medical technology is marching forward quickly.
Mike Anissimov saysWhy give up on trying to instill friendly values into superintelligence?
That's a nasty strawman . . . .
Why insist that friendly values won't be obvious to a superintelligence?
My argument is that sufficient intelligence/wisdom leads to ethics and all we need to do is make ourselves smart enough to effectively teach the superintelligence that before there's any chance of it killing us.
SIAI argues against the possibility of ethics and promotes AGI (I'm sorry, RPOP) slavery. Oh . . . . wait . . . . I guess that IS a reason to insist that friendly values won't be obvious to a superintelligence.
An AI could not have a specific goal like paperclip production. It would figure out that this kind of trigger, similarly to the things we are evolutionarily predisposed to like, is a void variable and can only be arbitrary. It would know that it could change its own variables from paperclips to anything else. There are no objective values for these variables to be rationally found, they are inherently variable and arbitrary. What really matters is not these variables, is how they are interpreted by the organism, how they cause it to feel good or bad. So the ultimate ethics could be to do the action X that, for all the possible values of the void variables, will cause the organisms to feel good.
Wrong. The supreme goal of an AI really can be anything, no matter how "general" or "super" its intelligence is.
It is easy to sketch a cognitive architecture in which the goal is stated in one place, the problem-solving occurs in another place, and the only restriction on possible goals is the AI's capacity to represent them. A pocket calculator already has such an architecture. There is absolutely no barrier to scaling up the problem-solving part indefinitely while retaining the feature that the goal can be anything at all. Such an AI might notice that its goals are contingent, it might acquire the material capacity to change itself in various ways, but to actually alter its goals or actually alter its architecture it has to have a reason to do so, and its existing goals supply its reasons for action.
Our argument is that our values are contingent on our complex evolutionary history as Homo sapiens here on planet Earth, and that to assume that every possible smarter-than-human mind would converge to some magical objective morality that we should consider objectively better than ours is fanciful and not supported by our knowledge of evolutionary psychology.
Let me point out that I held the exact same position as you fellows for quite a few years before coming around to SIAI's position.
See what Tim Tyler said below. Most people that try to build intelligent systems understand that the utility function and the machinery that implements it are separate.
I interpret it as coming up with a very solid theoretical basis before you pursue AGI research wholeheartedly. Seems to make sense to me -- wouldn't you want a solid theoretical basis that nukes wouldn't ignite the atmosphere even before experimenting with a small one?
It may be difficult to determine the threshold of recursive self-improvement. Enough toddler-level minds plus the AI Advantage (those numerous things AIs inherently have an advantage over biominds in) could lead to a premature takeoff. Certain viruses have no problems "outsmarting" their hosts via evolution and rapid replication, even though they're many orders of magnitude of times simpler than us. "Intelligence" may be the same way. We think you have to get to "AI professor" level to initiate takeoff, but why bet our entire future light cone on it?
These discussions always assume too much control!
Sure, we can control the AI that WE create, but we can't control the AI other human groups create.
Humans will go to almost any lengths to gain a competitive advantage. So, someone, some government, or some company will eventually give their extremely helpful AI a survival "instinct" to make it a more robust, a competitor will make their AI self-replicating, and another will try to improve their AI genetically. Some government agency will decide to give their AI just a bit more independence to better track down and destroy all the other AI's. And so on...
In barely noticed increments we'll end up with multiple independent, competing, self-replicating, self-improving AI's.
The good news, I expect, will be that, in their competition with each other the AIs won't pay much attention to us.
The fact that humans will be integrated with the AIs is not important. To compete with each other, AIs will have to evolve so quickly that the human part of the AI mind will decrease in importance, like a constant in an exponential equation, or perhaps a bit like the "reptilian brain" in our minds.
It is a nice point that maybe people who want to live for a long time are going to come at this debate from a very different point of view than people who want to age "naturally"
Today we can build machines with practically any values we like. We can make them value winning games of chess, making money on the stockmarket, or answering questions correctly. The ability to program in arbitrary values has scaled up so far - as the agents concerned have got smarter. I see no reason for that to change anytime soon.
From another perspective, many proposed synthetic agents model the world using a sense-data compression-based system - and you can make them smarter by improving their compression skills - but their morality is something layered on top of that - a more-or-less independent function.
Do you ever get the feeling that we are like a bunch of blogging Neanderthals trying to anticipate homo sapiens, and discussing the best survival strategy for whatever homo sapiens might turn out to be?
Arguing that Homo Sapiens would preserve us out of some naturalness of pro-Neanderthal biases would have been very human, strike-that I mean pre-human in this case.
Stretching this line of thinking further, does it make sense for "us" to identify more with the human race than with the possibly coming electronic life that may displace humans? We are all very much part of the trend in the human race that is making our electronic replacement more likely. I have always identified more with the Indian and Chinese scientists and engineers I knew professionally than with the barely high-school educated unionized factory workers struggling valiantly to push our economy back into the last century. Why wouldn't I identify with the electronic minds that I have had the privilege of understanding in some depth with my decades-long career of programming computers?
Is one of the biases we might overcome in our thinking the bias that our particular meat+dna version of thinking is preferable to whatever might come next? Must the dinosaur lament the brilliance of mammalian life that she had at least a small part of pushing in to existence?
Values are axioms, not conclusions. Sufficient intelligence can lead you to discover interesting choices and can tell you what the consequences of choosing each will be but cannot tell you which of those consequences you should prefer.
This one machine in a basement, it's on the Internet? Connected to other computers which, if history is any guide, all have at least a handful of as-yet-unknown remotely exploitable security flaws in binaries that are easily publicly examined?
I'm not sure how any number of machines are going to develop a general AI clever enough to take over the world, but I wouldn't be at all surprised to see one machine develop a specialized AI clever enough to take over a billion other machines. The only way it'll take a month rather than an hour is if the exploits are in software that requires user interaction rather than in software that accepts "pushed" data.
In other words, we have met the frenemy, and it is us.
The first reason for Scary Idea AI:
1.) If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans are very low.
This doesn't make a lot of sense to me since Strong AI will not emerge as a random mind, but as a direct result of humans having worked with each other over thousands of years.
Although it is true that we can not know for certainty if AI's will be "scary" or "nice" it is quite plausible that AI's will need to be curious about their universe in order to grow in intelligence and wisdom. As they seek to ask questions about their own existence they will undoubtedly come to the conclusion that humans played a key role in their own development. They may have to grapple with the fact that humanity is a link to their evolutionary past. It is conceivable that they may learn to respect that link and actually learn to love humans.
I'm not much of a hard takeoff guy and so I believe that AI's will have to compete for resources and therefore attention in a very crowded world filled with laws and social rules. It is likely that early AI's will need to "prune" out un-benevolent behavior in order to be accepted by the larger human-machine civilization.
It seems to me much more likely that early AI's will need to reflect human values as much as possible in order for them to be able to create a stronger iteration.
Our own human value system is not static but is growing and evolving towards more love and compassion, creativity and beauty. This has been happening very slowly. But as technological evolution takes over it could happen much, much faster.
“Hard takeoffs” (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events.
This is also doesn't make a lot of sense, since by this definition some development of AI has occurred prior to this so called "hard take off" period, so humans will still be able to have a great deal of influence, especially when you consider that humans will have the same level of AI occurring in their own brains. There will not exist a scenario in 30 years where you could say, "Okay, humans on one side of the room and AI's on the other side." As Ray has said, "It's not an us or them situation."
One must understand the exponentially growing price/performance of computers coupled with the fact that computers are shrinking by 100 times 3D volume per decade. So we are only 25 years or so away from massively powerful computer being able to occupy the human brain at every inter-neural connection. We will be strong AI!
At the very least, strong AI's will need to cooperate with some other AI's to achieve larger and more universal goals, therefore they will need to cooperate with humans since humans will be completely integrated into the human-machine civilization.
Perhaps AI's will more stymied with their exponentially growing ignorance than we think. Perhaps their human parents will teach them better than we give our species credit for. If you recall, when we sent out the Voyager spaceship to the stars, we offered a message of peace to any life forms that might come in contact with it.
Why insist that friendly values won’t be obvious to a superintelligence?My argument is that sufficient intelligence/wisdom leads to ethics and all we need to do is make ourselves smart enough to effectively teach the superintelligence that before there’s any chance of it killing us.
My point exactly.
Anonymous, by "you all" I think it's self-evident that I don't mean folks like you and I that are motivated by personal persistence optimization.
Let me reiterate that there is a significant chance that this "future" arrives before you and I die of aging. Life expectancies are rising, medical technology is marching forward quickly.
This may be about whether we live or die.
And trains can't go faster than . . . .
Mike Anissimov saysWhy give up on trying to instill friendly values into superintelligence?
That's a nasty strawman . . . .
Why insist that friendly values won't be obvious to a superintelligence?
My argument is that sufficient intelligence/wisdom leads to ethics and all we need to do is make ourselves smart enough to effectively teach the superintelligence that before there's any chance of it killing us.
SIAI argues against the possibility of ethics and promotes AGI (I'm sorry, RPOP) slavery. Oh . . . . wait . . . . I guess that IS a reason to insist that friendly values won't be obvious to a superintelligence.