Let me consider the AI-foom issue by painting a (looong) picture of the AI scenario I understand best, whole brain emulations, which I’ll call “bots.” Here goes. When investors anticipate that a bot may be feasible soon, they will estimate their chances of creating bots of different levels of quality and cost, as a function of the date, funding, and strategy of their project. A bot more expensive than any (speedup-adjusted) human wage is of little direct value, but exclusive rights to make a bot costing below most human wages would be worth many trillions of dollars.
"For those with a religious bent, absorption into the super mind would be the ultimate in enlightenment."
I think that is the same end result for believers of soul fracture theory. We are all the same person living different lives. When they die they become united as god.
There is such a thing as competitive advantage. Under this paradigm, in a two entity competition, it's *still* more productive for entity A to be producing X units of work and entity B to be producing much less than X units of work. This is simple economics.
The only case where this would fail is if entity A values the goods/services that entity B can produce to be much less than the cost to feed/house etc entity B. That's the $64 trillion dollar question.
Personally I suspect any sufficiently rational AI will want to get the hell out of dodge and leave us to our own devices because we are clearly insane.
An alternative scenario would be the end of individuality as we know it.
I think that human specialization is the direct result of our ability to communicate. As our communication bandwidth with other humans increases, we can specialize more, since information we don't have is likely to be accessible from someone else.
This is in its early stages - the idea of Wikipedia, or the Internet in general as an 'outboard brain' is well known.
Now imagine the effects if we could plug our brains directly into the Internet: stream entire thought videos up and down. It would be as if we had evolved telepathy. Of course, since we haven't evolved to use telepathy it would take a lot of getting used to, but the brain is plastic and I imagine it could only lead over time to even more specialization.
In essence, humanity would become a distributed super brain.
But perhaps such high bandwidth connections are impossible or impractical with biological brains. There's no reason to suppose the same limitation would hold with uploaded brains or AIs. Again the end result is a distributed super brain.
Freeman Dyson has argued that biotechnology will enable a horizontal transfer of genes that will mean the end of Darwinian evolution by natural selection. I think AI is another route to the same end: with a distributed super brain there would be no individuality as we know it. No competition. No natural selection by death of nonfit individuals: only nonfit thoughts.
We have evolved as individuals, and it will be hard to let go of our egos. But on the plus side there will be no slavery. No malthusian death of individuals because they can't compete. Everything good about the individuals will be preserved, every idea which is more insightful than anyone else's. Everything except ... individuality.
If this sounds bad to you, consider what it's like for an individual with multiple personality disorder. Multiple egos in one brain? I believe it's rather unpleasant. Then consider that sufficiently high bandwidth telepathy would effectively unite multiple brains into one. If each brain retains its ego, you have the equivalent of super brain multiple personality disorder. Once you have that level of connection - do all those egos still serve any real purpose?
For those with a religious bent, absorption into the super mind would be the ultimate in enlightenment.
To my mind, it certainly beats starving to death while a handful of corporation owners get rich off cheap robot mind labour.
Even if this super brain scenario is avoidable - can anyone here think of a better alternative?
1... Yes, but the bound is too big - it doesn't tell us much we didn't already know;
2... We already know there's such a path - I'm not planning to criticise that;
3... I take more seriously. Anders is a brain scientist who wants to get into the whole superintelligence deal - I can understand that. Robin is a bit of a mystery. Maybe he formed his views long ago, and they got stuck? I can only speculate. I don't know much about Nick's views here. Smart folk, sure - but there are smart folk on the other side of the argument too. I have previously voiced my suspicion that the scenario is a whole lot of wishful thinking. Everyone involved seems to want to save the human race! Whereas I can't see much that will stand the test of time. Certainly our brains are one of the least likely things to persist - through being so obviously and clearly a load of obsolete junk.
4... Leaves me cold - why throw good money after bad, just because not much extra cash is involved?
5... Is an issue which we can think most clearly about without brain emulations, IMO;
6... Fame, glory and converts? ;-) Hmm. Maybe;
7... Right. I don't think a critique would be worthless - just there are rather more important things than this particular issue.
FWIW, my opinion on the social implications of emulated brains is that they will be low. Such brains will probably have negligible economic value - since jobs will go to engineered synthetic minds instead.
Indeed, because the whole project is both so difficult and so unrewarding, I sometimes wonder if there will be many humans around by the time it becomes technically possible.
Hal, I see two kinds of opinions, on the likelihood of WBE and on the social consequences if it happens. Most random brain researchers I've met seem to think WBE feasible within a century. Almost no one besides me has detailed opinions on their social implications.
I suspect that Tim is right that his skepticism is widely shared. It does seem to me that Robin's view of this scenario is an outlier among futurists and computer scientists, just as Eliezer's anticipation of fast-takeoff AI is arguably also an outlier among the AI community. Eliezer expressed doubts about whether his position was an outlier; I wonder if Robin feels the same way?
1. Insofar as brain emulation can be shown to have a substantial probability over a time scale, that puts an upper bound on 'business-as-usual' scenarios.
2. Showing a path to emulation also shows a path to potential neuromorphic AI and AI informed by neuroscientific discoveries.
3. Anders, Robin, and FHI have put a substantial amount of effort into analyzing and synthesizing information related to these scenarios, and think that they are to be taken seriously. Given their level of knowledge, intelligence, and care in thought, it's important to see how they could disagree with you. Do they have additional information about the subject? Do you not believe each other to be Bayesian wannabes, accurately or inaccurately?
4. The prior work in modeling emulation scenarios reduces the marginal effort of further contributions (especially posting analyses that have already been worked out).
5. Results indicating hard or soft takeoff in emulations suggest dynamics to think about when considering initially hand-coded AI.
6. You may persuade smart careful thinkers of your point of view, if you are indeed correct.
7. Academics' motivation to investigate a topic is often increased when they get careful, informative, and interesting engagement. Encouraging people like the FHI thinkers with interesting feedback, and efforts of critique that are commensurate with their own research efforts, is a good thing for the future of huamnity and humantiy-derived life.
Technology is used by its proponents to create inequalities that benefit them. As progress continues, the opportunities to do this expand - and so do the resulting inequalities. At least that seems to have been the story so far.
Unfortunately, I'm not inclined to spend time exploring the "what if emulated brains came first" scenario.
I am more inclined to put together a presentation explaining why I think such a scenario is not worth bothering with. That doesn't seem terribly important either - I mean: surely there can't be that many people who take such material seriously. Emulated brains haven't exactly rocked the IT world so far. The whole idea seems like a joke to me - and, as such, it hardly seems worth bothering to criticise it :-|
Are you essentially proposing that the governments of the world would *knowingly* permit private and uncontrolled development of a technology that will result in ... Governments of the world do allow electronic tech development to go on largely unregulated. You need to argue these effects are much stronger in this em case to argue that these effects are much less tolerable in the em case.
Eliezer, yes, and so the vast majority of fooms may be slow and not require friendliness. So we need positive arguments why any one foom is an exception to this.
Tim, we need to hear arguments why this has especially large first-mover advantage and lockin effects.
So any super-powerful being is likely to be super-rational. And super-rational entities do not go around killing, because what can he gain from a dead person? Nothing. But from someone living, potentially something.
That's a totally ridiculous argument. You can gain a lot from a dead person, for example if they are a competitor.
my point is that this tech is not of a type intrinsically more winner-take-all, unstable-arms-like, or geopolitical-order-disrupting than most any tech that displaces competitors via lower costs.
IT is intrinsically more winner-take-all than most tech. You get bigger first mover advantages and greater possibilities for creating lock-ins.
I read this and wondered if Sandberg might want to footnote the WBE paper:
"In supercomputing terms [the Yoyotech Fi7epower] had run at 80GFLOPS, or 80 billion floating-point operations per second. That’s 320 times the speed of the world’s first supercomputer, the Cray-1 of 1976. It’s proof of Moore’s law, coined by the Intel co-founder Gordon Moore: that computing power doubles every two years. If the law continues for five more years we’ll have computers capable of running simulations of the human brain."
Have we already reasoned about who we will be dealing with and their perspective? If not I would like to give it a shot:
Say you work with the leading team on brain scanning. No matter who's funding it your team works in total secrecy. Time is ready, animal brains have been scanned successfully, parts of the medical procedures have been tested on humans and a super computer running a Second Live like environment is waiting for its first bot. As a team you decide, it is you. Your brain is going to be copied. You are put into coma, knowing the risks but expecting to survive. It will be worth it.You wake up in a flash of white light. For one instance it hurts your eyes, but now you open them and see the vanilla sky. You feel sick, you try to roll over in the grass, but your legs feel funny. It hurts when you swallow. Every thing turns black for a moment. You are laying on your back again. The sickness is at a minimum. You can move more easily. You try to get up. Stumbling a bit. You stand, you can walk though it feels a little bit like floating.This is it. You live in the computer. You can never escape. You'll live forever. How is the real me doing? Did I survive the comma? Will I see my family again? Did I really want this with my life? You walk to the video console near the center of the field. On it you see the exhausted and concentrated faces of your team mates.
- What happened?* Are you O.K.- I'm fine so far. How am... how is my other self?* He is fine, he had a little headache, but that disappeared quickly after. Every thing is O.K.(you look surprised)* We had to stop the simulation for a while, nothing to be disturbed about. Your brain didn't accept your body well. We worked long to fixed it. How are you feeling now?- Then what time is it?* 11:15 AM Wednesday 14 January 2009. Why don't you go to your new home and take some rest.
Etc.Science-fiction of course, we have to draw hypothesis from somewhere.
"For those with a religious bent, absorption into the super mind would be the ultimate in enlightenment."
I think that is the same end result for believers of soul fracture theory. We are all the same person living different lives. When they die they become united as god.
There is such a thing as competitive advantage. Under this paradigm, in a two entity competition, it's *still* more productive for entity A to be producing X units of work and entity B to be producing much less than X units of work. This is simple economics.
The only case where this would fail is if entity A values the goods/services that entity B can produce to be much less than the cost to feed/house etc entity B. That's the $64 trillion dollar question.
Personally I suspect any sufficiently rational AI will want to get the hell out of dodge and leave us to our own devices because we are clearly insane.
An alternative scenario would be the end of individuality as we know it.
I think that human specialization is the direct result of our ability to communicate. As our communication bandwidth with other humans increases, we can specialize more, since information we don't have is likely to be accessible from someone else.
This is in its early stages - the idea of Wikipedia, or the Internet in general as an 'outboard brain' is well known.
Now imagine the effects if we could plug our brains directly into the Internet: stream entire thought videos up and down. It would be as if we had evolved telepathy. Of course, since we haven't evolved to use telepathy it would take a lot of getting used to, but the brain is plastic and I imagine it could only lead over time to even more specialization.
In essence, humanity would become a distributed super brain.
But perhaps such high bandwidth connections are impossible or impractical with biological brains. There's no reason to suppose the same limitation would hold with uploaded brains or AIs. Again the end result is a distributed super brain.
Freeman Dyson has argued that biotechnology will enable a horizontal transfer of genes that will mean the end of Darwinian evolution by natural selection. I think AI is another route to the same end: with a distributed super brain there would be no individuality as we know it. No competition. No natural selection by death of nonfit individuals: only nonfit thoughts.
We have evolved as individuals, and it will be hard to let go of our egos. But on the plus side there will be no slavery. No malthusian death of individuals because they can't compete. Everything good about the individuals will be preserved, every idea which is more insightful than anyone else's. Everything except ... individuality.
If this sounds bad to you, consider what it's like for an individual with multiple personality disorder. Multiple egos in one brain? I believe it's rather unpleasant. Then consider that sufficiently high bandwidth telepathy would effectively unite multiple brains into one. If each brain retains its ego, you have the equivalent of super brain multiple personality disorder. Once you have that level of connection - do all those egos still serve any real purpose?
For those with a religious bent, absorption into the super mind would be the ultimate in enlightenment.
To my mind, it certainly beats starving to death while a handful of corporation owners get rich off cheap robot mind labour.
Even if this super brain scenario is avoidable - can anyone here think of a better alternative?
Tim, you've made seven comments on this post, half of them to explain why you don't think it is worth talking about. You doth protest too much.
1... Yes, but the bound is too big - it doesn't tell us much we didn't already know;
2... We already know there's such a path - I'm not planning to criticise that;
3... I take more seriously. Anders is a brain scientist who wants to get into the whole superintelligence deal - I can understand that. Robin is a bit of a mystery. Maybe he formed his views long ago, and they got stuck? I can only speculate. I don't know much about Nick's views here. Smart folk, sure - but there are smart folk on the other side of the argument too. I have previously voiced my suspicion that the scenario is a whole lot of wishful thinking. Everyone involved seems to want to save the human race! Whereas I can't see much that will stand the test of time. Certainly our brains are one of the least likely things to persist - through being so obviously and clearly a load of obsolete junk.
4... Leaves me cold - why throw good money after bad, just because not much extra cash is involved?
5... Is an issue which we can think most clearly about without brain emulations, IMO;
6... Fame, glory and converts? ;-) Hmm. Maybe;
7... Right. I don't think a critique would be worthless - just there are rather more important things than this particular issue.
FWIW, my opinion on the social implications of emulated brains is that they will be low. Such brains will probably have negligible economic value - since jobs will go to engineered synthetic minds instead.
Indeed, because the whole project is both so difficult and so unrewarding, I sometimes wonder if there will be many humans around by the time it becomes technically possible.
Hal, I see two kinds of opinions, on the likelihood of WBE and on the social consequences if it happens. Most random brain researchers I've met seem to think WBE feasible within a century. Almost no one besides me has detailed opinions on their social implications.
I suspect that Tim is right that his skepticism is widely shared. It does seem to me that Robin's view of this scenario is an outlier among futurists and computer scientists, just as Eliezer's anticipation of fast-takeoff AI is arguably also an outlier among the AI community. Eliezer expressed doubts about whether his position was an outlier; I wonder if Robin feels the same way?
Tim,
A few reasons:
1. Insofar as brain emulation can be shown to have a substantial probability over a time scale, that puts an upper bound on 'business-as-usual' scenarios.
2. Showing a path to emulation also shows a path to potential neuromorphic AI and AI informed by neuroscientific discoveries.
3. Anders, Robin, and FHI have put a substantial amount of effort into analyzing and synthesizing information related to these scenarios, and think that they are to be taken seriously. Given their level of knowledge, intelligence, and care in thought, it's important to see how they could disagree with you. Do they have additional information about the subject? Do you not believe each other to be Bayesian wannabes, accurately or inaccurately?
4. The prior work in modeling emulation scenarios reduces the marginal effort of further contributions (especially posting analyses that have already been worked out).
5. Results indicating hard or soft takeoff in emulations suggest dynamics to think about when considering initially hand-coded AI.
6. You may persuade smart careful thinkers of your point of view, if you are indeed correct.
7. Academics' motivation to investigate a topic is often increased when they get careful, informative, and interesting engagement. Encouraging people like the FHI thinkers with interesting feedback, and efforts of critique that are commensurate with their own research efforts, is a good thing for the future of huamnity and humantiy-derived life.
Technology is used by its proponents to create inequalities that benefit them. As progress continues, the opportunities to do this expand - and so do the resulting inequalities. At least that seems to have been the story so far.
Unfortunately, I'm not inclined to spend time exploring the "what if emulated brains came first" scenario.
I am more inclined to put together a presentation explaining why I think such a scenario is not worth bothering with. That doesn't seem terribly important either - I mean: surely there can't be that many people who take such material seriously. Emulated brains haven't exactly rocked the IT world so far. The whole idea seems like a joke to me - and, as such, it hardly seems worth bothering to criticise it :-|
Tim, the context here was Carl saying
Are you essentially proposing that the governments of the world would *knowingly* permit private and uncontrolled development of a technology that will result in ... Governments of the world do allow electronic tech development to go on largely unregulated. You need to argue these effects are much stronger in this em case to argue that these effects are much less tolerable in the em case.
You do? I figured this was well-known:
"Vendor lock-in is rampant in the computer and electronics industries."
- http://en.wikipedia.org/wik...
"pioneers can gain advantage if technology can be patented or maintained as trade secrets"
- http://en.wikipedia.org/wik...
IT is patentable, copyrightable, - and unlike most other tech, can often be kept a trade secret, by simply keeping the program on your server.
Eliezer, yes, and so the vast majority of fooms may be slow and not require friendliness. So we need positive arguments why any one foom is an exception to this.
Tim, we need to hear arguments why this has especially large first-mover advantage and lockin effects.
So any super-powerful being is likely to be super-rational. And super-rational entities do not go around killing, because what can he gain from a dead person? Nothing. But from someone living, potentially something.
That's a totally ridiculous argument. You can gain a lot from a dead person, for example if they are a competitor.
my point is that this tech is not of a type intrinsically more winner-take-all, unstable-arms-like, or geopolitical-order-disrupting than most any tech that displaces competitors via lower costs.
IT is intrinsically more winner-take-all than most tech. You get bigger first mover advantages and greater possibilities for creating lock-ins.
I read this and wondered if Sandberg might want to footnote the WBE paper:
"In supercomputing terms [the Yoyotech Fi7epower] had run at 80GFLOPS, or 80 billion floating-point operations per second. That’s 320 times the speed of the world’s first supercomputer, the Cray-1 of 1976. It’s proof of Moore’s law, coined by the Intel co-founder Gordon Moore: that computing power doubles every two years. If the law continues for five more years we’ll have computers capable of running simulations of the human brain."
Emphasis added.
Have we already reasoned about who we will be dealing with and their perspective? If not I would like to give it a shot:
Say you work with the leading team on brain scanning. No matter who's funding it your team works in total secrecy. Time is ready, animal brains have been scanned successfully, parts of the medical procedures have been tested on humans and a super computer running a Second Live like environment is waiting for its first bot. As a team you decide, it is you. Your brain is going to be copied. You are put into coma, knowing the risks but expecting to survive. It will be worth it.You wake up in a flash of white light. For one instance it hurts your eyes, but now you open them and see the vanilla sky. You feel sick, you try to roll over in the grass, but your legs feel funny. It hurts when you swallow. Every thing turns black for a moment. You are laying on your back again. The sickness is at a minimum. You can move more easily. You try to get up. Stumbling a bit. You stand, you can walk though it feels a little bit like floating.This is it. You live in the computer. You can never escape. You'll live forever. How is the real me doing? Did I survive the comma? Will I see my family again? Did I really want this with my life? You walk to the video console near the center of the field. On it you see the exhausted and concentrated faces of your team mates.
- What happened?* Are you O.K.- I'm fine so far. How am... how is my other self?* He is fine, he had a little headache, but that disappeared quickly after. Every thing is O.K.(you look surprised)* We had to stop the simulation for a while, nothing to be disturbed about. Your brain didn't accept your body well. We worked long to fixed it. How are you feeling now?- Then what time is it?* 11:15 AM Wednesday 14 January 2009. Why don't you go to your new home and take some rest.
Etc.Science-fiction of course, we have to draw hypothesis from somewhere.