The outcomes within any space-time region can be seen as resulting from 1) preferences of various actors able to influence the universe in that region, 2) absolute and relative power and influence of those actors, and 3) constraints imposed by the universe. Changes in outcomes across regions result from changes in these factors.
I think the argument is that humans would "diverge" into multiple competing or isolated non-competing value systems and civilizations. I do not believe it's reasonable to expect value systems would "converge" on only one location or state in value space. The distribution of values would like be fractal not a convergence on a point or single universal. IMO.
Does Foom create value space expansion? Perhaps some foom produces a from of value space expansion that produces breakaway civilizations and throw back civilizations at the same the time in largish numbers. Foom can be constructed to impose barriers to transmission inherently and may be self contained or constrained.
I have for you working "foom", "an innovation that is extremely lumpy compared to historical experience, and in addition posits an unusually high difficulty of copying or complementing this innovation." I am evaluating "the next big jump in growth rates"
The Value drift problem is not a significant problem because there is value evolution and divergence continually along with value containment strategies and value ecosystems that emerge. Basically the foom I'm working with multiplies the ecosystem space for values and also implements a sort of space between them to isolate and individualize value systems. But the ecosystems are still dependent on each other so that foom can possibly spread from ecosystem to ecosystem, but only sometimes and only by creating non-foom ecosystems.
Um, this particular foom also offers an almost immediate solution to poverty over most the world. It's an old foom that was suppressed for a long time and crippled over history and is currently very weak in our economic ecosystems and is known as "haggling". I evolved this foom (haggling) to make it faster and better and work in our modern economy. The solution to poverty is highly profitable and viral. It does not require AI to understand and is human generated with AI assistance, but basically requires humans for functioning and is of humans, by humans and for humans inherently. It appears benevolent and not that complicated to understand compared to existing government and political economic foom's in existence already that do less good. The URL for a better foom output that solves poverty and politics with human guidance and does nothing without human motivation and choice is https://sites.google.com/vi...
p.s. If this information is prohibited here, let me know and I'll delete my comment to censor my self at your request. The URL links only to output from foom and describes a foom generated solution to poverty and politics. The foom itself is inactive and disabled intentionally at this time and at the URL provided. I believe you may read it safely.
The values chosen by competition should be strongly dependent on the nature of the creature. Even our closest relatives the chimps seem to have somewhat different values than us. AI is quite different than us in how it reproduces. AI probably doesn't even look like a single species and certainly not like us. Our values have us treat other species even more poorly than we treat each other. What if AI optimizes to treat us the way we treat wolves or rats? What optimizes prevalence of AIs is what competition points towards. Do you really have no concern that what increases AI prevalence might not include constraining or starving humans?
Obviously humans have long ago escaped any constraint not to murder to let people starve to death. Naturally rare events are not much constrained by "default values".
Human nature also constrains values. The most elementary relate to the prohibition of murder and a duty to not let people starve to death -- which are actually related, since you can't expect whom is refused the basic needed to survive would respect any social contract.
If there possibly existed exceptions through history, I think it's pretty clear that the above are spontaneous default values...
...OTOH, the speech of "space-time regions" reminds me of a complex of ideas I'd developed around the end of the cold war and the first Gulf war. A matter of associating the speeches of leaders about "lesser evil" to concrete measures of aggregate evil. You need "space-time regions" to specify the set of events whose evil you'd aggregate.
The measures of evil I had in mind, where (1) the aggregate of casualties, regardless of side (2) a function of the kill ratio, given the consideration that when the kill ratio diverges, the situation morphs from one of war (with an ingredient of legitimate self-defense) to one of genocide (with no ingredient of legitimate self-defense).
"As this trend looks like a random walk, we see no obvious limit to how far values can drift."
But a random walk in a space of what dimensionality?Drifts towards the left or the right eventually reverse themselves -but in a high dimensional space, one never stumbles home again...
Yes, sorry, to be clear this was meant as a response to Paul's comments below, his claim c) here in particular. Though it applies to similar arguments others have made before, e.g. Yudkowsky's CEV idea.
The AI transition will be a time of great upheaval in any case.
Times of great upheaval test value systems, even ones tethered to an underlying stable evolved nature.
In the AI transition, that tether will be much weaker; unlike human values, AI values can change with the next SW update or parameter revision. And it is not clear what would compensate for the weakening of the tether.
So the likelihood of AI values changing (thus the risk to our most critical values) is higher in the AI transition than in previous human experience.
Robin, are you questioning that an entity's critical values are tied to its nature, or that human nature is less malleable than AIs' natures will be?
As to the overall question of the risk of the AI transition to our critical values, the burden of the proof should be on those claiming that the risk is manageable thus not deserving of extraordinary focus.
It seems to me that the claim that humans would converge on a set of true values given enough information and time to reflect on it makes (at least in its explicit argument) the unjustified assumption that values depend only on information. That is, the argument is that people hear more and better arguments that make them change their minds, and eventually after hearing all the arguments they converge on the truth - in this case the things they truly value, in their heart of hearts.
The problem is that values depend on all sorts of factors, not just info. So while the above argument is just about plausible as a defense of values chosen in light of more info, I don't see how you can claim any other relevant factor as producing truer values. Are true human values produced by rich humans, or poor? High status humans, or low status? Angry humans, sad humans, calm humans? Working or relaxing humans? And, well, why?
"As most human value drift is mediated by new humans, I don’t see why new humans with new values are less of a problem than new AIs with new values"
The nature of new humans is much more like previous humans, *and* much less malleable, than that of AIs. Critical values of the kind listed by Cambias are thus at much higher risk of going by the wayside in an AI transition, than in business-as-usual (or even previous-transition) handoffs among human generations.
Funny, this topic was addressed in a book by a guy named Robin Hanson, who felt pretty comfortable with using social science to predict that technological realities will cause values of ems to drift toward those of human farmers. I don't think that he saw the future as some hazy question mark. If anything, he was downright preachy about how we can and should use the insights from the social sciences to extrapolate credible predictions about how we will live, love and value in the future. Anyway, that guy was right. Some material realities of the future will surely surprise us, but our moral psychology is hard-wired enough to produce values that responsible academics can responsibly speculate about.
Edit: Maybe the best way to interpret the parent comment is that we underestimate the weirdness of future material realities, and if knew what we're really in for, we would worry more about the value drift this weirdness would cause. OK, maybe. I do think that some of us overextrapolate the recent trend in which the growth of technology correlates with a growth of our options. Age of Em is a great case study of how this correlation might reverse. But as long as our option-space keeps growing, it's not easy to picture why we would choose a world which is a worse fit for our innate value preferences. It's a good sci-fi premise.
I think the argument is that humans would "diverge" into multiple competing or isolated non-competing value systems and civilizations. I do not believe it's reasonable to expect value systems would "converge" on only one location or state in value space. The distribution of values would like be fractal not a convergence on a point or single universal. IMO.
Does Foom create value space expansion? Perhaps some foom produces a from of value space expansion that produces breakaway civilizations and throw back civilizations at the same the time in largish numbers. Foom can be constructed to impose barriers to transmission inherently and may be self contained or constrained.
I have for you working "foom", "an innovation that is extremely lumpy compared to historical experience, and in addition posits an unusually high difficulty of copying or complementing this innovation." I am evaluating "the next big jump in growth rates"
The Value drift problem is not a significant problem because there is value evolution and divergence continually along with value containment strategies and value ecosystems that emerge. Basically the foom I'm working with multiplies the ecosystem space for values and also implements a sort of space between them to isolate and individualize value systems. But the ecosystems are still dependent on each other so that foom can possibly spread from ecosystem to ecosystem, but only sometimes and only by creating non-foom ecosystems.
Um, this particular foom also offers an almost immediate solution to poverty over most the world. It's an old foom that was suppressed for a long time and crippled over history and is currently very weak in our economic ecosystems and is known as "haggling". I evolved this foom (haggling) to make it faster and better and work in our modern economy. The solution to poverty is highly profitable and viral. It does not require AI to understand and is human generated with AI assistance, but basically requires humans for functioning and is of humans, by humans and for humans inherently. It appears benevolent and not that complicated to understand compared to existing government and political economic foom's in existence already that do less good. The URL for a better foom output that solves poverty and politics with human guidance and does nothing without human motivation and choice is https://sites.google.com/vi...
p.s. If this information is prohibited here, let me know and I'll delete my comment to censor my self at your request. The URL links only to output from foom and describes a foom generated solution to poverty and politics. The foom itself is inactive and disabled intentionally at this time and at the URL provided. I believe you may read it safely.
The values chosen by competition should be strongly dependent on the nature of the creature. Even our closest relatives the chimps seem to have somewhat different values than us. AI is quite different than us in how it reproduces. AI probably doesn't even look like a single species and certainly not like us. Our values have us treat other species even more poorly than we treat each other. What if AI optimizes to treat us the way we treat wolves or rats? What optimizes prevalence of AIs is what competition points towards. Do you really have no concern that what increases AI prevalence might not include constraining or starving humans?
Obviously humans have long ago escaped any constraint not to murder to let people starve to death. Naturally rare events are not much constrained by "default values".
Human nature also constrains values. The most elementary relate to the prohibition of murder and a duty to not let people starve to death -- which are actually related, since you can't expect whom is refused the basic needed to survive would respect any social contract.
If there possibly existed exceptions through history, I think it's pretty clear that the above are spontaneous default values...
...OTOH, the speech of "space-time regions" reminds me of a complex of ideas I'd developed around the end of the cold war and the first Gulf war. A matter of associating the speeches of leaders about "lesser evil" to concrete measures of aggregate evil. You need "space-time regions" to specify the set of events whose evil you'd aggregate.
The measures of evil I had in mind, where (1) the aggregate of casualties, regardless of side (2) a function of the kill ratio, given the consideration that when the kill ratio diverges, the situation morphs from one of war (with an ingredient of legitimate self-defense) to one of genocide (with no ingredient of legitimate self-defense).
"As this trend looks like a random walk, we see no obvious limit to how far values can drift."
But a random walk in a space of what dimensionality?Drifts towards the left or the right eventually reverse themselves -but in a high dimensional space, one never stumbles home again...
Yes, sorry, to be clear this was meant as a response to Paul's comments below, his claim c) here in particular. Though it applies to similar arguments others have made before, e.g. Yudkowsky's CEV idea.
The AI transition will be a time of great upheaval in any case.
Times of great upheaval test value systems, even ones tethered to an underlying stable evolved nature.
In the AI transition, that tether will be much weaker; unlike human values, AI values can change with the next SW update or parameter revision. And it is not clear what would compensate for the weakening of the tether.
So the likelihood of AI values changing (thus the risk to our most critical values) is higher in the AI transition than in previous human experience.
Yes I am questioning the relative degree of value plasticity of humans vs AI. I didn't say value drift isn't an important issue.
Robin, are you questioning that an entity's critical values are tied to its nature, or that human nature is less malleable than AIs' natures will be?
As to the overall question of the risk of the AI transition to our critical values, the burden of the proof should be on those claiming that the risk is manageable thus not deserving of extraordinary focus.
I think you are disagreeing with someone else, not me.
It seems to me that the claim that humans would converge on a set of true values given enough information and time to reflect on it makes (at least in its explicit argument) the unjustified assumption that values depend only on information. That is, the argument is that people hear more and better arguments that make them change their minds, and eventually after hearing all the arguments they converge on the truth - in this case the things they truly value, in their heart of hearts.
The problem is that values depend on all sorts of factors, not just info. So while the above argument is just about plausible as a defense of values chosen in light of more info, I don't see how you can claim any other relevant factor as producing truer values. Are true human values produced by rich humans, or poor? High status humans, or low status? Angry humans, sad humans, calm humans? Working or relaxing humans? And, well, why?
Again with the bald claims, lacking supporting evidence.
"As most human value drift is mediated by new humans, I don’t see why new humans with new values are less of a problem than new AIs with new values"
The nature of new humans is much more like previous humans, *and* much less malleable, than that of AIs. Critical values of the kind listed by Cambias are thus at much higher risk of going by the wayside in an AI transition, than in business-as-usual (or even previous-transition) handoffs among human generations.
Funny, this topic was addressed in a book by a guy named Robin Hanson, who felt pretty comfortable with using social science to predict that technological realities will cause values of ems to drift toward those of human farmers. I don't think that he saw the future as some hazy question mark. If anything, he was downright preachy about how we can and should use the insights from the social sciences to extrapolate credible predictions about how we will live, love and value in the future. Anyway, that guy was right. Some material realities of the future will surely surprise us, but our moral psychology is hard-wired enough to produce values that responsible academics can responsibly speculate about.
Edit: Maybe the best way to interpret the parent comment is that we underestimate the weirdness of future material realities, and if knew what we're really in for, we would worry more about the value drift this weirdness would cause. OK, maybe. I do think that some of us overextrapolate the recent trend in which the growth of technology correlates with a growth of our options. Age of Em is a great case study of how this correlation might reverse. But as long as our option-space keeps growing, it's not easy to picture why we would choose a world which is a worse fit for our innate value preferences. It's a good sci-fi premise.