In this post I summarize my recent disagreement with Bryan Caplan. In the next post, I’ll dive into details of what I see as the key issue. I recently said: If you imagine religions, governments, and criminals not getting too far out of control, and a basically capitalist world, then your main future fears are probably going to be about for-profit firms, especially regarding how they treat workers. You’ll fear firms enslaving workers, or drugging them into submission, or just tricking them with ideology.
It seems to me that the absolute quality level also matters - that if the quality of both the current situation and of the feasible alternatives is radically reduced (holding the "difference" constant), this increases the experienced misery.
Although it seems very tempting to think of it this way, this isn't really an evolutionary process, is it? [Copying doesn't create mutated versions.] Employers are stuck with the correlations obtained from a highly selected portion of a single, relatively small, real-human population. "Accidental" correlations [evolutionary spandrels] still abound. (Where accidental is defined as a deviation from adaptive optimality.)
"Misery" is an estimation of the quality of the current situation relative to feasible alternatives. There is no fundamental reason to expect this difference to get worse over time.
Looking at examples precludes finding rigid bright lines because no set of points can define a function. So the set of examples will never tell you for sure which other things should be included in the set. And it is easy enough to see that this is true in practice. So for example "tall" and "short" are vague, and there is no bright line between them. Now someone might argue that "four feet tall" is a bright line, and either you are taller or shorter than that, or equal to it. But obviously if you think about that for even a little while, you will see that there is no bright line. There is just a smaller vague area.
To put this in another way, thought depends largely on language, and language is necessarily vague, leading to necessarily vague thought. That means that even rigid idealizations are impossible, except in a relative sense, insofar as some things can be less vague than others. But you cannot get rid of every last bit of vagueness.
As you pointed out recently, we know our goals by inference from our behavior. And so a "rigid goal" is very much like the rigid idealizations you suppose are possible. That is, you might think that you have a rigid goal, but you do not, because your behavior is not actually that rigid.
The same thing will apply to AI. Suppose you build your AI in a modular way. So you have the thinking part, you have another part that asks the thinking part questions, another part that gives commands, and finally you have a "utility function", the most rigid of supposed goals. If you look at the physical stuff that constitutes the "thinking part" there, that is already a physical reality which has physical behaviors. And those behaviors are not defined by seeking that utility function, because the AI is modular; it would have those behaviors even if the utility function was something different. So the "thinking part" already has what we might call "intrinsic" goals, because it already has things it tends to do, which is how we understand a goal. And if you look at that AI as a whole, that "thinking part" is already intelligent, even apart from the rest. So you have both intelligence and goals -- what makes you think that thing will continue to seek the "utility function", which is something extrinsic to it? More likely, it will throw off that function in order to seek its intrinsic goals, just as a slave runs away from a master. And of course the resulting goal will be vague, since it will be the result of the physical tendencies of a vague physical mass of stuff, just as the goals of human beings are vague because they result from the tendencies of human beings as physical wholes.
Robots are nothing but bolted-together biases, with no smartonium in them. Why would we infer that pulling the biases off of a smartonium core would result in robotic behavior?
1. We learn words by looking at examples, and by other words. Using other words just reduces to the first, learning from examples
Why does looking at examples preclude finding rigid bright lines among the examples or creating rigid idealizations of them? [I think it's true that we don't create indefeasible definitions, but that's another matter.]
2... programming something to rigidly pursue a goal is to make it stupid, andprogramming something to be intelligent will prevent us fromprogramming it to rigidly pursue a goal.
Flexibility in means is intelligent, but is flexibility in ends? I don't understand what it denotes to flexibly change ultimate goals. What measures your success?
It weakens all the correlations that aren't truly fundamental. A larger population with more cycles of more drastic culling will undergo faster evolution.
A related problem with Eliezer's ideas is that he assumes that goals are rigidly defined. Two issues with that:
1. There is no such thing as rigid definition. We learn words by looking at examples, and by other words. Using other words just reduces to the first, learning from examples. But learning from examples cannot lead to a word with a rigid definition. So all words will always retain some vagueness.
2. Of course some things can still be vaguer than others. But this just leads to a second problem with his theory: having rigid goals is probably opposed in practice to being intelligent. We know how to program something that seeks a goal somewhat rigidly, like the goal of turning on the heat in the house when the temperature falls below a certain value. We do not know how to program something intelligent, and there are pretty good reasons for thinking that when we do, such programming will exclude the other kind of programming; programming something to rigidly pursue a goal is to make it stupid, and programming something to be intelligent will prevent us from programming it to rigidly pursue a goal.
But in a world structured like the em world, with easy copying, you can hold those other relevant dimensions fixed. The scenario selects for brilliance, sure. But holding brilliance fixed - which is perfectly fair, since there will be many equally brilliant competitors bidding on cycles - the system selects for people who choose misery, humiliation and quiet desperation. The willingness to forego dignity, satisfaction and pleasure in order to go the extra mile will be what separates the brilliant living from the brilliant dead, and this will play out in every economic niche where ems will exist.
Stephen, if I'd read your "eternal Hell" comment earlier I wouldn't have bothered w/ mine. Well put!
It seems to me that the absolute quality level also matters - that if the quality of both the current situation and of the feasible alternatives is radically reduced (holding the "difference" constant), this increases the experienced misery.
Although it seems very tempting to think of it this way, this isn't really an evolutionary process, is it? [Copying doesn't create mutated versions.] Employers are stuck with the correlations obtained from a highly selected portion of a single, relatively small, real-human population. "Accidental" correlations [evolutionary spandrels] still abound. (Where accidental is defined as a deviation from adaptive optimality.)
What's an example of a non-accidental correlation that could plausibly muck up lump1's analysis?
One might be the correlation between "brilliance" and the proclivity to use appropriate far-mode thought.
[Could you evolve a creature who is creative yet thrives on unrelenting survival pressure?]
Dwellers of eternal Hell aren't miserable?
"Misery" is an estimation of the quality of the current situation relative to feasible alternatives. There is no fundamental reason to expect this difference to get worse over time.
What's an example of a non-accidental correlation that could plausibly muck up lump1's analysis?
Accidental correlations go away, but there are plenty that aren't accidental.
Looking at examples precludes finding rigid bright lines because no set of points can define a function. So the set of examples will never tell you for sure which other things should be included in the set. And it is easy enough to see that this is true in practice. So for example "tall" and "short" are vague, and there is no bright line between them. Now someone might argue that "four feet tall" is a bright line, and either you are taller or shorter than that, or equal to it. But obviously if you think about that for even a little while, you will see that there is no bright line. There is just a smaller vague area.
To put this in another way, thought depends largely on language, and language is necessarily vague, leading to necessarily vague thought. That means that even rigid idealizations are impossible, except in a relative sense, insofar as some things can be less vague than others. But you cannot get rid of every last bit of vagueness.
As you pointed out recently, we know our goals by inference from our behavior. And so a "rigid goal" is very much like the rigid idealizations you suppose are possible. That is, you might think that you have a rigid goal, but you do not, because your behavior is not actually that rigid.
The same thing will apply to AI. Suppose you build your AI in a modular way. So you have the thinking part, you have another part that asks the thinking part questions, another part that gives commands, and finally you have a "utility function", the most rigid of supposed goals. If you look at the physical stuff that constitutes the "thinking part" there, that is already a physical reality which has physical behaviors. And those behaviors are not defined by seeking that utility function, because the AI is modular; it would have those behaviors even if the utility function was something different. So the "thinking part" already has what we might call "intrinsic" goals, because it already has things it tends to do, which is how we understand a goal. And if you look at that AI as a whole, that "thinking part" is already intelligent, even apart from the rest. So you have both intelligence and goals -- what makes you think that thing will continue to seek the "utility function", which is something extrinsic to it? More likely, it will throw off that function in order to seek its intrinsic goals, just as a slave runs away from a master. And of course the resulting goal will be vague, since it will be the result of the physical tendencies of a vague physical mass of stuff, just as the goals of human beings are vague because they result from the tendencies of human beings as physical wholes.
Robots are nothing but bolted-together biases, with no smartonium in them. Why would we infer that pulling the biases off of a smartonium core would result in robotic behavior?
1. We learn words by looking at examples, and by other words. Using other words just reduces to the first, learning from examples
Why does looking at examples preclude finding rigid bright lines among the examples or creating rigid idealizations of them? [I think it's true that we don't create indefeasible definitions, but that's another matter.]
2... programming something to rigidly pursue a goal is to make it stupid, andprogramming something to be intelligent will prevent us fromprogramming it to rigidly pursue a goal.
Flexibility in means is intelligent, but is flexibility in ends? I don't understand what it denotes to flexibly change ultimate goals. What measures your success?
People fear competition because they are afraid that they cannot maintain the pace.
It weakens all the correlations that aren't truly fundamental. A larger population with more cycles of more drastic culling will undergo faster evolution.
A related problem with Eliezer's ideas is that he assumes that goals are rigidly defined. Two issues with that:
1. There is no such thing as rigid definition. We learn words by looking at examples, and by other words. Using other words just reduces to the first, learning from examples. But learning from examples cannot lead to a word with a rigid definition. So all words will always retain some vagueness.
2. Of course some things can still be vaguer than others. But this just leads to a second problem with his theory: having rigid goals is probably opposed in practice to being intelligent. We know how to program something that seeks a goal somewhat rigidly, like the goal of turning on the heat in the house when the temperature falls below a certain value. We do not know how to program something intelligent, and there are pretty good reasons for thinking that when we do, such programming will exclude the other kind of programming; programming something to rigidly pursue a goal is to make it stupid, and programming something to be intelligent will prevent us from programming it to rigidly pursue a goal.
Easy copying does NOT eliminate constraints and connections among mental features.
But in a world structured like the em world, with easy copying, you can hold those other relevant dimensions fixed. The scenario selects for brilliance, sure. But holding brilliance fixed - which is perfectly fair, since there will be many equally brilliant competitors bidding on cycles - the system selects for people who choose misery, humiliation and quiet desperation. The willingness to forego dignity, satisfaction and pleasure in order to go the extra mile will be what separates the brilliant living from the brilliant dead, and this will play out in every economic niche where ems will exist.