Thank you esteemed co-blogger Eliezer, for your down payment on future engagement of our clash of intuitions. I too am about to travel and must return to other distractions which I have neglected. Some preliminary comments. First, to be clear, my estimate of future growth rates based on past trends is intended to be unconditional – I do not claim future rates are independent of which is the next big meta innovation, though I am rather uncertain about which next innovations would have which rates.
I have to agree with athmwiji that Robin is seriously discounting the presence of humans in the fossil record. While human artifacts aren't spread broadly through the fossil record, their presence is extremely noticeable. There's contention at this point, but the disappearance of the North American megafauna roughly coincides with the appearance of humans there. You can't directly detect the presence of humans throughout the region, but where you do find evidence, it's roughly contemporaneous.
"The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively. Perhaps two of these five events are at Eliezer's level two, and none at level one. Relative to these events, the first introduction of human culture isn't remotely as noticeable. "
This seems like a very odd statement to me. As far as I know there isn't any evidence of pottery or large hadron colliders in the fossil record that predates the first introduction of human culture. Surely these are dramatic events.
Robin: "The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively."
- surely you've missed off the most important event in the fossil record: namely the beginning of life on earth. This interpretation would indicate that the data supports what eliezer is saying.
Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. you did not intend at all to say that when we look at the actual times when "everything sped up" we would tend to find such events to have been fundamentally caused by such meta-level changes? Even though you say these "meta-level improvements ... structure the evolutionary epochs of life on Earth" you did not mean the epochs as observed historically or as defined by when "everything proceeds faster from there"? If there is no relation in the past between speedup causes and these key meta-level changes, why worry that a future meta-level change will cause a speedup then?
I feel that I am being perhaps a bit over-interpreted here.
For one thing, the thought of "farming" didn't cross my mind when I was thinking of major innovations, which tells you something about the optimization viewpoint versus the economic viewpoint.
But if I were to try to interpret how farming looks from my viewpoint, it would go like this:
1) Evolution gives humans language, general causal modeling, and long-range planning.
2) Humans figure out that sowing seeds causes plants to grow, realize that this could be helpful six months later, and tell their friends and children. No direct significance to optimization.
3) Some areas go from well-nourished hunter-gatherers to a hundred times as many nutritively deprived farmers. Significance to optimization: there are many more humans around, optimizing... maybe slightly worse than they did before, due to poor nutrition. However, you can, in some cases, pour more resources in and get more optimization out, so the object-level trick of farming may have hit back to the meta-level in that sense.
4) Farming skills get good enough that people have excess crops, which are stolen by tax collectors, resulting in the creation of governments, cities, and above all, professional specialization.
5) People in cities invent writing.
So that's how I would see the object/meta interplay.
Tom and Tim, need I quote Eliezer yet again in the comments?
Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. ... This tiny handful of meta-level improvements ... structure the evolutionary epochs of life on Earth. ... Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history. By all accounts farming was one of the epoch change markers, where "everything proceeds faster from there." Eliezer says such changes are due to things like sex or science or writing. Out of this list writing is the event that coincided roughly in time with farming. The other big transition was to industry, and science is the listed event that coincided roughly in time with that transition.
"Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not "textual.""
I'm not an Inca historian, but the important thing seems to have been long-term records, not text as we currently know it. The phonetic alphabet was first adopted by the Phoenicians around 1000 BC, long after the birth of human civilization. In addition, there *are* many historians who believe that the Incas encoded language as well as numbers; see http://www.rso.cornell.edu/... for more on this.
"Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references - or else it should be rapidly abandoned."
For the record, he makes no such claim in his last post at http://www.overcomingbias.c..., or anywhere else that I can remember.
"The truth is we don't really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments."
TGGP-I've assumed that agriculture enabled population density, but it seems to be in dispute. Here's a course description at UW that summarizes the various theories: http://courses.washington.e....
The exact details of the development of farming are, of course, not going to change the basic issue of weighing inside versus outside predictions.
I thought increased population density was the result of agriculture, not it's cause. The extra people did allow agricultural societies to defeat hunter gatherers in battle though.
The methods and tools to farm successfully were built up over a long period through oral history and instruction. Whether it was the generation of intergenerational oral history or writing that caused farming, however, is fairly immaterial to the basic conflict.
Eliezer is making a prediction based on a rough estimate of a number of factors. In many cases he's assuming best-case scenarios, at least in my view.
Here's a non-exhaustive list of factors that must be weighed in making this prediction:1.How difficult each increase in intelligence is.2.How effective an AI can be at implementing new advances--and what sort of opposition it runs into.3.How well the initial AGI is programmed. It's entirely possible that we will create something with it's own set of biases which it will have to overcome.
The truth is we don't really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments.
Robin is making the argument of experience. In the past, we've had a certain curve. He's estimating that the unknown factors will roughly add up to a continuation of the trends that we've seen so far. The voice of experience often wins these arguments, because sunny predictions tend to run into unpredicted difficulties. The future tends to be . . . messy. Progress being slow is usually the safe bet.
The best argument that I can come up with for Eliezer's point of view is that all previous progress has been multipliers of human effort. With AGI we will be creating a new species, which may have its own rules. We don't really know what we'll be creating, so it's hard to say what rules will apply.
Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references - or else it should be rapidly abandoned.
"Civilization, on any significant scale, is impossible without writing"
Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not "textual." This is my third post, so I'm outta here. So far I'm believing the evidence we have supports Robin over E. I'll put a 90% on that belief, that Robin is correct.
"I don't know enough about the first humans to comment, but I know enough about farming and industry to say Eliezer seems wrong there. Yes the introduction of writing did roughly correspond in time with farming, but it just doesn't seem plausible that writing caused farming, rather than vice versa. Few could write and what they wrote didn't help farming much. Farming seems more plausibly to have resulted from a scale effect in the accumulation of innovations in abilities to manage plants and animals - we finally knew enough to be able to live off the plants near one place, instead of having to constantly wander to new places."
Farming, by itself, was obviously a major change in the basic structure of human life. It wasn't, however, that *dramatic* of an event- adding farming to a tribe of two hundred hunter-gatherers simply gets you a village two hundred subsistence farmers. In Africa, this is still the predominant form of civilization, and we can see that it doesn't accomplish much. Farming was already well-developed in various places by 4,000 BC, and the period from 10,000 BC - 4,000 BC is usually glossed over in general-audience textbooks, as there isn't much to write about.
The key advance, I think, was *civilization*- the ability of large groups of people to come together and work together, allowing for greater specialization and a larger body of general knowledge. Civilization, on any significant scale, is impossible without writing, as there's no way to do large-scale organization. Managing one city of fifty thousand people is difficult enough when you can't record anything. How would you manage a dozen cities? The only way to communicate would be by word-of-mouth, which is notoriously unreliable; to talk to someone, you would have to go through three or four different people, each of whom probably has their own agendas and political goals.
"These three small, inscribed tablets started a debate that is challenging the conventional wisdom of European prehistory, because they have been dated from around 6.500 years ago. Some scholars argue they date even earlier at 7,300 years old. More prudent researchers, date the stones to 6,000-5,800 years ago."
I have to agree with athmwiji that Robin is seriously discounting the presence of humans in the fossil record. While human artifacts aren't spread broadly through the fossil record, their presence is extremely noticeable. There's contention at this point, but the disappearance of the North American megafauna roughly coincides with the appearance of humans there. You can't directly detect the presence of humans throughout the region, but where you do find evidence, it's roughly contemporaneous.
"The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively. Perhaps two of these five events are at Eliezer's level two, and none at level one. Relative to these events, the first introduction of human culture isn't remotely as noticeable. "
This seems like a very odd statement to me. As far as I know there isn't any evidence of pottery or large hadron colliders in the fossil record that predates the first introduction of human culture. Surely these are dramatic events.
Roko, the first clear fossils in the traditional fossil record are of cells - whatever came before cells didn't leave clear fossil records.
Robin: "The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively."
- surely you've missed off the most important event in the fossil record: namely the beginning of life on earth. This interpretation would indicate that the data supports what eliezer is saying.
Eliezer, so even though you said
Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. you did not intend at all to say that when we look at the actual times when "everything sped up" we would tend to find such events to have been fundamentally caused by such meta-level changes? Even though you say these "meta-level improvements ... structure the evolutionary epochs of life on Earth" you did not mean the epochs as observed historically or as defined by when "everything proceeds faster from there"? If there is no relation in the past between speedup causes and these key meta-level changes, why worry that a future meta-level change will cause a speedup then?
I feel that I am being perhaps a bit over-interpreted here.
For one thing, the thought of "farming" didn't cross my mind when I was thinking of major innovations, which tells you something about the optimization viewpoint versus the economic viewpoint.
But if I were to try to interpret how farming looks from my viewpoint, it would go like this:
1) Evolution gives humans language, general causal modeling, and long-range planning.
2) Humans figure out that sowing seeds causes plants to grow, realize that this could be helpful six months later, and tell their friends and children. No direct significance to optimization.
3) Some areas go from well-nourished hunter-gatherers to a hundred times as many nutritively deprived farmers. Significance to optimization: there are many more humans around, optimizing... maybe slightly worse than they did before, due to poor nutrition. However, you can, in some cases, pour more resources in and get more optimization out, so the object-level trick of farming may have hit back to the meta-level in that sense.
4) Farming skills get good enough that people have excess crops, which are stolen by tax collectors, resulting in the creation of governments, cities, and above all, professional specialization.
5) People in cities invent writing.
So that's how I would see the object/meta interplay.
Tom and Tim, need I quote Eliezer yet again in the comments?
Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. ... This tiny handful of meta-level improvements ... structure the evolutionary epochs of life on Earth. ... Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history. By all accounts farming was one of the epoch change markers, where "everything proceeds faster from there." Eliezer says such changes are due to things like sex or science or writing. Out of this list writing is the event that coincided roughly in time with farming. The other big transition was to industry, and science is the listed event that coincided roughly in time with that transition.
"Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not "textual.""
I'm not an Inca historian, but the important thing seems to have been long-term records, not text as we currently know it. The phonetic alphabet was first adopted by the Phoenicians around 1000 BC, long after the birth of human civilization. In addition, there *are* many historians who believe that the Incas encoded language as well as numbers; see http://www.rso.cornell.edu/... for more on this.
"Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references - or else it should be rapidly abandoned."
For the record, he makes no such claim in his last post at http://www.overcomingbias.c..., or anywhere else that I can remember.
"The truth is we don't really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments."
See http://www.singinst.org/blo....
TGGP-I've assumed that agriculture enabled population density, but it seems to be in dispute. Here's a course description at UW that summarizes the various theories: http://courses.washington.e....
The exact details of the development of farming are, of course, not going to change the basic issue of weighing inside versus outside predictions.
I thought increased population density was the result of agriculture, not it's cause. The extra people did allow agricultural societies to defeat hunter gatherers in battle though.
The methods and tools to farm successfully were built up over a long period through oral history and instruction. Whether it was the generation of intergenerational oral history or writing that caused farming, however, is fairly immaterial to the basic conflict.
Eliezer is making a prediction based on a rough estimate of a number of factors. In many cases he's assuming best-case scenarios, at least in my view.
Here's a non-exhaustive list of factors that must be weighed in making this prediction:1.How difficult each increase in intelligence is.2.How effective an AI can be at implementing new advances--and what sort of opposition it runs into.3.How well the initial AGI is programmed. It's entirely possible that we will create something with it's own set of biases which it will have to overcome.
The truth is we don't really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments.
Robin is making the argument of experience. In the past, we've had a certain curve. He's estimating that the unknown factors will roughly add up to a continuation of the trends that we've seen so far. The voice of experience often wins these arguments, because sunny predictions tend to run into unpredicted difficulties. The future tends to be . . . messy. Progress being slow is usually the safe bet.
The best argument that I can come up with for Eliezer's point of view is that all previous progress has been multipliers of human effort. With AGI we will be creating a new species, which may have its own rules. We don't really know what we'll be creating, so it's hard to say what rules will apply.
Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references - or else it should be rapidly abandoned.
@Tom McCabe
"Civilization, on any significant scale, is impossible without writing"
Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not "textual." This is my third post, so I'm outta here. So far I'm believing the evidence we have supports Robin over E. I'll put a 90% on that belief, that Robin is correct.
"I don't know enough about the first humans to comment, but I know enough about farming and industry to say Eliezer seems wrong there. Yes the introduction of writing did roughly correspond in time with farming, but it just doesn't seem plausible that writing caused farming, rather than vice versa. Few could write and what they wrote didn't help farming much. Farming seems more plausibly to have resulted from a scale effect in the accumulation of innovations in abilities to manage plants and animals - we finally knew enough to be able to live off the plants near one place, instead of having to constantly wander to new places."
Farming, by itself, was obviously a major change in the basic structure of human life. It wasn't, however, that *dramatic* of an event- adding farming to a tribe of two hundred hunter-gatherers simply gets you a village two hundred subsistence farmers. In Africa, this is still the predominant form of civilization, and we can see that it doesn't accomplish much. Farming was already well-developed in various places by 4,000 BC, and the period from 10,000 BC - 4,000 BC is usually glossed over in general-audience textbooks, as there isn't much to write about.
The key advance, I think, was *civilization*- the ability of large groups of people to come together and work together, allowing for greater specialization and a larger body of general knowledge. Civilization, on any significant scale, is impossible without writing, as there's no way to do large-scale organization. Managing one city of fifty thousand people is difficult enough when you can't record anything. How would you manage a dozen cities? The only way to communicate would be by word-of-mouth, which is notoriously unreliable; to talk to someone, you would have to go through three or four different people, each of whom probably has their own agendas and political goals.
@Tim Tyler
Fascinating link. Thanks. Both your link & mine support Robin's contention that farming causes writing and work to disprove E.
For evidence of older writing, see: http://en.wikipedia.org/wiki/Tărtăria_tablets.
"These three small, inscribed tablets started a debate that is challenging the conventional wisdom of European prehistory, because they have been dated from around 6.500 years ago. Some scholars argue they date even earlier at 7,300 years old. More prudent researchers, date the stones to 6,000-5,800 years ago."
- http://www.prehistory.it/ftp/arta_populara01.htm