Eliezer’s Meta-Level Determinism

Thank you esteemed co-blogger Eliezer, for your down payment on future engagement of our clash of intuitions.  I too am about to travel and must return to other distractions which I have neglected. 

Some preliminary comments.  First, to be clear, my estimate of future growth rates based on past trends is intended to be unconditional – I do not claim future rates are independent of which is the next big meta innovation, though I am rather uncertain about which next innovations would have which rates.   

Second, my claim to estimate the impact of the next big innovation and Eliezer’s claim to estimate a much larger impact from "full AGI" are not yet obviously in conflict – to my knowledge, neither Eliezer nor I claims full AGI will be the next big innovation, nor does Eliezer argue for a full AGI time estimate that conflicts with my estimating timing of the next big innovation. 

Third, it seems the basis for Eliezer’s claim that my analysis is untrustworthy "surface analogies" vs. his reliable "deep causes" is that while I use long-vetted general social science understandings of factors influencing innovation, he uses his own new untested meta-level determinism theory.  So it seems he could accept that those not yet willing to accept his new theory might instead reasonably rely on my analysis. 

Fourth, while Eliezer outlines his new theory and its implications for overall growth rates, he has as yet said nothing about what his theory implies for transition inequality, and how those implications might differ from my estimates. 

OK, now for the meat.  My story of everything was told (at least for recent eras) in terms of realized capability, i.e., population and resource use, and was largely agnostic about the specific innovations underlying the key changes.  Eliezer’s story is that key changes are largely driven by structural changes in optimization processes and their protected meta-levels:


The history of Earth up until now has been a history of optimizers … generating a constant optimization pressure.  And creating optimized products, not at a constant rate, but at an accelerating rate, because of how object-level innovations open up the pathway to other object-level innovations. … Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. …

Natural selection selects on genes, but generally speaking, the genes do not turn around and optimize natural selection.  The invention of sexual recombination is an exception to this rule, and so is the invention of cells and DNA. … this tiny handful of meta-level improvements feeding back in from the replicators … structure the evolutionary epochs of life on Earth. …

Very recently, certain animal brains have begun to exhibit both generality of optimization power … and cumulative optimization power (…. as a result of skills passed on through language and writing). … We have meta-level inventions like science, that try to instruct humans in how to think.  …  Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history; but they do not rival the brain itself in complexity, and their effect upon the brain is comparatively shallow. …

Now… some of us want to intelligently design an intelligence that would be capable of intelligently redesigning itself, right down to the level of machine code. … That … breaks the idiom of a protected meta-level. .. Then even if the graph of "optimization power in" and "optimized product out" looks essentially the same, the graph of optimization over time is going to look completely different from Earth’s history so far.

OK, so Eliezer’s "meta is max" view seems to be a meta-level determinism view, i.e., that capability growth rates are largely determined, in order of decreasing importance, by innovations at three distinct levels:

  1. The dominant optimization process, natural selection, flesh brains with culture, or full AGI
  2. Improvements behind the protected meta-level of such a process, i.e., cells, sex, writing, science
  3. Key "object-level" innovations that open the path for other such innovations

Eliezer offers no theoretical argument for us to evaluate supporting this ranking.  But his view does seem to make testable predictions about history.  It suggests the introduction of natural selection and of human culture coincided with the very largest capability growth rate increases.  It suggests that the next largest increases were much smaller and coincided in biology with the introduction of cells and sex, and in humans with the introduction of writing and science.  And it suggests other rate increases were substantially smaller.

The main dramatic events in the traditional fossil record are, according to one source:  Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively.  Perhaps two of these five events are at Eliezer’s level two, and none at level one.  Relative to these events, the first introduction of human culture isn’t remotely as noticeable.  While the poor fossil record means we shouldn’t expect a strong correspondence between the biggest innovations and dramatic fossil events, we can at least say this data doesn’t strongly support Eliezer’s ranking.   

Our more recent data is better, allowing clearer tests.  The last three strong transitions were humans, farming, and industry, and in terms of growth rate changes these seem to be of similar magnitude.  Eliezer seems to predict we will discover the first of these was much stronger than the other two.  And while the key causes of these transitions have long been hotly disputed, with many theories in play, Eliezer seems to pick specific winners for these disputes: intergenerational culture, writing, and scientific thinking. 

I don’t know enough about the first humans to comment, but I know enough about farming and industry to say Eliezer seems wrong there.  Yes the introduction of writing did roughly correspond in time with farming, but it just doesn’t seem plausible that writing caused farming, rather than vice versa.  Few could write and what they wrote didn’t help farming much.  Farming seems more plausibly to have resulted from a scale effect in the accumulation of innovations in abilities to manage plants and animals – we finally knew enough to be able to live off the plants near one place, instead of having to constantly wander to new places. 

Also for industry, the key innovation does not seem to have been a scientific way of thinking – that popped up periodically in many times and places, and by itself wasn’t particular useful.  My guess is that the key was the formation of networks of science-like specialists, which wasn’t possible until the previous economy had reached a critical scale and density. 

No doubt innovations can be classified according to Eliezer’s scheme, and yes all else equal relatively-meta innovations are probably stronger, but if as the data above suggests this correlation is much weaker than Eliezer expects, that has important implications for how "full AGI" would play out.  Merely having the full ability to change its own meta-level need not give such systems anything like the wisdom to usefully make such changes, and so an innovation producing that mere ability might not be among the most dramatic transitions. 

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • burger flipper, CA

    I’m curious to see how the advent of bloggy binary fission affects personal Internet usage over time.

  • Larry D’Anna

    I think Eliezer would classify both farming and industry as level 3, while writing and science would be at level 2. I don’t think his theory demands that a level 2 innovation should immediately produce a surge of innovation. Instead it speeds up the pace of innovation (maybe only by a little bit) long into the future. So I don’t think his theory needs to prove that writing caused farming. The key difference between level 2 and level 3 is that level 3 innovations can be very dramatic (like farming) but they do not increase the rate of new innovations. Level 2 innovations can look rather useless at first (like writing probably did), but will be recognized by history as being very important because they enabled many other innovations.

  • http://hanson.gmu.edu Robin Hanson

    Larry, if we allow an arbitrary relation between the timings of the innovation and their resulting speedups, we lose all ability to test innovations theories via observed speedups.

  • Tim Tyler

    Re: “Relative to these [dramatic events in the fossil record], the first introduction of human culture isn’t remotely as noticeable.”

    IMHO, the introduction of human culture is precisely what led to modern humans – and is what triggered the current mass extinction.

    The current mass extinction is not yet enormous – but this is still the very beginning of it.

  • Cyan

    Farming seems more plausibly to have resulted from a scale effect in the accumulation of innovations in abilities to manage plants and animals – we finally knew enough to be able to live off the plants near one place, instead of having to constantly wander to new places.

    Thanks to Jared Diamond, I was under the impression that farming was a forced change of lifestyle — people tend to prefer wandering to new places, but once population densities are high enough, this is no longer possible. I recall the evidence he cited was slow adoption of agriculture that tracked population density (i.e., people didn’t do it until they had to), and nutritional deficits and increased disease among settled communities (inferred from study of skeletal remains).

  • http://hanson.gmu.edu Robin Hanson

    Cyan, I don’t disagree with anything you reported.

    Tim, consider that if innovation-impact delays are that long, then the full AGI impact would be long after its arrival, giving plenty of time to deal with if after it arrived.

  • http://profile.typekey.com/halfinney/ Hal Finney

    It is striking how slow early human cultural innovation was. The Acheulian handaxe (many photos) was used for over a million years, spanning Homo Erectus and Homo Sapiens, up to about 100,000 years ago. Throughout this time, man was using fire and making various tools. But handaxes changed very little during this period. Oddly, we don’t have a strong understanding of what they were used for, despite their astonishing cultural longevity. Many specimens would seem to be hard to grip and employ effectively.

    I find it amazing to imagine a million years of such stagnation, with even savage and uneducated humans who nevertheless possessed roughly the same intelligence as modern man. Some researchers suggest that language did not arise until as recently as 50,000 years ago, which would suggest that that was a much more important innovation than mere human intelligence (or else, whatever caused language to arise was the more important factor).

  • Tim Tyler

    The “full impact” of synthetic minds may indeed occur long after their origin.

    It doesn’t necessarily follow that the immediate impact is going to be any smaller, though – since “full impact” may refer to something utterly enormous.

    It would be necessary to specify some “impact scale” (which we could see had
    a definite upper limit) to make this discussion more concrete – but such fixed upper limits are not trivial to find.

  • Unknown

    Robin points to a serious gap in Eliezer’s predictions when he says that “Merely having the full ability to change its own meta-level need not give such systems anything like the wisdom to usefully make such changes.” Eliezer’s idea is that we can’t program AGI until we understand intelligence, and when we understand it, we can teach this understanding to the AGI, which will then be able to modify itself continuously for increased intelligence.

    But it is quite possible that people will develop AGI without understanding intelligence. I realize that Eliezer says that this is impossible, but since evolution did it, people might do it too, and a good deal faster. And if this happens, there is no particular reason the AGI would know how to make itself smarter.

  • http://www.iphonefreak.com frelkins

    If this is helpful, may I say that the earliest writing has been dated to approx. 3300 BC. (http://news.bbc.co.uk/2/hi/science/nature/235724.stm). As would be expected, these writings appear to be tax records. Agriculture is usually dated to 9 or 10,000 BC.

    So writing comes much later, it seems, and is dictated by a very practical need – for the king to keep track of oil and other agricultural items paid to him in taxes.

    Farming causes permanent settlement, settlement causes government, and government causes writing. Just to clarify. I think the order is important, I don’t mean to be stupid.

  • Tim Tyler

    For evidence of older writing, see: http://en.wikipedia.org/wiki/Tărtăria_tablets.

    “These three small, inscribed tablets started a debate that is challenging the conventional wisdom of European prehistory, because they have been dated from around 6.500 years ago. Some scholars argue they date even earlier at 7,300 years old. More prudent researchers, date the stones to 6,000-5,800 years ago.”

    http://www.prehistory.it/ftp/arta_populara01.htm

  • http://www.iphonefreak.com frelkins

    @Tim Tyler

    Fascinating link. Thanks. Both your link & mine support Robin’s contention that farming causes writing and work to disprove E.

  • http://www.acceleratingfuture.com/tom Tom McCabe

    “I don’t know enough about the first humans to comment, but I know enough about farming and industry to say Eliezer seems wrong there. Yes the introduction of writing did roughly correspond in time with farming, but it just doesn’t seem plausible that writing caused farming, rather than vice versa. Few could write and what they wrote didn’t help farming much. Farming seems more plausibly to have resulted from a scale effect in the accumulation of innovations in abilities to manage plants and animals – we finally knew enough to be able to live off the plants near one place, instead of having to constantly wander to new places.”

    Farming, by itself, was obviously a major change in the basic structure of human life. It wasn’t, however, that *dramatic* of an event- adding farming to a tribe of two hundred hunter-gatherers simply gets you a village two hundred subsistence farmers. In Africa, this is still the predominant form of civilization, and we can see that it doesn’t accomplish much. Farming was already well-developed in various places by 4,000 BC, and the period from 10,000 BC – 4,000 BC is usually glossed over in general-audience textbooks, as there isn’t much to write about.

    The key advance, I think, was *civilization*- the ability of large groups of people to come together and work together, allowing for greater specialization and a larger body of general knowledge. Civilization, on any significant scale, is impossible without writing, as there’s no way to do large-scale organization. Managing one city of fifty thousand people is difficult enough when you can’t record anything. How would you manage a dozen cities? The only way to communicate would be by word-of-mouth, which is notoriously unreliable; to talk to someone, you would have to go through three or four different people, each of whom probably has their own agendas and political goals.

  • http://www.iphonefreak.com frelkins

    @Tom McCabe

    “Civilization, on any significant scale, is impossible without writing”

    Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not “textual.” This is my third post, so I’m outta here. So far I’m believing the evidence we have supports Robin over E. I’ll put a 90% on that belief, that Robin is correct.

  • Tim Tyler

    Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references – or else it should be rapidly abandoned.

  • http://profile.typekey.com/GavinBrown/ GavinBrown

    The methods and tools to farm successfully were built up over a long period through oral history and instruction. Whether it was the generation of intergenerational oral history or writing that caused farming, however, is fairly immaterial to the basic conflict.

    Eliezer is making a prediction based on a rough estimate of a number of factors. In many cases he’s assuming best-case scenarios, at least in my view.

    Here’s a non-exhaustive list of factors that must be weighed in making this prediction:
    1.How difficult each increase in intelligence is.
    2.How effective an AI can be at implementing new advances–and what sort of opposition it runs into.
    3.How well the initial AGI is programmed. It’s entirely possible that we will create something with it’s own set of biases which it will have to overcome.

    The truth is we don’t really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments.

    Robin is making the argument of experience. In the past, we’ve had a certain curve. He’s estimating that the unknown factors will roughly add up to a continuation of the trends that we’ve seen so far. The voice of experience often wins these arguments, because sunny predictions tend to run into unpredicted difficulties. The future tends to be . . . messy. Progress being slow is usually the safe bet.

    The best argument that I can come up with for Eliezer’s point of view is that all previous progress has been multipliers of human effort. With AGI we will be creating a new species, which may have its own rules. We don’t really know what we’ll be creating, so it’s hard to say what rules will apply.

  • http://entitledtoanopinion.wordpress.com TGGP

    I thought increased population density was the result of agriculture, not it’s cause. The extra people did allow agricultural societies to defeat hunter gatherers in battle though.

  • http://profile.typekey.com/GavinBrown/ GavinBrown

    TGGP-
    I’ve assumed that agriculture enabled population density, but it seems to be in dispute. Here’s a course description at UW that summarizes the various theories: http://courses.washington.edu/anth457/agorigin.htm.

    The exact details of the development of farming are, of course, not going to change the basic issue of weighing inside versus outside predictions.

  • http://www.acceleratingfuture.com/tom Tom McCabe

    “Tell that to the Inca; they might have cause to dispute you. Their system of knotted lariats known as khipu appear mostly numerical, not “textual.””

    I’m not an Inca historian, but the important thing seems to have been long-term records, not text as we currently know it. The phonetic alphabet was first adopted by the Phoenicians around 1000 BC, long after the birth of human civilization. In addition, there *are* many historians who believe that the Incas encoded language as well as numbers; see http://www.rso.cornell.edu/scitech/archive/97spr/inca.html for more on this.

    “Somehow the notion appears to have arizen that Eliezer thinks that writing led to farming(!) AFAICS, this idea needs to be supported with references – or else it should be rapidly abandoned.”

    For the record, he makes no such claim in his last post at http://www.overcomingbias.com/2008/06/optimization-an.html, or anywhere else that I can remember.

    “The truth is we don’t really have a clue as to a lot of these factors. For all we know, the first AGIs will have crippling elements imposed on them by frightened programmers or governments.”

    See http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/.

  • http://hanson.gmu.edu Robin Hanson

    Tom and Tim, need I quote Eliezer yet again in the comments?

    Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there. … This tiny handful of meta-level improvements … structure the evolutionary epochs of life on Earth. … Our significant innovations in the art of thinking, like writing and science, are so powerful that they structure the course of human history.

    By all accounts farming was one of the epoch change markers, where “everything proceeds faster from there.” Eliezer says such changes are due to things like sex or science or writing. Out of this list writing is the event that coincided roughly in time with farming. The other big transition was to industry, and science is the listed event that coincided roughly in time with that transition.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    I feel that I am being perhaps a bit over-interpreted here.

    For one thing, the thought of “farming” didn’t cross my mind when I was thinking of major innovations, which tells you something about the optimization viewpoint versus the economic viewpoint.

    But if I were to try to interpret how farming looks from my viewpoint, it would go like this:

    1) Evolution gives humans language, general causal modeling, and long-range planning.

    2) Humans figure out that sowing seeds causes plants to grow, realize that this could be helpful six months later, and tell their friends and children. No direct significance to optimization.

    3) Some areas go from well-nourished hunter-gatherers to a hundred times as many nutritively deprived farmers. Significance to optimization: there are many more humans around, optimizing… maybe slightly worse than they did before, due to poor nutrition. However, you can, in some cases, pour more resources in and get more optimization out, so the object-level trick of farming may have hit back to the meta-level in that sense.

    4) Farming skills get good enough that people have excess crops, which are stolen by tax collectors, resulting in the creation of governments, cities, and above all, professional specialization.

    5) People in cities invent writing.

    So that’s how I would see the object/meta interplay.

  • http://hanson.gmu.edu Robin Hanson

    Eliezer, so even though you said

    Occasionally, a few tiny little changes manage to hit back to the meta level, like sex or science, and then the history of optimization enters a new epoch and everything proceeds faster from there.

    you did not intend at all to say that when we look at the actual times when “everything sped up” we would tend to find such events to have been fundamentally caused by such meta-level changes? Even though you say these “meta-level improvements … structure the evolutionary epochs of life on Earth” you did not mean the epochs as observed historically or as defined by when “everything proceeds faster from there”? If there is no relation in the past between speedup causes and these key meta-level changes, why worry that a future meta-level change will cause a speedup then?

  • http://transhumangoodness.blogspot.com Roko

    Robin: “The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively.”

    – surely you’ve missed off the most important event in the fossil record: namely the beginning of life on earth. This interpretation would indicate that the data supports what eliezer is saying.

  • http://hanson.gmu.edu Robin Hanson

    Roko, the first clear fossils in the traditional fossil record are of cells – whatever came before cells didn’t leave clear fossil records.

  • athmwiji

    “The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively. Perhaps two of these five events are at Eliezer’s level two, and none at level one. Relative to these events, the first introduction of human culture isn’t remotely as noticeable. ”

    This seems like a very odd statement to me. As far as I know there isn’t any evidence of pottery or large hadron colliders in the fossil record that predates the first introduction of human culture. Surely these are dramatic events.

  • http://pancrit.org Chris Hibbert

    I have to agree with athmwiji that Robin is seriously discounting the presence of humans in the fossil record. While human artifacts aren’t spread broadly through the fossil record, their presence is extremely noticeable. There’s contention at this point, but the disappearance of the North American megafauna roughly coincides with the appearance of humans there. You can’t directly detect the presence of humans throughout the region, but where you do find evidence, it’s roughly contemporaneous.