Debating Yudkowsky

On Wednesday I debated my ex-co-blogger Eliezer Yudkowsky at a private Jane Street Capital event (crude audio here, from 4:45; better video here, transcript here).

I “won” in the sense of gaining more audience votes — the vote was 45-40 (him to me) before, and 32-33 after the debate. That makes me two for two, after my similar “win” over Bryan Caplan (42-10 before, 25-20 after). This probably says little about me, however, since contrarians usually “win” such debates.

Our topic was: Compared to the farming and industrial revolutions, intelligence explosion first-movers will quickly control a much larger fraction of their new world. He was pro, I was con. We also debated this subject here on Overcoming Bias from June to December 2008. Let me now try to summarize my current position.

The key issue is: how chunky and powerful are as-yet-undiscovered insights into the architecture of “thinking” in general (vs. on particular topics)? Assume there are many such insights, each requiring that brains be restructured to take advantage. (Ordinary humans couldn’t use them.) Also assume that the field of AI research reaches a key pivotal level of development. And at that point, imagine some AI research team discovers a powerful insight, and builds an AI with an architecture embodying it. Such an AI might then search for more such insights more efficiently than all other the AI research teams who share their results put together.

This new fast AI might then use its advantage to find another powerful insight, restructure itself to take advantage of it, and so on until it was fantastically good at thinking in general. (Or if the first insight were super-powerful, it might jump to this level in one step.) How good? So good that it could greatly out-compete the entire rest of the world at the key task of learning the vast ocean of specific knowledge and insights useful for functioning in the world. So good that even though it started out knowing almost nothing, after a few weeks it knows more than the entire rest of the world put together.

(Note that the advantages of silicon and self-modifiable code over biological brains do not count as relevant chunky architectural insights — they are available to all competing AI teams.)

In the debate, Eliezer gave six reasons to think very powerful brain architectural insights remain undiscovered:

  1. Human mind abilities have a strong common IQ factor.
  2. Humans show many specific mental failings in reasoning.
  3. Humans have completely dominated their chimp siblings.
  4. Chimps can’t function as “scientists” in human society.
  5. “Science” was invented, allowing progress in diverse fields.
  6. AGI researchers focus on architectures, share little content.

My responses:

  1. Human mental abilities correlate across diverse tasks, but this can result from assortative mating, from task ability complementarities, or from an overall brain chemistry resource parameter. There is little reason to believe high IQ folks have a brain architecture feature that low IQ folks lack.
  2. Mind design must trade reliability and accuracy for speed and cost. It is not clear that humans suffer greatly in typical real choices from their many biases. Yes future brains with lower compute costs will have higher reliability. But this is hardly a new architectural insight.
  3. The key human advantage was accumulatings insights via culture. Yes chimps have “culture,” but not enough. Humans had more precise and portable culture via language, and more use for it due to free hands and wider ranges. Culture has a threshold effect of giving only minor benefits until it has “enough” support. And in contrast to the farming and industry revolutions, where second movers still made big gains, chimps couldn’t copy or complement humans enough to gain from humans getting culture first. No big architectural advantages are needed to explain human domination.
  4. Low IQ humans also can’t function at top levels of human society, and we have no reason to believe they lack some special architecture that the high IQ have. Chimp inability to function at our society’s low levels, where their intelligence seems plenty sufficient, is explained by only a tiny fraction of animal species ever being domesticated. Most animals refuse to take our orders, even when they are plenty smart enough to understand them.
  5. The intellectual community called “science” required a sufficient scale of people, communication, and activity to be feasible. Similar behavior was probably tried many times before, but at insufficient scale. “Science” required no brain architecture changes.
  6. The vast majority of AI researchers focus on collecting and implementing small insights. The fact that a small community of AGI (Artificial General Intelligence) researchers focus on architecture hardly says architecture gives huge gains. And academia discourages the large team projects needed to integrate a lot of content – it is hard to publish on small local changes to large projects.

My five reasons to think powerful architectural insights are quite rare:

  1. The literature on economic, technical, and other innovation says most value comes from many small innovations – more useful and wider scope innovations are rarer, and usually require many small supporting innovations. “Intelligence” covers an extremely wide scope, basically all mental tasks. In general, innovations come from diverse users and builders, so the more users the better.
  2. Whatever appeared first in humans gave them no immediate gains in their ability to support a larger population, but only increased the growth rate of that ability. The same held in the farming and industry revolutions, the two other most disruptive events by far in human history. The key to all these changes seems to be better ways to spread innovations further faster. Thus any brain architectural gains must have focused mainly on spreading innovations.
  3. The usual lore among older artificial intelligence researchers is that new proposed architectural concepts are almost always some sort of rearranging of older architectural concepts. They see little new under the AI sun.
  4. The AI system Eliezer most respects for its promising architecture is Eurisko.  Its author, Doug Lenat, concluded from it that our main obstacle is not architecture but mental content – the more one knows, the faster one can learn. Lenat’s new CYC system has much content, though it still doesn’t learn fast. CYC might not have enough content yet, or perhaps Lenat sought the wrong content or format.
  5. Most AI successes come when hardware costs fall enough to implement old methods more vigorously. Most recent big AI successes are due to better ability to integrate a diversity of small contributions. See how Watson won, or Peter Norvig on mass data beating elegant theories. New architecture deserves only small credit for recent success.

Future superintelligences will exist, but their vast and broad mental capacities will come mainly from vast mental content and computational resources. By comparison, their general architectural innovations will be minor additions. It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

GD Star Rating
a WordPress rating system
Tagged as: , ,
Trackback URL:
  • Kip Werking

    I don’t understand.

    You write:

    P1. Future superintelligences will exist, but their vast and broad mental capacities will come mainly from vast mental content and computational resources.
    P2. By comparison, their general architectural innovations will be minor additions.
    C. It thus seems quite unlikely that one AI team could find an architectural innovation powerful enough to let it go from tiny to taking over the world within a few weeks.

    I tend to agree with P1 and P2. But I don’t think that C follows from P1 and P2. It seems to me that P1 and P2 could be true, and that is could also, still, be true that AI could go from very tiny to taking over the world very fast.

    Suppose, for example, that there is a relatively simple algorithm for intelligence. Suppose that it is something like Bayes theorem, applied to goals, and interfacing with an input system that provides information about the world.

    Suppose that this algorithm, although seeming simple on the surface, is relatively difficult to implement. it is difficult enough that nobody has made AI yet, but simple enough that Darwinian evolution stumbled upon it.

    In that case, an intelligence explosion wouldn’t depend on finding lots of complex and groundbreaking insights into intelligence algorithms. Rather, an intelligence explosion would require a team to be the first to properly integrate an input system with a goal system, with enough hardware power to reach escape velocity for the explosion.

    Any of these could be the holdback:

    1. we don’t have the algorithm yet, however, simple it is, or, if we have it, we haven’t refined and implemented it properly
    2. we don’t have the hardware power yet to use the algorithm in a manner competitive with the human brain
    3. we haven’t integrated the algorithm properly with an input and world-modeling system, so that the AI could properly apply the algorithm to the real world

    Those three are just off the top of my head. Any one of those three could be the dam holding back an intelligence explosion. The breaking of any one of those dams could cause an AI explosion in a matter of weeks.

    So, Robin, your logic here completely escapes me…

  • Tyrrell McAllister

    Wouldn’t your first and third scenarios qualify as “general architectural innovations”? The second doesn’t seem like it would abruptly become available to one team before all the rest got it.

    • Tyrrell McAllister

      Sorry, that was in reply to Kip Werking.

  • Marcus

    This sort of sponsorship raises some questions in my mind about your independence and what other ties you have to big finance.

  • http://entitledtoanopinion.wordpress.com TGGP

    Your remark about building a city with better architecture out in the desert sounds like a bit of a knock against Paul Romer’s charter cities. But I think he’s trying to compete with third-world cities rather than New York.

  • Vladimir Nesov
  • burger flipper

    so did 20 people walk out or just abstain from voting at the end?

  • Patri Friedman

    The key issue is: how chunky and powerful are as-yet-undiscovered insights into the architecture of “thinking” in general (vs. on particular topics)?

    I don’t understand why they need to be chunky (each is big). Isn’t enough for there to be significantly more powerful insights available to more intelligent researchers?

    We might imagine that there are “mines” of insights at various levels of researcher intelligence, that we’ve found all the best insights available at human-level research already, but that a smarter-than human AI would have access to new mines such that the highest quality mines quickly give a set of new insights which are powerful enough to open access to new mines…

    Note that the advantages of silicon and self-modifiable code over biological brains do not count as relevant chunky architectural insights — they are available to all competing AI teams.

    But advantages of silicon & self-modifable code related to differences in researcher intelligence are *not* available to all competing teams – only to the first team which has found smarter-than-human AI to assist. In the above example, if the “mines of insight” are insight into developing faster computer hardware or better thinking procedures (or anything else that results in smarter AI), that still gives the first movers an advantage.

    I also find it somewhat odd that you focus on rates of growth in these past population changes but seem to neglect them for intelligence, when the arguments for intelligence explosion are also based on growth. Suppose that AI smarts are simply proportional to computing power, but that the growth rates of computing power depend on researcher intelligence. If insights on improving computing power are independent & stackable, then while world computing power grows at 59% annualized (2x every 18 mo), perhaps the first mover’s will grow at 69% annualized (an extra 10% from the help of their AI). Different fixed exponents results in wildly diverging long-term performance, and if the relative difference widens over time (because it depends on absolute performance), then it diverges even faster.

    This doesn’t get us an explosion in weeks, but isn’t anything where output feed back into a higher growth exponent very unstable? Seems to me that it depends mostly on things like whether AI insights can be kept private and whether they “stack” with global growth.

    Another growth potential comes from total financial assets, which seem like they could grow very quickly from having the best AI. Suppose arbitrages are a tournament won by the best participant – the best AI might be able to make money extremely quickly day-trading, using the money to add computational resources which both make it smarter and enable it to remain best at finding new arbitrages. This would lead to an explosion in control of world resources with no insights on thinking, solely from AI intelligence being proportional to computing power.

  • http://hanson.gmu.edu Robin Hanson

    Kip, an implementation trick or an “integration” approach that makes the difference from complete inability and full ability to run “the” key “algorithm for intelligence” counts as a huge architectural innovation. If hardwire cost were the limit then many competing teams could do it as hardware became cheap enough.

    TGGP, I didn’t intend a dig, and didn’t have that in mind, but yes there is an implicit critique.

    Patri, if you assume only one team has access to a much “smarter-than human AI” which can see things that no number of human level minds working together can see, then you’ve already assumed a huge team asymmetry, which is what is at issue. I don’t follow your computing power example – is your super AI building its own entire computer hardware industry?

    • Kip Werking

      Robin writes:

      “an implementation trick or an “integration” approach that makes the difference from complete inability and full ability to run “the” key “algorithm for intelligence” counts as a huge architectural innovation.”

      No it doesn’t. When I think of huge architectural innovations, I think of projects like building the Brooklyn Bridge or the Space Station.

      But amazing things like the Brooklyn Bridge and the Space Station can be frail. They can have key vulnerabilties, and it doesn’t take huge architectural innovations to correct those frailties and vulnerabilities. For example, if the Brooklyn Bridge used the wrong kind of cement, it could have collapsed on the spot. Using the right cement, instead of the wrong one, is a “huge architectural innovation” – it simply putting the last piece of the puzzle together that was already almost completed.

      Similarly, if you take oxygen out of the Space Station, humans can’t breathe there and will die, making it useless for its intended purpose. Add oxygen back, and it now it works perfectly. Oxygen is not a “huge architectural innovation”, but a lack of oxygen ruins the whole system.

      Fast-conquering AI could be just like that. That’s the whole idea, practically, behind the intelligence explosion.

      You also write: “If hardwire cost were the limit then many competing teams could do it as hardware became cheap enough.”

      But this doesn’t contradict my point. Reread the beginning of my comment. I argued that P1 and P2 can still be true, but C doesn’t follow. You quote immediately above doesn’t touch upon my argument. If it is supposed to, it’s not clear how.

      • Kip Werking

        I meant to write:

        Using the right cement, instead of the wrong one, is NOT a “huge architectural innovation”

  • paulfchristiano

    How surprised would you be by a few architectural insights leading to automatic mathematicians which could outcompete humans? How much content do you think humans have acquired to facilitate mathematics, or simple symbolic games? What do you believe about the relationship about such narrow capabilities and “general intelligence”?

    I would not be surprised by a single architectural change leading to huge improvements in this area, and similar considerations are important to my stance on the question you are debating.

  • mjgeddes

    I think it’s pretty clear that there are only two key insights for AGI:

    (1) cross-domain integration is the first key insight, and it’s all done with a small set of pre-defined ontological primatives (27 to be exact), which are used as prototypes or templates for categorization, enabling the generation of novel insights via defining reference classes and forming analogies between different domains (Guy with a clue: John Sowa),

    and…

    (2) An information theoretic definition of ‘beauty’ in terms of minimal complexity, enabling unified representations of our goals in terms of narratives. (Guy with a clue: Juergen Schmidhuber)

    It’s the combination of (1) and (2) that will produce the intelligence explosion, and these two insights could indeed be done by a small team or even a single person…like..er…me? ;)

  • http://cyborg.blogspot.com Mentifex

    The ideal architecture for artificial intelligence is the neural architecture of the human brain — if only somebody could and did figure it out. You were right to be contra the idea that intelligence explosion first-movers will quickly control a much larger fraction of their new world. As such a first-mover, Mentifex here can report that the creators of the AI will have almost no say in how the new AI is put to use.

  • Pingback: News Bits

  • Sarah

    I don’t have a clear idea who “won” this debate (don’t know the relevant field too well) but one strong argument in favor of Eliezer is that the brain just isn’t that complex in terms of bits of information. I’ve seen one estimate that the human brain stores roughly a terabyte. Since we have “small” brains that can do many things, isn’t it likely that many of our cognitive abilities must share the same machinery or proceed along the same algorithms? There has to be some intrinsic versatility, so that our brains can even fit inside our skulls.

  • Evan

    I find some of Robin’s arguments plausible, but my mind keeps drifting to a hypothetical debate between members of a Homo erectus tribe in the past. One erectus argues that it is possible for one tribe of erectus to evolve new cognitive abilities that will allow them to conquer the world, killing all the other erectus tribes in the process. Another, whom we might call Hanson erectus, argues that this is unlikely, that instead erectus around the world will all evolve into Cro Magnons at once, and that one tribe will certainly not be able to take over the world.

    Maybe AI development in the industrial globalized world is qualitatively different from evolution among hunter gatherer tribes, and for that reason Robin’s objections hold. But I’m not at all sure that that’s the case.

  • http://www.youtube.com/user/TheRationalFuture?feature=mhee Adam A. Ford

    Here is the video of the debate up on youtube. I found it took me a fair while to download the whole thing from the link at the top of this article before watching, it may be easier to watch it streaming on youtube.

    • Konkvistador

      @Adam A. Ford: Thanks!

  • Pingback: Overcoming Bias : Shulman On Superorgs

  • Pingback: Overcoming Bias : Hurry Or Delay Ems?

  • John Maxwell IV

    If the world’s best human AI researcher produces insights about AGI at the rate of H insights per week, how many insights per week should we expect from the first computer-based AI researcher that’s superior? Even if intelligence insights aren’t chunky, as Robin suggests, I wouldn’t be surprised to see the first computer-based AI researcher that’s better than the best humanity has to field producing 10H or 100H insights per week. This could be a very significant first mover advantage if we expect an unconstrained intelligent reasoning architecture, like an economy, to improve its capabilities in an exponential fashion.

  • Pingback: Overcoming Bias : When Is “Soon”?

  • Pingback: Alexander Kruel · SIAI/lesswrong Critiques: Index

  • Pingback: Alexander Kruel · What I would like the Singularity Institute to publish

  • Pingback: Overcoming Bias : Foom Debate, Again

  • Pingback: Overcoming Bias : Debate Is Now Book