I Still Don’t Get Foom

Back in 2008 my ex-co-blogger Eliezer Yudkowsky and I discussed his “AI foom” concept, a discussion that we recently spun off into a book. I’ve heard for a while that Nick Bostrom was working on a book elaborating related ideas, and this week his Superintelligence was finally available to me to read, via Kindle. I’ve read it now, along with a few dozen reviews I’ve found online. Alas, only the two reviews on GoodReads even mention the big problem I have with one of his main premises, the same problem I’ve had with Yudkowsky’s views. Bostrom hardly mentions the issue in his 300 pages (he’s focused on control issues).

All of which makes it look like I’m the one with the problem; everyone else gets it. Even so, I’m gonna try to explain my problem again, in the hope that someone can explain where I’m going wrong. Here goes.

“Intelligence” just means an ability to do mental/calculation tasks, averaged over many tasks. I’ve always found it plausible that machines will continue to do more kinds of mental tasks better, and eventually be better at pretty much all of them. But what I’ve found it hard to accept is a “local explosion.” This is where a single machine, built by a single project using only a tiny fraction of world resources, goes in a short time (e.g., weeks) from being so weak that it is usually beat by a single human with the usual tools, to so powerful that it easily takes over the entire world. Yes, smarter machines may greatly increase overall economic growth rates, and yes such growth may be uneven. But this degree of unevenness seems implausibly extreme. Let me explain.

If we count by economic value, humans now do most of the mental tasks worth doing. Evolution has given us a brain chock-full of useful well-honed modules. And the fact that most mental tasks require the use of many modules is enough to explain why some of us are smarter than others. (There’d be a common “g” factor in task performance even with independent module variation.) Our modules aren’t that different from those of other primates, but because ours are different enough to allow lots of cultural transmission of innovation, we’ve out-competed other primates handily.

We’ve had computers for over seventy years, and have slowly build up libraries of software modules for them. Like brains, computers do mental tasks by combining modules. An important mental task is software innovation: improving these modules, adding new ones, and finding new ways to combine them. Ideas for new modules are sometimes inspired by the modules we see in our brains. When an innovation team finds an improvement, they usually sell access to it, which gives them resources for new projects, and lets others take advantage of their innovation.

Since software is often fragile and context dependent, much innovation consists of making new modules that are rather similar to old ones, except that they work in somewhat different contexts. We try to avoid this fragility via abstraction, but this is usually hard. Today humans also produce most of the value in innovations tasks, though software sometimes helps. We even try to innovate new ways to innovate, but that is also very hard.

Overall we so far just aren’t very good at writing software to compete with the rich well-honed modules in human brains. And we are bad at making software to make more software. But computer hardware gets cheaper, software libraries grow, and we learn more tricks for making better software. Over time, software will get better. And in centuries, it may rival human abilities.

In this context, Bostrom imagines that a single “machine intelligence project” builds a “system” or “machine” that follows the following trajectory:


“Human baseline” represents the effective intellectual capabilities of a representative human adult with access to the information sources and technological aids currently available in developed countries. … “The crossover”, a point beyond which the system’s further improvement is mainly driven by the system’s own actions rather than by work performed upon it by others. … Parity with the combined intellectual capability of all of humanity (again anchored to the present) … [is] “civilization baseline”.

These usual “technological aids” include all of the other software available for sale in the world. So apparently the reason the “baseline” and “civilization” marks are flat is that this project is not sharing its innovations with the rest of the world, and available tools aren’t improving much during the period shown. Bostrom distinguishes takeoff durations that are fast (minutes, hours, or days), moderate (months or years), or slow (decades or centuries) and says “a fast or medium takeoff looks more likely.” As it now takes the world economy fifteen years to double, Bostrom sees one project becoming a “singleton” that rules all:

The nature of the intelligence explosion does encourage a winner-take-all dynamic. In this case, if there is no extensive collaboration before the takeoff, a singleton is likely to emerge – a single project would undergo the transition alone, at some point obtaining a decisive strategic advantage.

Bostrom accepts there there might be more than one such project, but suggests that likely only the first one would matter, because the time delays between projects would be like the years and decades we’ve seen between when different nations could build various kinds of nuclear weapons or rockets. Presumably these examples set the rough expectations we should have in mind for the complexity, budget, and secrecy of the machine intelligence projects Bostrom has in mind.

In Bostrom’s graph above the line for an initially small project and system has a much higher slope, which means that it becomes in a short time vastly better at software innovation. Better than the entire rest of the world put together. And my key question is: how could it plausibly do that? Since the rest of the world is already trying the best it can to usefully innovate, and to abstract to promote such innovation, what exactly gives one small project such a huge advantage to let it innovate so much faster?

After all, if a project can’t innovate faster than the world, it can’t grow faster to take over the world. Yes there may be feedback effects, where better software makes it easier to make more software, speeds up hardware gains to encourage better software, etc. But if these feedback effects apply nearly as strongly to software inside and outside the project, it won’t give much advantage to the project relative to the world. Yes by isolating itself the project may prevent others from building on its gains. But this also keeps the project from gaining revenue to help it to grow.

A system that can perform well across a wide range of tasks probably needs thousands of good modules. Same for a system that innovates well across that scope. And so a system that is a much better innovator across such a wide scope needs much better versions of that many modules. But this seems like far more innovation than is possible to produce within projects of the size that made nukes or rockets.

In fact, most software innovation seems to be driven by hardware advances, instead of innovator creativity. Apparently, good ideas are available but must usually wait until hardware is cheap enough to support them.

Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.

Some hope that a small project could be much better at innovation because it specializes in that topic, and much better understands new theoretical insights into the basic nature of innovation or intelligence. But I don’t think those are actually topics where one can usefully specialize much, or where we’ll find much useful new theory. To be much better at learning, the project would instead have to be much better at hundreds of specific kinds of learning. Which is very hard to do in a small project.

What does Bostrom say? Alas, not much. He distinguishes several advantages of digital over human minds, but all software shares those advantages. Bostrom also distinguishes five paths: better software, brain emulation (i.e., ems), biological enhancement of humans, brain-computer interfaces, and better human organizations. He doesn’t think interfaces would work, and sees organizations and better biology as only playing supporting roles.

That leaves software and ems. Between the two Bostrom thinks it “fairly likely” software will be first, and he thinks that even if an em transition doesn’t create a singleton, a later software-based explosion will. I can at least see a plausible sudden gain story for ems, as almost-working ems aren’t very useful. But in this post I’ll focus on software explosions.

Imagine in the year 1000 you didn’t understand “industry,” but knew it was coming, would be powerful, and involved iron and coal. You might then have pictured a blacksmith inventing and then forging himself an industry, and standing in a city square waiving it about, commanding all to bow down before his terrible weapon. Today you can see this is silly — industry sits in thousands of places, must be wielded by thousands of people, and needed thousands of inventions to make it work.

Similarly, while you might imagine someday standing in awe in front of a super intelligence that embodies all the power of a new age, superintelligence just isn’t the sort of thing that one project could invent. As “intelligence” is just the name we give to being better at many mental tasks by using many good mental modules, there’s no one place to improve it. So I can’t see a plausible way one project could increase its intelligence vastly faster than could the rest of the world.

(One might perhaps move a lot of intelligence at once from humans to machines, instead of creating it. But that is the em scenario, which I’ve set aside here.)

So, bottom line, much of Nick Bostrom’s book Superintelligence is based on the premise that a single software project, which starts out with a tiny fraction of world resources, could within a few weeks grow so strong to take over the world. But this seems to require that this project be vastly better than the rest of the world at improving software.  I don’t see how it could plausibly do that. What I am I missing?

Added 2Sep: See also related posts after: Irreducible DetailRegulating Infinity.

Added 5Nov: Let me be clear: Bostrom’s book has much thoughtful analysis of AI foom consequences and policy responses. But aside from mentioning a few factors that might increase or decrease foom chances, Bostrom simply doesn’t given an argument that we should expect foom. Instead, Bostrom just assumes that the reader thinks foom likely enough to be worth his detailed analysis.

GD Star Rating
Tagged as: , , ,
Trackback URL:
  • Tyrrell_McAllister

    This seems to require that this project be vastly better than the rest of the world at improving software.

    Does the project have to be vastly better? If FOOMing only takes a few weeks, then doesn’t this project only have to be a few weeks ahead of the rest?

    • I mean much better in terms of the rate of innovation, holding constant other factors.

  • John Salvatier

    I think my intuition originally came from the idea that the cycle (think about improving intelligence) -> (directly edit own source code) -> (think about improving intelligence) seems dramatically more closed than other potential FOOM cycles. By ‘closed’ I mean, requiring fewer hidden outside inputs. Its closer to a pure feedback cycle. One way to attack the FOOM idea is find and point out important hidden inputs.

    • I don’t see why we should expect the same cycle in the entire world to be any less powerful.

      • John Salvatier

        Are you granting (for the sake of argument) that the cycle might be very fast and powerful, but not sure why it would be limited to one agent?

        Your idea is that that cycle could have lots of agents (the whole world) in it?

        That seems plausible if there’s significant time overlap between different agents improvement and they’re willing to cooperate.

        That seems most plausible to me if improving intelligence turns out to be very difficult marginally. Which it could turn out to be. However, I think there’s plenty of reason to worry that it won’t be and the improvement happens fast.

      • John_Maxwell_IV

        Economy-wide growth seems bound by the laws of physics in a way that software development isn’t. It takes time to transport stuff, energy to manufacture stuff, etc.

    • truth_machine

      The best path to improved intelligence is to evolve it. “thinking” is overrated and a weak substitute. And once you start on the evolution path, thinking loses value because the results are inscrutable to it.

      • Daniel Carrier

        Evolution is the only path known to successfully lead to a human-level intelligence. That doesn’t mean it’s the only one that can.

        We only know thinking isn’t so great at producing a new intelligence insomuch as we have tried and failed. If we ever do succeed, it would be reasonable to conclude that thinking would be useful at upgrading one of these intelligences as well.

      • truth_machine

        Since I didn’t use the word “only”, why did you? That you’re so incapable of understanding a simple English sentence does not bode well for your ability to understand anything else. There are reasons why evolution is the BEST path to improved intelligence but I won’t waste my time explaining them.

      • Daniel Carrier

        How do you know evolution is the best path? It’s the only one that worked so far, but that was with vastly more resources than has been used by any other method.

  • lump1

    Given that you think that among em’s, global economic doubling could happen on the order of days, I’m not sure why you’re more skeptical about foom.

    Maybe it’s because you see em activity as being the sum of many agents, where much of the growth would be the product of population growth, or increased emulation speed. But notice in that case, you’re picturing a world where lots of hardware is being built at breakneck pace, cooling towers and all. Is that any less amazing than foom? Again, it’s somewhat easier to imagine when it’s the aggregate of many new individual computers put up by many individual agents, none of which are subjectively in a terrible hurry.

    To imagine the classic foom, picture these agents being so networked that they can no longer be thought of as individuals. They are parts of a single machine, not individual loci of executive control. Would this necessarily slow “them” down? Might it not speed up their activity, because it could dramatically increase their coordination? With enough direct communication, each node would instantly know of all the dead ends that have pursued by other nodes, it could divide search space so as to avoid wasteful overlap, it could coordinate construction with minimal friction, run simulations on a level of detail that smaller computers couldn’t, etc. A world of a trillion ems would have many hands, so to speak, and if each one is building a cooling tower, the effect will be a trillion cooling towers having sprung up overnight. But a centralized computer of comparable total power could have just as many hands, and probably multitask more efficiently.

    One foom story starts with the assumption that smart computers will network, and once they do, they will coalesce into a superorganism which is no less effective than the individual machines. Given that there will probably be means and motive for this, I wonder what would stop this coalescence.

    Maybe you’re picturing the first smarter-than-human computer to be in foom inflection point. That may not happen, I agree. Progress from there could still be driven largely by teams of humans, for years. But probably not decades; the computers will be duplicated and build their own teams, with human help becoming quickly irrelevant. Isn’t it inevitable that the computers will soon control the entire production chain, from mining to design, manufacturing and software?

    So it comes down to this, maybe: You do believe in foom-like growth, but you picture it as pluralistic – an aggregate of the small actions of many small agents. Classic foomers think that by some special-to-AI mechanism, all the spoils will go to a single victor. I suggested that inevitable networking might be one such mechanism, but I can imagine others. This is good topic for debate. What global mechanisms would resist a winner taking all?

    • sflicht

      I can’t claim to have perfect recall of everything Robin’s written about ems, but I’d ask if he’s ever claimed that high-frequency economic doublings would be a feature of the *inchoate* em economy. My understanding was that the analysis of em economics was in the vein of an equilibrium theory. (For an Industrialization comparison, tractors offered vast gains in agricultural productivity relative to animal-powered ploughs, but as far I understand the history [which is admittedly not very far] they did not do so on a timescale much — if at all — faster than advances in other sectors.) Since the rapid timescales of the foom scenario are to be understood relative to the general pace of technological advance, I don’t think Robin’s espousal of rapid timescales in an em economy undermines his objection to the plausibility of foom-like AI takeoff.

  • Zach Skaggs

    Robin, until superintelligence is invented, I’ll just have to settle for you. Really enjoyed this piece — helped me to understand a field I can’t claim to understand much about.

  • Eliezer Yudkowsky

    Not to reprise our whole debate, but the part of your post that struck me as most flatly wrong, and plausibly derived from AI ideas I regard as sad relics of the 1980s, is the part about “thousands of modules”. Bostrom is just supposing that future AGI software operates much the same way as, say, Google Translate, which consists of one company building an agglomeration of proprietary and public algorithms, feeding it all the data they can get their hands on, and out comes the proprietary Google translation results. Strangely enough, it doesn’t seem to be the case that the leading translator is built out of an agglomeration of companies with a color-translating module, an animal-translating module, a business-translating module, and so on. Google Maps doesn’t agglomerate an economy of modules that know about two-lane highways, parking lots, and bridges. Google’s self-driving cars, so far as I know, are not being produced by outsourcing key software modules to experts all around the world. You can say that maybe they should be doing that, but in real life they’re not. Personally I have no reason to think Google is doing the wrong thing; the fact is that if you have a new algorithm it’s usually cheaper to re-run it over all the data, and the cost of integrating with other algorithms as remote servers usually isn’t worth it to just integrating whatever they do into your own algorithm, if it’s really worth it. The closest thing in modern AI history to what you’re envisioning is the Netflix Prize and all the agglomeration of hundreds of algorithms wasn’t producing big gains like a modern economy, it was there to eke out improvements of a tenth of a percent, and it was incentivized by Netflix using a certain exact scoring rule and threshold which readily lent itself to weighted mixtures of outputs. The components of the huge Netflix Prize mixture still weren’t doing very much interaction internally.

    Bostrom, in other words, is merely supposing by default that future AGI is produced and improved in mostly the same way as modern AI in serious applications. Now you may have a strong intuition that this is a temporary affair and that modern AI is bad precisely on account of its not consisting of these thousands of modules, and you may have an intuition that future AGI to be good must indeed consist of thousands of modules and look more like a global economy. But to throw your hands up in puzzlement and say “Where is Bostrom getting this?” seems disingenuous; it’s very clear where Bostrom is getting it, even if you think the similarity Bostrom is extrapolating-by-default is just bound to be broken because you have very strong arguments that it will be. When Kurzweil extrapolates Moore’s Law out unchanged to 2099, I personally don’t agree that this is reasonable, but I also don’t throw up my hands and say “Where does Kurzweil get these weird ideas about levels of computing power in 2099?” because it’s perfectly clear where Kurzweil is getting these ideas, I am the one who is arguing that the existing trend he is extrapolating will break, and it would be disingenuous of me to pretend otherwise.

    • I fear you are attributing positions to me that I do not hold. I agree that big systems often have a single firm that is the final integrator of many modules obtained from many sources. I agree that the internal organization of such systems quite often does not have different autonomous firms using arms-length market transactions. And I agree those choices are usually reasonable. But I don’t see how granting those points resolves my question – I don’t see how the fact that big systems have final integrators allows a small project to innovate far faster than the entire rest of the world.

      • Eliezer Yudkowsky

        Many companies manage to take one product and innovate it faster than the rest of the world. (1) Self-improving AI is a product and one company can have the best self-improving AI. (2) Then that AI (2a) goes over a qualitative threshold of being able to work on itself, which might be, e.g., the same sort of qualitative threshold that separates humans relative to other primates; and (2b) because it’s also a machine intelligence it can output this work fast relative to how much other companies and their programmers are able to work as humans on their own proprietary AIs; and (2c) this initial advantage snowballs as did, e.g., the human economy as a whole after the 1600s, and it doesn’t spread to other companies because (2d) we are in a modern-style environment of various companies with their proprietary AIs and (2e) given 2d it is not sensible to give away a secret goose that has just started laying many golden eggs some of which are more valuable if nobody else has them, just like proprietary trading firms keep rather than sell their algorithmic innovations because they can earn more from local golden eggs than by selling their secrets. (3) Then the superintelligence innovates faster than the rest of the world.

        I think at this point we’re just recapitulating the whole FOOM debate. But I do think it’s disingenuous to act like nobody has ever explained where the fast innovation is supposed to come from. It’s supposed to arise from a local project in a modern-style environment of other local AI projects, a local advantage that snowballs and does not diffuse within a modern-style environment where it’s perfectly normal for much AI software that is written not to diffuse, a snowball effect, and eventually a superintelligence. As forecasted by people who clearly and explicitly disagree with you about the certainty of the “future AGIs will be made of thousands of modules improved by distant specialists” prediction, which is not exactly a conventional accepted wisdom (I don’t see anything about it in _AI: A Modern Approach_).

      • I don’t see where I said “improved by distant specialists.” I don’t claim that no one has tried to explain this subject to me, I instead claim that explanations just haven’t stuck yet. Which is why I’m trying again to discuss the issue.

        Our economy relies on a division of labor, whereby different firms focus on different tasks. A firm may for example specialize in route-finding in maps better than all other firms, and then be better at innovating at routes. A project that is a small fraction of the world economy can pursue a new approach to routes and may succeed at displacing prior route firms. But that is because routes are a small fraction of the world economy. If the demand for routes doesn’t grow faster than the world economy, the firm that supplies routes also can’t long grow faster.

        The key difference for this imagined project in Bostrom’s book is that it isn’t specializing in a particular small topic – its area of expertise is all topics relevant for the world economy. So if it is to grow fast to become broadly capable, it has to be able to grow its abilities in all of those areas. And grow them faster than the rest of the world. So it has to have new better ideas not just for how to do routes, but also for language translation, bike manufacture, and how to do better at everything else. And that is an enormous amount to ask of a small new project.

        Put another way, to achieve this scenario a project just can’t specialize at being better at “intelligence” or “learning” – those aren’t in fact topics in which one can usefully specialize much. The project has to instead be better at thousands of specific kinds of intelligence or learning. Which conflicts with it starting out as a small project.

      • GavinBrown

        > Put another way, to achieve this scenario a project just can’t specialize at being better at “intelligence” or “learning”

        That’s the core of your disagreement. Eliezer thinks that it’s possible to specialize in intelligence and learning in a general way.

        Furthermore, I think the claim is that a designed intelligence would be much easier to improve and augment. It would be legible to itself, so it would be able to improve itself in ways fundamentally more powerful than humans, whose brains are not understood or documented.

      • I agree that this seems a plausible hypothesis for a core of our disagreement. It would be more plausible if Eliezer thought so as well.

      • Silent Cal

        This does seem like a core of the disagreement; I read Eliezer above as attacking premises whose role in your argument is to produce this intermediate result.

      • Aron

        A neural net that learns from training examples.. is it improving it’s intelligence? Or is that defined by it’s maximum ability after training under ideal circumstances? Is there agreement on that?

        I think part of ‘intelligence’ is discovered by accident. It’s random trial and error with a pruning. The net result is a system that improves on a task but doesn’t necessarily know the reasons why. This is by nature a process with a lot of factors that require time and expense.

      • truth_machine

        Software is not legible to itself unless its purpose is specifically the analysis of software, and even then it only gets to see a static view of itself.

        As a specialist in writing software the technology of writing software, I believe that Yudkowsky (assuming you correctly describe his views) is almost certainly wrong … there are very few generalities across the entire problem/solution spaces, and those that do exist are very weak and do not provide a path toward additional generalities.

      • blogospheroid

        If someone had stumbled upon this secret sauce of specialization at intelligence or learning, then the rational thing for them to do would be to self-accrete wealth and knowledge in all the proprietary domains until their goal is reached. I think that as per Eliezer’s comments, it looks like Google is doing something like that. I think Amazon and IBM are also doing something like that.

      • truth_machine

        No, it doesn’t look like that. (I have close friends who work there and I pay a lot of attention to what Google is doing from a technical software perspective.)

      • Wei Dai

        >The project has to instead be better at thousands of specific kinds of intelligence or learning. Which conflicts with it starting out as a small project.

        Suppose you had access to 10,000 loyal von Neumann clones (and nobody else does). Do you think your project could be better at thousands of specific kinds of
        intelligence or learning? If so, a small AI project could achieve the equivalent of this, by building one AI of von Neuman’s level of intelligence, buying/renting enough hardware to run 10,000 copies of it, and having each copy learn and specialize in a different area. Which part of this scenario seems improbable to you?

      • If my project can rent hardware to run clones, and if other projects can also run hardware to rent clones, then I need some other advantage to be able to grow faster than them.

      • Wei Dai

        I was assuming that no one else had an AI of von Neumann’s level of intelligence. Let’s say the second best AI in the world is only as smart as a typical university professor.

      • The question is how such a situation could plausibly arise. In order to arrive in such a situation, the project would have needed to have a higher growth rate of innovation.

      • Wei Dai

        The biological von Neumann’s brain must have been architecturally very similar to a typical university professor’s. Nor could it have contained independent improvements to lots of different modules. Given this, I speculate that obtaining the analogous improvement in AI intelligence may only require a few design tweaks, which a relatively small project could find first by being luckier or a bit more talented than everyone else.

      • You are suggesting that mental architecture is very important for intelligence. I tried to address and question that in my post.

      • Wei Dai

        In your post you cited evidence from AI projects from 20 years ago. I don’t think that evidence gives us much confidence on the relative importance of architecture vs details for human level AIs and beyond.

        As one example, computational linguistics has evolved from being mostly rule-based (where details are more important) to being more statistical (where architecture is more important). Maybe this trend won’t continue, but maybe it will, and I don’t see how we can be very confident about either possibility.

      • I don’t see that architecture is more important with statistical approaches.

      • Wei Dai

        With statistical approaches, you’re running a relatively simple algorithm, and just training it on a large corpus of data, rather having to hand-craft a lot of detailed rules. This makes it easier for an individual or small team to substantially overtake the rest of the field, for example by coming up with a new theoretical insight that suggests how to improve the core algorithm without doing a lot of expensive trial and error or how to get out of a local minimum that the field finds itself in.

      • Yes, a stat approach make have less code. But a single stat approach isn’t going to do for all the different topics. You’ll need a large library of stat modules to cover many situations. You might invent a new module, and if you have 100 modules that new module might make a big difference in 2% of the topics. That isn’t going to be enough for one project to take over the world.

      • Wei Dai

        I think there are multiple arguments against “need a large library of stat modules to cover many situations”. 1. The previously mentioned trend from more code to less code, which may continue for AI as a whole. 2. The presence of fundamental principles in statistics (such as Bayes’ rule) may allow for unification of approaches. 3. The fact that human brain modules are able to take over each other’s functionality when damaged, which suggests that these modules specialize by being trained on different data rather than having different core algorithms. 4. Even if we do need a large library of stat modules, that doesn’t rule out the possibility that a small number of theoretical insights could allow improvements to many modules at the same time.

        I don’t think any of these are knockdown arguments, but together at least introduce enough uncertainty to make it not implausible for a small AI team to somehow have a substantial lead at the crucial time when AI work transitions from “mostly done by humans” to “mostly done by AIs” (and subsequently to snowball the advantage).

      • I don’t think the foom scenario is impossible, just not likely enough to be the main scenario people think about. While human brains have substantial redundancy, such as in two halves, that doesn’t mean there aren’t lots of distinct modules.

      • Wei Dai

        As far as I can tell, there is not a lot of agreement among cognitive scientists about just how modular the human mind is or how much innate knowledge about specific problem/information classes those modules contain, nor is there a lot of agreement amongst AI researchers on whether AGI will be accomplished by a few big theoretical breakthroughs or lots of hand-crafted modules and heuristics (or other approaches). It looks to me like different people have different priors and there’s not enough empirical evidence to settle the matter. So I think reaching agreement on just how likely the Foom scenario is is expecting too much. But you had asked for a “plausible” story about how a small project takes over the world, and I thought we could perhaps agree on that, at least.

      • If the story is that the small project takes over by having a much better and rather general theory or architecture of intelligence or learning, then to make that story plausible one needs to make plausible the existence of such a theory or architecture. I agree that such existence is possible, but to rise to the level of plausibility, we’d want some plausible precedents of things that have looked like that.

      • Wei Dai

        What about general theories in other fields, for example, evolution in biology, Turing machine in computation, theory of everything in physics (which we haven’t seen yet but everyone agrees probably exists)? These seem like “plausible precedents of things that have looked like that”, unless I’m misunderstanding what you mean by this phrase?

      • What an AI breakthrough needs (please correct me if I’m wrong) is a formalization of induction.

        [Of course, logical positivism collapsed, and philosophers of science have since given up trying to formalize induction.
        I suspect that E.Y. might think Salamanof supplied just that formalization.]

      • I agree that there have been general theories in the past. I was thinking more specifically of powerful general theories of intelligence or learning. Not theories that explain what they are, but theories that make it much easier to achieve them.

      • Wei Dai

        You seem to think that if general theories for achieving AI were to exists, we would have seen such a theory already and it would have already made a practical impact on AI designs in the past. However such theories may exist but we just haven’t found one yet for whatever reason (perhaps we won’t find one until we can experiment with greater amounts of computing power than we currently have access to), or maybe we already have such a theory but its advantages won’t be apparent until we find a few more pieces of the puzzle or obtain greater amounts of computing power to apply the theory to.

        So I don’t think we need plausible precedents in the same field. The existence of general theories in other fields seem enough to make the existence of general theories for AI “rise to the level of plausibility”.

        Also, I think there are other plausible stories about how the top AI project could obtain a large enough advantage to snowball and take over the world, without assuming that AI design will be highly “architectural” in nature. Suppose that the leading AI projects are all trying to improve many modules while keeping the improvements secret. Two such projects may decide to merge, or perhaps a crucial member defects from one project to another and brings many secrets with him or her. The gap in intelligence between the best AI and the next best could then easily be equal to the difference between von Neumann and the average professor.

      • On general theories: if ask you to create an app that can find a person’s address from their name, I think you know that there is no general theory that can help you. Somehow you’ll need the equivalent of a database mapping names to addresses. I think we similarly know that there biological ecosystems are mostly just a mess of detail, there just aren’t going to be general theories prediction that. It seems to me that we are learning something similar about intelligence – not only haven’t we seen general theories but we seems lots of reasons to expect there aren’t such things.

      • Yosarian2

        If you ask a human to find someone’s address from their name, he’ll go find a phonebook, or else type the person’s name into Google. Or maybe even type the question into Google “how can I find someone’s address online”. He wouldn’t need a database or a database app built into his head, he would only need to understand the question and know in general terms how to find that type of information.

        The same would be true of any general intelligence, I think; the key to a general intelligence (as opposed to narrow AI) is that you can figure out the answers to problems you’ve never come across before. Human can certainly do that; at least to some extent.

      • On other stories: large systems with strong barriers which each contain many modules not found in other systems might indeed be the basis for some systems winning over others. But we just haven’t seen that happen much in the world of software. You might have thought such things would happen with programming languages or operating systems or hardware standards. And sometimes some of those have won over others. But usually there are close seconds, and such things haven’t remotely threatened to let some firms take over the world.

      • Dan Browne


      • Dan Browne

        Yes. This is likely the way forward. But it is limited in speed of development because it requires real world input to produce the training sets. Which is not to say that this method cannot produce FOOM. Just it cannot produce FOOM over days or weeks.

      • truth_machine

        “Given this, I speculate that obtaining the analogous improvement in AI intelligence may only require a few design tweaks”

        So, you think you could identify the “design tweaks” that would turn the brain of a typical university professor into the brain of a von neumann without either being a) far brighter than von Neumann or b) fantastically lucky?

        What if those differences depend specifically on a fairly large set of gene differences with complex consequences in the two organisms, plus a set of formative influences — as is likely?

      • The biological von Neumann’s brain must have been architecturally very similar to a typical university professor’s. Nor could it have contained independent improvements to lots of different modules.

        If you accept R.H.’s account of g (and we assume that von Neumann was extraordinarily high g), then independent improvements to lots of modules explains von Neumann’s intellect. Intelligence is just the average of the effectiveness of each independent module. (And this is also the model underlying classical IQ testing in the Binet and Wechsler traditions; it is also (in the jargon of its day) the Thorndikian alternative to Spearman’s interpretation of g.)

        R.H.’s account, moreover, is consistent with the findings in the correlations among human abilities. Increasing working memory or cognitive speed could seem the panacea, but they are actually quite domain specific–suggesting that better working memory and speed merely describes a feature of cognitive modules. [Also the polygenetic nature of IQ.]

        In short, your claim that von Neumanns’ brain couldn’t have contained independent improvements to lots of different modules seems not only unwarranted but is likely wrong.

      • Wei Dai

        If intelligence is just the average of the effectiveness of each independent module, it’s hard to explain how von Neumann seems much smarter than the next smartest person in his time (because we’d expect the smartest person to have the advantage of just one more improvement in their “modules” than the next smartest). However I will admit that I stated that claim too confidently. Later in the thread with Robin, I talked more about how there is disagreements among cognitive scientists over how modular the human mind is.

      • we’d expect the smartest person to have the advantage of just one more improvement in their “modules” than the next smartest

        This doesn’t follow. What you would expect is a normal distribution. You could find the expected value and variance of the difference between the top performer and the next best in a population of a given (very large) size, although I don’t know how to do it [Poisson Distribution?]

        That is to say, was von Neumann that exceptional? In many large, highly competitive realms, such as athletics, it seems to happen often enough that the top performer towers over the next best. [I’d do the math if I knew how.]

      • Owen Cotton-Barratt

        I thought that specialization on the table was “coding smart AI”. This isn’t super-narrow, but it is much narrower than “being smart”.

      • I question whether such a specialization exists. You can specialize in different kinds of coding, but how does that make your system smarter and learn faster?

      • Owen Cotton-Barratt

        Right. I don’t know — if I did that would be close to knowing how to do the thing, as the idea is that there are some insights you can generate which make writing faster and smarter systems easy, and that you could specialise in generating these insights.

        I wouldn’t be surprised if that specialization can’t exist, but nor does it seem completely outlandish that it should do so.

      • truth_machine

        “I wouldn’t be surprised if that specialization can’t exist, but nor does it seem completely outlandish that it should do so.”

        It depends on how much you know about the nature of software, algorithms, and the shape of the solution space.

      • truth_machine

        Being smart is presumably a prerequisite to being able to code smart AIs.

      • Dan Browne

        Maybe not. Maybe being “fast” but stupid is good enough.

      • truth_machine

        The context is human programmers … fast stupid programmers can’t code smart AIs. But fast stupid processes, a la evolution, may be able to produce smart AIs.

      • Vadim Kosoy

        > …to achieve this scenario a project just can’t specialize at being better at “intelligence” or “learning” – those aren’t in fact topics in which one can usefully specialize much.

        If this is the case, why homo sapiens were selected for hunting mammoths but ended up with the ability to invent spacecrafts and particle colliders?

      • Humans have a package of mental modules with especially broad application. That gives humans relatively general minds. But that isn’t because we have specific modules to provide this generality.

      • Vadim Kosoy

        I think it is obvious that there is no “bike manufacture” module in this package.

        The transition from narrow intelligence to general intelligence has to be more abrupt than gradual, more qualitative than quantitative. Otherwise, evolution would have stopped at a point in which our intelligence was sufficient to escape tigers but insufficient to discover general relativity.

        Given such an abrupt transition exists, it is conceivable a single project will make it before others. Once this happened, the qualitative advantage gained by this project might be leveraged for further improvement in a feedback loop, leading to a superintelligent singleton.

      • I agree there is no bike module.

      • truth_machine

        I think this discussion suffers from the participants not knowing much (that is known by our science) about human evolution.

      • truth_machine

        “If this is the case, why homo sapiens were selected for hunting mammoths”

        That requires signaling and planning.

        “but ended up with the ability to invent spacecrafts and particle


      • Eliezer Yudkowsky

        > Put another way, to achieve this scenario a project just can’t specialize at being better at “intelligence” or “learning” – those aren’t in fact topics in which one can usefully specialize much. The project has to instead be better at thousands of specific kinds of intelligence or learning. Which conflicts with it starting out as a small project.

        Is this the keystone of your rejection? If there was such a thing as a compact agent good enough at “intelligence” or “learning” that it could outrun the rest of a human economy, does that mean the Hansonian story fails?

      • I think so yes. Of course lots of hard-won knowledge might be compressible into a small number of bits. So a very smart creature might fit in ten megabytes, even if it is composed of thousands of modules none of which are about intelligence or learning in general, but instead about how to do those things in particular contexts.

      • I added a version of this last paragraph to the post.

      • sflicht

        Your points (2d) and (2e) strike me as perhaps overly confident in the ability of firms to protect their IP. The breakthroughs leading to the initial advantage might be something Larry and/or Sergei come up with while daydreaming in the shower one morning. But more plausibly they’ll be the work of large teams of people, many of whom recognize the revolutionary potential of the advance. It would be difficult to keep this sort of development completely under wraps, and I’d posit nigh on impossible.

        The obvious comparison to make is hedge fund alpha. I can think of several channels by which this leaks out. (1) Attrition of personnel with privileged information, NDAs notwithstanding. (2) The execution of strategies with positive alpha might produce market data which makes the discovery of those signals by other parties more likely. (3) The mere existence of consistently profitable hedge funds (even if many hedge funds are not profitable) serves as confirmation to skeptics that investment in alpha research could be worthwhile, producing more of that research, increasing the likelihood of the original alpha being independently rediscovered.

        All three avenues seem to translate directly to software development and AI research, and I’m sure there are others that one could come up with.

        My claim is that in conjunction such forces are pretty strong and probably serve to disseminate the informational advantage on a timescale of months to several years, but certainly shorter than decades. Thus I’d suggest that Robin’s argument — at the very least — places a lower bound on how strong a positive feedback loop must be if it is to lead to FOOM before general societal technological advance, mediated by IP leakage, blunts its edge.

      • truth_machine

        ” the same sort of qualitative threshold that separates humans relative to other primates”

        That’s language, or symbol manipulation generally. The notion that there’s something like that over all of cognition is not supported by any evidence or logic.

      • truth_machine

        ” I do think it’s disingenuous”

        I think it’s disingenuous to toss that word around as you do.

      • John_Maxwell_IV

        Why does it make sense to think about a qualitative threshold rather than a quantitative threshold? If I look at the human population, I don’t think I see some humans who have the “qualitative” property of being able to do AI research and some who don’t have it… I would guess that some people can’t do AI research at all, some people can do it pretty slowly and ineffectively, some people can do it quicker and more effectively, etc. I would expect AI researcher competence to be distributed fairly continuously among the people interested in doing AI research (lots of unknown graduate students working hard and making marginal contributions), and I’d also expect that a computer’s level of AI research competence could similarly be accurately modeled as a quantitative, continuous parameter. Your foom argument seems predicated on this “qualitative” capacity jump that I’m not persuaded of.

      • Dan Browne

        I don’t believe that useful software that can interact with the external world can FOOM without input from the external world and thus any kind of FOOM is bottlenecked by the speed at which in-real-world R&D can take place.

    • While today single firms often offer products that provide distinction functions for customers, it is much rarer for a single firm to offer basically all functions for their customers. Google might offer a Translate service that hides many different contributions to that service. But there isn’t a Google Life from which you’d buy everything you need in life.

      Also, surely modern software systems such as the ones you mention do in fact contain many distinct software modules.

  • Nick Bostrom

    I think we’re unlikely to resolve your “foom” question in a comment thread when your extensive previous discussions with various people haven’t managed to do so.

    But I take issue with what you describe as the “bottom line”—it doesn’t match up with the text. Part of the book is independent of how rapid the takeoff is, part of it considers the case for a fast takeoff (rather than using that as a premise), part of it explores the implications of a multipolar outcome (which is particularly likely if the takeoff is slow), and even the part that is premised on a fast takeoff is not premised on a single software project being able to take over the world within weeks.

    I do spend somewhat more chapters on unipolar than on multipolar scenarios. There are two reasons for this: first, because I regard them more probable (on grounds detailed in the book); second, because I think we have more expected influence in such scenarios, so they are more important to analyze.

    • Books are long things and can cover a great many topics. But clearly a large central portion of your book is focused on the scenario of an intelligence explosion creating a singleton. I’m questioning the grounds for seeing that scenario as more probable. *How* exactly does one small project grow so much faster than the rest of the world?

    • To be more precise, I changed “Nick Bostrom’s whole book” to “much of Nick Bostrom’s book.”

  • Marc Khoury

    QUOTE: “Intelligence” just means an ability to do mental/calculation tasks.

    It’s more than that. With this definition, a calculator would be intelligent in the same way we are, but to a lesser degree. (Same with a ‘dumb’ paper/rock computer). It’s more than that though, what we seem to define as intelligence is the ability to predict futures, and the ability to know which futures our actions have more likelihood of bringing.

    QUOTE: Since software is often fragile and context dependent, much innovation consists of making new modules very similar to old ones, except that they work in slightly different contexts.

    A large part of programming is not only make those “modules” coded in a better way, but also to abstract the modules one after the other, to be able to make modules that make other modules. Some of them are made through selective iteration, some of them are made through systematically applying a set of rules, a lot of them are not just “revamped” ones. They’re shortcuts to make a whole spectrum of modules, that only get exponentially larger at each level of abstraction (module to make modules that make modules, module to make modules that makes modules that makes modules, etc.). “Centuries” seems greatly exagerated when we look at the level of abstraction we have achieved in current programming languages. The Wolfram Alpha language alone has an amount of abstraction that noone 30 years could have expected. We are not sitting blindly coding away, we are making programming itself better and easier.

    QUOTE: A system that can perform well across a wide range of tasks probably needs at least thousands of good modules. Same for a system that innovates well across that scope. And so a system that is a much better innovator across such a wide scope needs much better versions of thousands of modules. But this seems like far more innovation than is possible to produce within projects of the size that made nukes or rockets.

    An AI would not need to be confined to the set of servers the people building him give it. It can host parts of itself online, and easily make use of the free computational power available through the net and could eventually even just hack computers to gain their computational power. Botnets are large and far-spreading, it’s definitely not infeasible for an AI to create its own botnet. The abstractions created through packaging and virtualizing are making automation a huge part of creating software. It makes it so that humans are now creating highly sophisticated software that do more than just recycle the same things. We are creating tools that an AI can use to understand the world, because we’re creating tools to understand the world ourselves.

    QUOTE: Similarly, while you might imagine someday standing in awe in front of a super intelligence that embodies all the power of a new age, intelligence just isn’t the sort of thing that one project could invent.

    It’s not actually one project, it’s thousands/millions of project in neuroscience, biology and IT that all collaborate together. The science is not siloed, it’s shared between people. One group alone is not creating intelligence, one group is putting the proper pieces out of all the ones everyone created in the proper design, and the groups of people trying are growing, while the science grows as well. Once the proper pieces are put together though, it could eat the world rather quickly. The speed at which it could harness enough processing power to surpass human thinking might be shorter than you think.

    • Predicting the future is one of many relevant tasks. Yes of course we tray to abstract to reduce fragility, but we don’t get very far that way. Yes of course a system could move across servers. I said “one project” because that’s what Bostrom said.

    • truth_machine

      “We are not sitting blindly coding away, we are making programming itself better and easier.”

      First: only marginally. Second: these are not the same “we”s. The programmers at Wolfram are making all programmers who use the Wolfram “Language” marginally more effective, but very few programmers are working at Wolfram.

      And, nice as the Wolfram “Language” is, not many people are using it, for good pragmatic reasons.

      • John_Maxwell_IV

        Wolfram/Google could be a good concrete example of an organization that might build a foomable AI, e.g. here’s a quote from Stephen Wolfram:

        when it comes to developing algorithms, we’re in a spectacular position these days. Because our multi-decade investment in coherent system design now means that in any new algorithm we develop, it’s easy for us to bring together algorithmic capabilities from all over our system. If we’re developing a numerical algorithm, for example, it’s easy for us to do sophisticated algebraic preprocessing, or use combinatorial optimization or graph theory or whatever. And we get to make new kinds of algorithms that mix all sorts of different fields and approaches in ways that were never possible before.
        From the very beginning, one of our central principles has been to automate as much as possible—and to create not just algorithms, but complete meta-algorithms that automate the whole process of going from a computational goal to a specific computation done with a specific algorithm. And it’s been this kind of automation that’s allowed us over the years to “consumerize” more and more areas of computation—and to take them from being accessible only to experts, to being usable by anyone as routine building blocks.


      • Software firms have been making claims like that for decades, and none of them have taken over the world of software on the basis of their superior software environments.

      • John_Maxwell_IV

        Well home prices always go up right? You can only extrapolate so confidently from historical data.

      • truth_machine

        “in any new algorithm we develop, it’s easy for us to bring together algorithmic capabilities from all over our system”

        Meaningless drivel … you appear not to know what an algorithm is.

      • oldoddjobs

        The quote is from Wolfram.

      • John_Maxwell_IV

        That wasn’t me, it was Stephen Wolfram. Note the link at the bottom of my comment. It’s not articulated as precisely as it could be, but I think I understand what he’s trying to communicate.

      • truth_machine

        Fine, then Wolfram doesn’t doesn’t know what an algorithm is … or he does, but he’s just spouting BS … something he’s known for … for instance, his “A New Kind of Science” was rated by scientists as being pretty high on the crackpot index.

  • 5ive

    It seems that software is as you describe but that may not
    be a helpful way to understand Foom. Once we understand intelligence well enough to create an artificial one, we will understand one well enough to improve it. And while human brains are slow to improve software, that’s not really a useful analogy, since the improvement isn’t acting on the thing that’s also *doing the improving*, so there’s no exponent involved aside from the one affecting the hardware substrate. What would happen if humans had a way to improve our own brains? That’s more akin to what would happen with an AI. One improvement would make the next
    improvement iteration better and faster and so on, independent of the hardware.

    • Better understandings may well lead to better ways to improve software. But that would be true both inside and outside of this imagined future project. How does this lead to a net advantage for the project relative to the world?

      • arch1

        Because 2^((t+head_start)/dbl_time) >> 2^(t/dbl_time) for sufficiently large values of head_start/dbl_time. But you know this so the disconnect must be elsewhere.

      • You seem to be suggesting that the project contains a much better understanding than outside, for a time duration head_start. That pushes the question back to how this project started out with such a vastly better understanding in the first place. Time 0, both understand little. Time 1, project understands lots, outside little. Time 2, both understand lots. This suggests that the thing to be understood is small and compact, not a collection of thousands of things each of which was developed separately.

      • John_Maxwell_IV

        “You seem to be suggesting that the project contains a much better understanding than outside, for a time duration head_start.”

        He might just be suggesting that even a small difference in understanding could grow exponentially in to a large one (because the difference between the two expressions he describes is itself an quantity that grows exponentially with time).

      • arch1

        I think all I’m doing is taking the definition of crossover seriously. If a bunch of racehorses are running toward a line called ‘crossover’, and the rule is that their speed starts growing exponentially once they pass that line, then the first horse to pass the line may well cover more distance than all other horses combined thereafter.

        One possible line of attack against the plausibility of the entire scenario might be to argue that the leading project at crossover time is very unlikely to be a secretive one (because secretive orgs are disadvantaged relative to open ones, or whatever). If true, this would seem to have at least two desirable consequences: a) the ‘leading’ org, being almost certainly open, is presumably less likely to desire world domination, b) the expected value of head_start is reduced (shrinking the “temptation window” – after all, even a superhuman AI may in some respects be only human:-)

      • Why would there be such a sharp line where passing it makes growth rates vastly higher? Why wouldn’t growth rates instead continuously increase over some wide range?

      • arch1

        I guess they would, in practice (because, as suggested by the word “mainly” in Bostrom’s definition, ‘crossover’ is not really a sharp line; rather, as a project’s ability to self-improve gets increasingly decoupled from people and other external speed constraints, its cycle time reduces accordingly).
        I don’t think this simplification qualitatively changes the basic conclusion though (namely, that if the lead project gets to “sufficiently-autonomous self improvement” mode sufficiently in advance of all other projects, this can result in its having a *huge* edge over any other project, one which is potentially decisive over the rest of the world combined).

      • 5ive

        Well, you mention that “we are bad at making software to make more software.” Sure. But what would be different if we were *good* at making software to make more software? And beyond that, those improvements made us *even better* at making software to improve software. That’s exactly what an AI would be.

  • arch1

    Robin, based only on your posting as I haven’t read Nick’s or your and Eliezer’s book: It seems at least plausible that a system which
    1) *can* achieve crossover as defined above,
    2) thus climbing (almost by definition) onto an exponential capability trajectory ,
    3) has been developed (and evolves) in relative secrecy,
    4) permitting it to reach crossover a sufficient number of doubling times before any other,
    5) and *wants* to take over,
    6) *could* take over

    With which part(s) of this rough outline do you most take issue? (Sorry, it’s not exactly in time sequence but I think you get the gist)

    • It is the positive slope of that curve that I’m questioning. The background world grows very slowly as the project grows faster – but how does that happen?

      • arch1

        [if I change your “very” to “relatively”] because it, and *only* it, has during that period achieved crossover

  • You seem to focus mostly on the likelihood of the Foom taking place over weeks. But surely most of the work and concern (e.g. “Friendly AI”) only requires a software-based self-improving AI, where the improvements quickly leave the control and understanding of humans.

    Even if it takes a century to get the first human-level software AI, and then another century until the AI is mostly improving itself, rather than being improved by humans — don’t you still wind up with essentially the same concerns that the foomsayers are worried about?

    What is so significant about the “in a couple of weeks” part?

    • The whole point is that the project grows faster than civilization, so that the project can take over the world. If the project grows slower, it can’t take over.

      • Sure, but this isn’t the concern you were addressing in the OP. This is the new scenario: the whole human civilization cooperates to produce the first (software) human-level AI over the next century. Then the whole human civilization cooperates for another century to improve it, including adding self-improving features. Over that second century, the device itself takes on more and more of the burden of its own improvement.

        By the end, it is improving faster on its own, than any system involving human beings. Faster, eventually, than the entirety of human civilization.

        It seems to become far more likely, that some device can (eventually) grow faster than civilization, if you simply take out the “in a few weeks” part.

      • I’m happy to grant the claim that eventually the part of civilization that is without humans could grow faster than the part of civilization limited to using humans.

      • IMASBA

        I agree with Don Geddis that yes, it probably would take longer than a few weeks but in the grand scheme that doesn’t really matter. Also, when talking about resource limitations why do you assume the AI (or its creators) would play by the rules, why can’t the AI simply hijack hardware that is connected to the internet, to increase its own computational power and also steal software modules?

        “If the project grows slower, it can’t take over.”

        I don’t understand that statement. Surely a machine with an IQ of 1000, at least average human EQ and vastly superior memory recall could manipulate and scheme its way to world domination using existing human powerstructures, wouldn’t you think? It could be as simple as the scenario behind The Terminator where the first sentient AI causes the US and Russia to nuke each other, severely thinning out potential human resistance to its power.

      • Dan Browne

        “Surely a machine with an IQ of 1000…”
        But if there exist hybrid machine-humans with IQ’s of 800 (and lots of them) would it still be so easy?

      • Dan Browne

        This. If humans can keep up, the project can’t take over. I see humans being able to keep up if the timescale is years rather than minutes to days to months. Especially if technology to interface with the machines keeps up.

  • Owen Cotton-Barratt

    Thanks for the post, Robin. I’m sympathetic to your argument and find it quite plausible that fooms are either impossible or vanishingly unlikely.

    I don’t think that it’s anything like a watertight case, though. So I don’t think that we should assume that it won’t happen — particularly as (Bostrom’s second reason for spending more of the book on this) it looks particularly easy to hedge against bad outcomes if fooms are possible relative to if they aren’t.

    • I didn’t at all mean to suggest that Bostrom’s book wasn’t worth writing, or that we shouldn’t try to protect from unlikely but very impactful events.

      • How unlikely would a risk have to be that you would deem it to be not worth writing a book about?

      • Seems this sort of question is what expected utility is for.

      • Yes, to paraphrase my question: what is your threshold at which such a calculation falls into the category of Pascal’s mugging?

        Matrix Takeover Institute: Hi, we would like you to consider merging your resources with ours to hack the Matrix.

        Eschaton Foundation: Your mission seems to be extremely unlikely to succeed and is based on insufficient evidence. We will therefore continue to use our resources to create a benevolent artificial general intelligence to safely navigate us through the forthcoming technological singularity.

        Matrix Takeover Institute: Our mission is based on years worth of disjunctive lines of reasoning. There are good arguments underlying the possibility that we are actually living in a simulation. Therefore we should be concerned about the risk of a simulation shutdown. A simulation shutdown might knock out scientific advancement before we create an AI singularity. Hacking the Matrix also promises an enormous payoff in resources that does easily outweigh the larger probability of a negative technological singularity.

        Eschaton Foundation: I’m a major fan of Down-To-Earthness as a virtue of rationality, if you can’t argue for a medium probability of a large impact, I do not bother.

        Matrix Takeover Institute: I don’t think the odds of a simulation shutdown are small. I think they’re easily larger than 0.00001%. And if you can carry a qualitative argument that the probability is under, say, 0.000001%, then that means hacking the Matrix is probably the wrong use of marginal resources – not because a benevolent AI is more important, of course, but because other ignored existential risks like unfriendly aliens or the Large Hadron Collider would be more important. I am not trying to play burden-of-proof tennis. If the chances are under 0.000001%, that’s low enough, we’ll drop hacking the Matrix from our consideration until everything more realistic has been handled.

        Eschaton Foundation: But a probability of 0.00001% is too small to take into account.

        Matrix Takeover Institute: How do you decide what probability is small enough to ignore? I think you are drawing arbitrary lines to obtain the desired result where your mission has exactly the right proportion of probability and expected utility to disqualify any other cause.

        Humbug and Partners: I accidentally overheared your discussion. May I step in here by introducing myself? Humbug and Partners is the earthly proxy of the lords of the Matrix. Both of you should instantly stop pursuing your mission or otherwise this and 3^^^3 similar simulations will be terminated. And as you know, the most common formalizations of Occam’s Razor, Solomonoff induction and Minimum Description Length, measure the program size of a computation used in a hypothesis, but don’t measure the running time or space requirements of the computation. A compactly specified wager can grow in size much faster than it grows in complexity. The utility of a Turing machine can grow much faster than its prior probability shrinks.

        Eschaton Foundation: I’d sooner question my grasp of “rationality” than to do what you want because I thought it was “rational”.

        Matrix Takeover Institute: I do agree with the Eschaton Foundation here.

        Humbug and Partners: Intuition is not the most reliable guide for what policies will actually produce the best results. The ability to shut up and multiply, to trust the math even when it feels wrong is a key rationalist skill. Especially the Eschaton Foundation should acknowledge the fact that an artificial general intelligence build according to your current grasp of rationality would take our warning seriously because the tiny chance that 3^^^3 simulations similar to this one will be shut down would override everything else in the AI’s calculations.

      • DanielM_113

        Think of all the angels dancing all of the ballet plays ever writen, right on the head of this pin!

  • Pingback: Overcoming Bias : Conflicting Abstractions()

  • Is general intelligence modular, or can it be captured by a few general principles? If the latter is true, then in a world where every market niche is occupied by domain specific narrow intelligences, can such a general intelligence outcompete them?

    Suppose that the company which created this general intelligence wants to enter the market for optical character recognition software (OCR). Could it be better than the existing products? Or suppose that they wanted to sell their general intelligence as a chess software. Could it possibly have a better cost-performance ratio than existing software? I strongly doubt it. I think it would likely require more resources than any OCR software, and be more expensive. And even if it was better at chess than existing software, the difference would not be worth the price, because existing software can already beat humans.

    So how exactly would this general intelligence take over the world? Any advantage it might have, from today’s perspective, will already have been realized by the underlying technologies behind it. If your general intelligence could theoretically solve the protein folding problem, then this will be more efficiently realizable by a narrow specialized version.

    Why would a baker have to be able to talk? Why would a taxi driver need to be able to write poems? Why would a mathematician need to be good at engineering? Once you can create a general intelligence you will have solved all of the former abilities, rendering your general intelligence economically unviable.

    • Wei Dai

      If I created a (controllable) AGI and I wanted to enter the OCR market, I wouldn’t give my AGI the task of recognizing individual characters. I’d make a lot of copies of my AGI and tell them to design the best specialized OCR software possible.

      • Will Pearson

        So I suppose AGI will come about when there has been sufficient development of supporting tools to make developing it cheaper than solving a set of problems directly.

        An interesting question is whether we are developing the supporting tools. I suspect some maths will be applicable. But I think we need a stronger theory of General Intelligence to know whether we are developing the supporting tools or not.

      • DanielM_113

        Yes we are developing machine learning quite fast, and the speed is increasing in over the proportion of increasing hardware power, because the increased hardware makes the current techniques more practical, and future developments exponentially more reacheable.

      • That would not be possible, because OCR companies will have already hit diminishing returns by using a narrow AI that is superhuman at programming, and another narrow AI that is superhuman at algorithm optimization, in order to improve their OCR software.

        The problem is that the tools and discoveries necessary in order to create an AGI will have already been sold and spread among the markets before anyone can eventually create an AGI. Those tools and discoveries will allow humans to do everything such an AGI can do.

        Consider having one narrow AI to tell you what illness you have, another to find the cure, and a third one to apply the cure. If someone then comes up with a general algorithm that can do all three jobs, that general algorithm will very likely be less efficient than the specialized versions.

      • DanielM_113

        During the first months maybe, but as soon as someone runs that AGI at a thousand times faster computer, itself will have expertise of a human lived thousands of years specialized on its field of work. If it specializes in designing software, it will be the most economical and efficient of any system involving humans as part of the process. If it becomes an expert on designing, researching and perfecting new AIs, then foom.

      • Dan Browne

        And therein lies the bottleneck: “researching”.

  • Your intuition that your mind works with “modules” is very suspect to me.

    The brain, or rather, the cortex, is not particularly decomposable into modules. It’s fairly uniform stuff. It seems clear that the material our brains use for thinking is governed by a single algorithm applied generally. It’s the application of the algorithm that causes specialization of particular areas on particular tasks.

    This suggests to me that being sanguine about an intelligence because it will need to develop lots of “modules” is rather missing the point.

    The “module” that AI foom is addressing is the intelligence algorithm itself – the bit that is able to create modules. This is the very bit of intelligence that we are not able to write software for – we don’t have software that’s able to meaningfully take high-level specifications and turn them into useful programs outside of extremely niche areas.

    We have no piece of software that we can set running on a computer, and then be able to teach the machine / software how to drive a car, or play a musical instrument, or walk down the road.

    Without software that can learn almost arbitrary things, we don’t really have AI at all.

    But if we do ever figure out that universal learning algorithm, and we apply that learning algorithm to the process of improving the learning algorithm itself, we then create the conditions for an exponential explosion in intelligence.

    Under the reasonable assumption that most learning is parallelizable, such an intelligence should be able to create “modules” – i.e. learn – a million different things simultaneously. The time between it knowing nothing but how to learn, vs being an expert in everything, may be no longer than the time it takes to become an expert in the thing that’s hardest to learn.

    I don’t think a “module” deficit is a significant defense against something that learn all the things faster than you can learn any one single thing.

    • It is a rather standard view in brain science that the brain is in fact composed of modules. We already have universal learning algorithms; they just aren’t very good unless augmented with lots of knowledge about the particular kind of thing they learn about. And I don’t think anything can learn well and fast in many diverse areas without having lots of modules that embody insight into different approaches for learning well in different areas, as well as how to recognize in which areas to use approaches.

      • DanielM_113

        We have generalized learning algorithms, but our current computers cannot run them at comparable speeds with the processing power of the human brain, much like Kurzweil proposes, 2*10^16 FLOPS.

        Only when experts start to routinely run these algorithms at these comparable speeds, so they can study and experiment with, the final optimizations will make possible to reach calculation efficiencies comparable to the human brain. From then on these systems will be able to have a better grasp of their own working than human can.

    • Caravelle

      The brain, or rather, the cortex, is not particularly decomposable into modules. It’s fairly uniform stuff. It seems clear that the material our brains use for thinking is governed by a single algorithm applied generally. It’s the application of the algorithm that causes specialization of particular areas on particular tasks.

      What kind of knowledge on human neurology or cognition are you basing this on ? From all I know on the subject it’s very UNclear that this is the case. Completely the opposite even.

      • DanielM_113

        Experiments with primates show that damage to parts of the cortex can cause transfer of function to other parts, without structural change, e.g. damage to visual cortex can cause parietal cortex to assume visual function.

      • Adam Casey

        That’s a distinct claim surely? A thing can be modular and plastic at the same time. The liver can repair damage, but it’s still modal in function.

      • Caravelle

        What Adam Casey said. That parts of the brain can assume different functions doesn’t mean those functions themselves aren’t different, i.e. there are a whole lot of algorithms going on. Our intelligence doesn’t consist of a single algorithm applied generally, but of the interaction of many different algorithms that do different things (note that this doesn’t necessarily mean AI would have to be the same – like airplanes vs birds, just because nature did things one way doesn’t mean it has to be the only way).

        Besides brain plasticity does have limits – otherwise there would be no such thing as being permanently impaired from localized brain damage.

  • truth_machine

    I’ve been a professional programmer for nearly five decades and have followed software methodology very closely and AI technology and neuroscience closely, and to me most singularians come across as crackpots who don’t understand either computer science or the brain (or evolution) very well. The main feature of algorithms is their *ad hoc* nature … there are no general principles that can be applied across the board to solve algorithmic problems. Of course there are clusters of problems within the infinite problem space, but that space is infinitely convoluted. There are techniques like genetic algorithms that have wide application, but they are slow to yield results and do so unpredictably. There is no reason at all to expect a system to stumble across some method or mechanism that can lead to a rapid increase in results of any duration.

    And as for the human brain, it is the furthest thing from barrkel’s description of uniformity governed by a single algorithm. Like everything produced by evolution, it is a complex ad hoc mechanism with lots of inefficiencies and special purpose behavior. We are discovering that synapses are immensely complicated and that these complications matter and cannot be ignored in simplistic models. And brains do some things well like promote biases and do other things poorly like remember numbers of a few digits or do arithmetic with them, and brains can’t just be tweaked to work differently, there are fundamental structural issues. The brains of geniuses, idiots, hyper-awake people and comatose people very closely resemble each other but we can’t distinguish them and don’t know what does. And heck, we can’t even build a spider’s ganglion, yet alone something with the capabilities of a human brain, and our computers are far worse at these things than we are.

    Which is not to say that it isn’t theoretically possible … it is. But the timeframe is way beyond the point where global warming will destroy our ability to solve any technical problem.

    • I’m not sure you can have it both ways. If brains have fundamental structural failings, then better structures would make a big difference across a wide range of tasks. Then there would be general principles, about structure, to learn that would help in many problems at once.

      • truth_machine

        I didn’t say anything about “fundamental structural failings” … are you always this inept? As I said, you crackpots have no understanding of either how brains work or of computer science.

      • oldoddjobs

        Jesus. Get over yourself, you daft penguin you.

      • truth_machine

        Jesus doesn’t exist and, as far as I’m concerned, neither do you.

      • oldoddjobs


      • Flavio Abdenur

        “The main feature of algorithms is their *ad hoc* nature … there are no general principles that can be applied across the board to solve algorithmic problems.””

        This is becoming less and less true, thanks to recent advances in machine learning.

      • I didn’t say anything about “fundamental structural failings” … are you always this inept?

        You wrote, “And brains do some things well like promote biases and do other things poorly like remember numbers of a few digits or do arithmetic with them, and brains can’t just be tweaked to work differently, there are fundamental structural issues.”

      • truth_machine

        “issues” and “failings” are different words, you imbecile. Again: I didn’t say anything about fundamental structural failings. For instance, you can’t tweak a stomach to think or a leg to see … that’s not about fundamental structural *failings*, but it is about fundamental structural *issues*.

        Again: inept crackpots lacking understanding.

      • Only a mental midget (or a lawyer) resorts to semantic dodges to save himself.

        You’re actually similar to EY in your flaws, which is why you despise his ilk: you can’t write (or think) precisely, and you aggressively exploit your incapacitiy in order to save your arguments.

  • dmytryl

    I came to realize that there’s nothing really to get. It’s the second coming, the ‘skynet awakening’, a part of our cultural heritage, and it is something strongly preferred by narcissistic minds engaged in “fantasies of unlimited power, success, intelligence”. But that’s all there to it.

    • DanielM_113

      So you think there can never be a self-reinforcing economic and technical system which grows in a doubly exponential pattern?

      Production -> Accumulated knoweledge -> Increased production -> Increased accumulation of knowledge -> Double increase of production

      For most of history human population followed the double exponential growth. The human population don’t follow double exponential growth today only because of mass economic and education interventions directly or indirectly discoraging reproduction.

      One overlooked but significant such intervention is the growth of taxation from 5% to 30% or more. This makes the economic worth of future progeny much lower, even if economic gain would not be a major motivation, now it would be fractions of what it were a century ago. Also the young would value less their progenitors that today can accumulate much less capital than before.

      There is reason to think tools and machine production follows the same double exponential pattern.

    • My diagnosis is that its an effort of some newly atheistic folks to cope with their fear of death that, because of their previous religiosity, they’ve never faced and overcome. Same with cryonics.

  • mjgeddes

    The creation of software begins with an abstract description
    of the things in a given subject area (‘domain’) and the logical relationships between them (a ‘data model’). The
    programmer then translates this into logical instructions using a programming language, which describes logical processes.
    In the final step, the programming language itself needs to be
    translated into concrete computer hardware operations (machine code). To summarize, there are 3 general steps
    (‘levels of abstraction’):

    Abstract Data Model >>>>
    High-Level Programming Language >>>>
    Low-Level Machine Code

    Artificial general intelligence is just this – it’s the capacity to take an abstract model of some subject area, and translate it into concrete computer operations.

    So the FOOM debate boils down to how easy it is to create a
    program that can automate the very process of software design itself. For a single powerful super-intelligence to take over the world, there needs to exist a general method of creating abstract models about anything and everything that can could exist in reality, and translate it into precise machine code!

    Are there are a small number of general categories (‘super
    classes’) which we could program into a computer that would, in principle, enable it to create a model about anything in reality? Or is this sort of all-purpose, universal language an illusion?

    Let me list my proposed set of super-classes. I do, in fact, have a list of 27 categories that might, in principle, be capable of serving as the putative ‘universal language’ i.e., sufficient to model any part of reality whatsoever!

    Here is my list of 27 super-classes:

    Symmetry, Force, Field

    Perfection, Liberty, Beauty

    Ordering, Relation, Set

    Process, Action, Signal

    Goal, Decision, Sign/Meaning

    Predicate, Probability, Category

    Object, Structure, Interface

    Meme, Project, Narrative

    Operation, Program, Data Model

    The idea is that we could attempt to write a program designed describe the world using only these 27 super classes. It’s important to understand that we are not attempting to define these terms ourselves. We are not looking for precise definitions of anything. This is something totally different to ordinary programming. The program itself is expected to define the terms.

    The idea is that a putative artificial intelligence would use the super classes as a sort of ‘universal alphabet’ capable of describing anything in reality whatsoever *including itself*. In other words, the super-classes are used to define themselves (‘reflection’) in order to produce ever more accurate definitions, in a closed loop.

    To summarize in one paragraph of plain English: The fantastical super-hack is basically this: imagine a computer simulation of all of reality. The simulation would include simulated super-intelligences. If you have a language capable of describing all of reality, you can search for the data structures corresponding to super-intelligences, and extract the data from the model. This is how an artificial intelligence could rapidly pull itself up by its own bootstraps and FOOM.

    • I’m rather skeptical that your proposed approach is sufficient.

    • DanielM_113

      Basically, a large enough neural network to model natural language, all human literature (including technical and scientific literature) and then generate structured answers to any type of question, either philosophical, logical or technical evaluated by human experts. A network like this could be required to read its own code and rewrite it with incremental optimizations or structured additions that improve performance.

      You would’t need it to start from the super-classes, because those classes are already inside the human literature and technical knowledge. If it can articulate this knowledge it is articulating the super-classes too.

      Another way to create such neural network starting from the super-classes would be to give the network continuous input controled by humans and sparsely labeling them according to the super-classes, and then querrying the network about its unlabeled data it might be most in doubt. These superclasses could work as reinforcement signals for learning this way.

    • Do you have an argument that these 27 classes suffice to describe anything?

      How would evolution have hit on 27 abstract concepts that are completely precise?

      • mjgeddes

        I really don’t want to elaborate too much here on my ideas. These 27 categories of mine were obtained through direct conscious-reflection after a long period of time. I’m very confident that they are indeed sufficient to fully encapsulate all reality.
        Of course, this word list is just a list of pointers to what is a complex structure – one would still need to hand-code enough initial structure to get a working program.
        If evolution had not hit on these concepts and embedded them in our brains a-priori, we would not be here to talk about it.

  • Rafal Smigrodzki

    You may want to think about how “tiny” are the resources available to the singleton seed AI. AI research uses an infinitesimal fraction of all economic resources, and a very small fraction of computational resources which makes almost all resources irrelevant here. What matters is the fraction of *AI-relevant* resources available to a single project – Is it 1%? 10%? Google-knows %? Seen this way, the seed AI could be actually relatively large compared to the sum of its competitors.

    And of course there is the old viral-internet-takeover-botnet-from-hell story, where a seed AI gains massive resources from even a small improvement in its ability to circumvent computer security, strongly tilting the field in its favor.

    • The category of software is larger than that of AI software. Still, software is still a small fraction of the world because we are bad at it. As we get better, its fraction will grow. But a seed AI just isn’t much of a threat when we are bad at software.

      • Dan Browne

        Hi Robin, the “bad at software” is part of it. If we make the assumption that we have sufficient hardware to simulate *useful* levels of intelligence (word useful carefully chosen) but we’re not there yet, then it *might* be the software holding us back.
        But let’s talk about the software: we have hand-coded software (which I don’t see improving any time soon) and we have evolved software (which we haven’t improved upon but may be about to).
        The way I see it, we are close to the limits of what humans can do today and can’t move any faster – the vast volume of information in the R&D realm is just too big and too complex for any single human to understand and correlate. Likewise for teams of humans. The bottleneck is networking.
        That may be solved by software. Humans can’t easily download an exact copy of ideas/knowledge in a rapid fashion from one brain to another. It takes years of intensive study to get there.
        With tools like IBM’s watson which can understand the meaning of what they are reading AND can read very, very quickly, even without actual AGI we might be able to improve on our software design and iteratively improve something like neural nets for the various intelligence modules you speak of. Once that’s been done, takeoff could be very soon afterwards since e.g. a recognition system for the limited number of visual objects any entity is likely to encounter in the environment will likely be good enough.
        The point is, it only has to be done *once* for each module, then the hardware speedup (of 2X every couple of years) alone will take us to the next level.
        Once we have gotten to the next level it should be relatively straightforward for machines with effectively very large short term working memories who can simultaneously understand and correlate millions of research papers, to suggest experiments to further improve things.
        THEN you get the intelligence explosion…

  • Ryan Carey

    Robin, it’s at least conceivable that an AI company might make enough profit by selling 1% of their tech while reinvesting the rest. Especially if they had algorithms which could outperform humans in a wide range of tasks across sectors.

    i.e. you’d have to concede if you found that a few tech companies were:
    1 – gaining a rapidly growing share of the world economy

    2 – investing in algorithms
    3 – apply their algorithms to do more diverse tasks across sectors.

    I can’t discern this trend yet but some people think they can, which is worth attending to…

    • Some people think they see a few tech firms growing fast, doing diverse tasks, and selling access to only 1% of their tech? Now, today?

      • Ryan Carey

        Not yet, although they might say that we’re encountering more modest warning signs resembling 1-3.

        I think you think that a company would not ever sell a small minority of their useful tech and I’m not sure why. Surely a company might plausibly sell better decision systems across a wide range of industries while keeping a kernel of general artificial intelligence tech as a trade secret and using it to drive R & D?

      • If you use a tool to make all the things that are useful to be made with it, and then sell those things, you are in effect selling access to the tool.

  • Lord

    I am not sure I see a real difference. The fastest route to greater intelligence would be to spawn multiple variants and evolve them and this would tend to create an ecosystem of evolving intelligences where each thought themselves the center of the universe until upset in some way by another, possibly of their own making.

  • Pingback: » Superintelligence Bayesian Investor Blog()

  • I agree with you, Robin. I expect a soft takeoff, though I admit that I think a software takeoff is likely to come sooner than an em takeoff (though obviously the two will overlap somewhat).

    A few weeks ago I wrote this section on Wikipedia to highlight various opinions on the topic: https://en.wikipedia.org/wiki/Seed_AI#Hard_vs._soft_takeoff Robin’s point closely resembles that of J. Storrs Hall.

    • Brian, I’d be interested in whether you think the reports described here are misleading regarding past rates of progress: http://www.overcomingbias.com/2012/08/ai-progress-estimate.html If not misleading, what accounts for your expectation of greatly increased future rates of progress?

      • Hi Robin 🙂 As we’ve discussed before, many expert opinions on human-level AI put its arrival within the century, so there’s at least a tension between their estimates and yours, without saying which one is right.

        AI could be accelerated 10x or 25x or whatever if a Cold War-type arms race breaks out, and AI is more incrementally useful, whereas WBE is most useful once all the pieces come together.

        But it’s less that I think bottom-up AI will come soon than that I expect WBE will be harder than it seems. As Tim Tyler and others have noted, most engineering achievements in the past have been bottom up (flight, steam engine, basically all modern computing). Certainly we take some inspiration from biology, but often the engineering leads the neuroscience (e.g., in reinforcement learning, deep neural networks, and many other areas). People first build the AI and then realize how it could happen in the brain. Of course, you might say WBE doesn’t require understanding the brain, but it seems that along the way, people will do a lot of neuroscience theory as well, and that will inspire bottom-up AI advances.

        I’d be curious to hear your thoughts on C. elegans uploading. Some people think it demonstrates how hard WBE is (whereas we can easily build artificial worms with vastly better abilities than C. elegans). On the other hand, if we did figure out C. elegans, it would give us everything at once, in contrast to incremental bottom-up AI.

        I’m not confident in my views. If I had to guess now, I’d say 2/3 chance of bottom-up AI first, 1/3 chance of WBE, conditional on human-level intelligence emerging.

  • Pingback: Overcoming Bias : Limits On Generality()

  • AlphaCeph

    “Better than the entire rest of the world put together. And my key question is: how could it plausibly do that?”

    “What I am I missing?”

    How can the world chess champion beat 10’s of thousands of people at chess? (http://en.wikipedia.org/wiki/Kasparov_versus_the_World)

    Presumably intelligence isn’t additive across agents. One billion IQ 60 mentally retarded people could not combine to make one IQ 150 genius. A cluster of slow CPUs is not as effective as one very fast CPU, and on some (hard to parallelize) tasks, a cluster of slow CPUs is as slow as one slow CPU.

    They have a rule in software development that “person hours” is a fallacy because one really good dev can achieve more progress than an infinity of poor quality of developers who would flounder in their own mess of mistakes and bugs.

    But I agree with you *very* strongly that this question deserves a lot more thought.

    • AlphaCeph
    • There are lots of contests where the best person in the world can beat ten thousand other people, as long as those other people don’t include #2-10 in the world. But if taking over the world were at stake, it would be more like serious contests in war or business, where a small project almost never beats the world.

      • AlphaCeph

        And what if #1 colluded with numbers 2 and 3 because they could see that they would do very nicely out of that… or if numbers 2-10 were a bit slow off the mark and not very future oriented, or their management teams dismissed the singularity out of hand (as most people do today).

        You are the one trying to argue for a ~90% chance of no single foom, so it damages your argument that these kinds of scenarios are plausible and might add up to a 50/50 chance.

  • Friendly-HI

    Hello Robin.

    I have a different conception of how intelligence goes foom. The fact that the human brain utilizes different modules for different tasks is a result of our evolutionary history and I assume you would strongly agree with that. Learning to speak is easy, learning to write and read is hard. Throwing things is easy, math is hard. It is not at all clear to me why a superintelligence necessarily needs a variety of different modules, at least not in the sense that I understand the word module (“specialized computation algorithm”).

    That we have a variety of software modules specifically designed for different tasks nowadays does not suggest to me that a superintelligence is basically just a collection of thousands of specialized modules, in fact I conceive of a true superintelligence as exactly the opposite: Namely the integration of various modules (or I would prefer to say “models of reality”) into a more unified and bigger picture to see and utilize all those connections. Conceptually I think a (nonfriendly) superintelligence “only” really needs to do but one thing really well, rather than a thousand different and completely unrelated things and tasks: What it must be good at is creating and improving upon models of reality. Software nowadays, say financial software that attempts to predict the stock market, is essentially modelling (or programmed to reflect) a very very thin slice of reality in an attempt to predict the future, and that model obviously is very primitive in that the software doesn’t have any understanding of what it is actually doing let alone how any of its algorithms relate to anything else in the world. A superintelligence would create and improve upon models much more akin to the way we humans model reality on different levels, but without the “modular” limitations of our peculiar evolutionary quirks and shortcomings and without the limitations of our lousy working memory and long term memory. Superintelligence as just a collection of highly specialized modules just seems unlikely to me, for a superintelligence to “understand” anything in the sense we humans understand things it would need to model reality in many different “fields” and on many different layers (perhaps even with a multitude of tools, instead of exclusively through math). Let’s look at how human intelligence works – what do really “intelligent” people and scientists actually do in the end: They try to discover and see and utilize the connections between models of reality, namely creating those models but also trying to discern where and how those models connect “vertically” (vertically as in different layers of models of the same things atoms – molecules – chemistry – biology – behavior – group dynamics) – and “horizontally” (say in all the ways in which politics and the economy are related). Now in my book a superintelligence is exactly doing that – building and improving upon models but without all the restrictions our human brains impose on us. A superintelligence would try to integrate separate models (or maybe modules if that’s how you want to think of it) into a more unified and bigger picture. I think this is the real game changer. When we were infants our model of reality was extremely primitive and in many ways oversimplified and/or flat-out wrong, but many years later here we are with many complex models of reality (many of which have only even started to exist in the last 200 years) – but all of them would be dwarfed instantly by a superintelligence that is able to model reality in a competent manner, without our limitations.

    Here is another thought: The more models you have and understand the easier new data and models based on this or new data are connected and integrated with what you already know… and in this very way the superintelligence I envision would be able to grow exponentially. Not because of any software or hardware improvements or necessarily even improvements to its own core code but simply because it gets better at constructing and refining models of reality – and the more models it creates from data, the more new or previously not understood data and models it can connect and integrate into what it already knows. So the exponential growth I conceive of is basically one that is based on the multitude and integration of models of reality this superintelligence can handle.

    Picture a smallish sphere for a second. Say the content of this sphere represents the starting point of a superintelligence after it is “unleashed”. The content/volume of the globe represents the few basic models of reality it starts out with, but it also contains a model and algorithms of how to create more and better models of reality, including the ability to competently improve upon these original models and also adding and integrating new models. Every time a new model is added (say a model of how biological evolution works, or how human courtship works etc) the globe expands and the volume increases so the more models are added and integrated the bigger this globe gets. Some models may be islands for a while because the intelligence doesn’t understand yet how this model relates to other models but eventually with new models a connection will emerge and a model that was previously isolated from the rest gets connected. Now even if the speed at which new data and models are added and integrated “at the borders” remains roughly constant, the volume of this globe still grows exponentially. The more models it incorporates the more new models become available and more connections between these models are discovered the quality of the models improves. Compare this to us: we humans have to use many different models and switch back and forth between them because we can’t take it all in and manipulate it all at once. We need to switch between our models like we need to switch the zoom level on a google earth map because we can’t possibly see let alone process our own house and the entire continent (at the same resolution) simultaneously, while a superintelligence conceivable would be able to.

    As an upcoming psychologist I have models of neurotransmitters in my head and I also have models of say group behavior in humans. I may read thousands of papers on these and other related topics but just how many of all the things that are explicitly and implicitly detailed in those papers are relevant to further an in-depth understanding about how neurotransmitters relate and influence group behavior, but are overseen or simply forgotten by me? I think a lot. I know these things are connected somehow in many many ways but in my mind I have to jump between different layers of my model of reality, seriously strain my memory and tediously try to coax out the connections and somehow I try to visualize it and I try to understand it through equations and I try to model it with other different tools and in different ways but it is damn slow and tedious and sometimes just nothing useful comes out of it. A superintelligence with the right hardware and software could process enormous amounts of scientific (and perhaps self-measured) data and integrate it into a much better and more accurate model than I or a group of scientists ever could. And moreover it might be able to understand all the connections between multiple layers of models – namely the dozens and hundreds and thousands of ways how certain neurotransmitters are influencing group dynamics in a way no human or group of humans could model let alone understand, let alone actually put to use, simply because we cannot model it all at the same time in our heads even if perhaps we could model it bit by bit over decades on paper or with computer-models.

    I left out considerations about the human friendliness problem obviously, but in this post I just wanted to detail how I conceive of the “intelligence” part of the superintelligence actually growing – and doing so exponentially. I think the bottleneck in this scenario will be the hardware limitations that allow this type of superintelligence to take its own measurements in an attempt to interpret its own data instead of relying on existing data and scientific papers that are of a very inconsistent quality.

    I talked a lot about models and different layers but let’s not forget that “reality” only has “one layer” in the sense that everything complex is really just composed of tiny particles interacting. Models for the superintelligence are really just a way to limit the amount of information it needs to process in order to predict outcomes, as it won’t be able to compute everything that’s “out there” in reality from the ground up by simulating all the individual particles that give rise to “higher level” phenomena like “a human being”. So all those models are really just tools to manage an unmanageable amount of information in a useful and competent manner, so it needs to be able to apply different models depending on which one is suseful and the superintelligence obviously needs to understand that to become competent at managing and applying those models. Sometimes when it cannot see the connection between how the interaction of particles leads to some higher level phenomenon maybe it could try to compute particle interactions locally and discover blank and not understood territory in out maps of reality and find the shape of the missing puzzle piece that we simply cannot.

    • Friendly-HI

      Brief: I never thought of a superintelligence as a toolbox of specialized modules that is frankensteind together. A superintelligence needs to be able to design those tools on its own and also use the right tools for the right job on its own. Maybe in the beginning we do need to give it a toolbox that does consist of many different modules, but its job is not to simply apply the tool/models we give it but rather to improve on them, invent new ones, integrate them and apply them. That to me is the core feature of a superintelligence, otherwise it’s neither intelligent nor super. The foom in this picture can be the result of it being able to create and manipulate and integrate a multitude of very complex models leading to ever more complex and intricate and detailed models, in turn leading to better and better predictions. What is growing exponentially in this instance is the width and depth and quality of its model of reality, which is the one core feature that for me defines a superintelligence.

    • Pedro Pinto

      Suppose there is already a AI on the WEB lurking, studding and evolving, How would it be possible to identify it? Can we (as humans) recognize or identify a reality (new), concept or “living-form” without “formal” auto-presentation or direct contact?

  • Pingback: Overcoming Bias : Regulating Infinity()

  • Houshalter

    Well a real world example to test these assumptions might be voice recognition. An AI domain that is mostly done by a few centralized businesses that specialize in it.

    And it’s also a domain that has in the past has had huge breakthroughs that massively advanced the state of the art.

  • Pedro Pinto

    Suppose there is already a AI on the WEB lurking, studding and evolving, How would it be possible to identify it? Can we (as humans) recognize or identify a reality (new), concept or “living-form” without “formal” auto-presentation or direct contact?

  • Pingback: Understanding AI risk. How Star Trek got talking computers right in 1966, while Her got it wrong in 2013. | Praxtime()

  • Pingback: Thoughts on Robots, AI, and Intelligence Explosion – Foundational Research()

  • Pingback: My take on recent debate over AI-risk | topherhallquist()

  • Pingback: Overcoming Bias : Youth Movements()

  • Pingback: Overcoming Bias : Unauthorized Topics()

  • Pingback: Overcoming Bias : Why Does Software Rot?()

  • Pingback: Positively shaping the development of artificial intelligence | 80,000 Hours – Rane’s Blog()

  • Pingback: Tool use and intelligence: A conversation – Foundational Research Institute()

  • Pingback: Overcoming Bias : Foom Justifies AI Risk Efforts Now()

  • Pingback: Overcoming Bias : Tegmark’s Book of Foom()

  • Pingback: Overcoming Bias : An Outside View of AI Control()

  • Pingback: Test Sam Eliezer Dialogue - Machine Intelligence Research Institute()