43 Comments

Huge light saber-I'm wheezingBish imagine him being gay then a botiful man will appear kyak:Me can watch patiently yah knowB.866J.CO/Y5560SR

Expand full comment

Software has incorporated self-improvement almost since the concept of software was invented, though perhaps it could be claimed that it doesn't yet *want" anything.

Expand full comment

What I never see when I read things about the singularity is who programmed the computer to *want* to improve itself. All computers now do ONLY what you tell them to do. If you do not tell the computer to reprogram itself it will not do it.

Having said that, I am certain that someone is right now trying to insert the desire to improve functioning into software and that in the end the world is fucked.

Expand full comment

Re: point 2, I can give you my interpretation of Robin's argument -- whether he would agree with that interpretation is another matter.

Borrowing I. J. Good's example, we can imagine somebody at the dawn of the industrial era defining an "ultraproductive factory" as "a factory that can produce any economic good". Since ultraproductive factories are themselves entirely made from economic goods, an ultraproductive factory can be used to produce another ultraproductive factory, which together can produce two more, then those four can produce eight, then sixteen, and so on. Clearly, whichever industrialist builds the first ultraproductive factory will quickly explode in capability to take over the world.

What is the problem with this argument? After all, given the definition of an ultraproductive factory, it's tautologically correct. (And exponential growth in productivity is a real phenomenon!) The main problem, surely, is in assuming the relevance of the definition given -- of taking 'fully general factories' as even remotely resembling how things are produced in an industrial economy.

It's probably theoretically possible to create a factory to fabricate everything by yourself. Every time you need an intermediate product, you either create it from raw materials, or you determine what products you need in order to create it, and create those by recursively following the same process. The main impediment is that this is totally utterly uncompetitive with an industrial economy in which all the factories (production mechanisms more generally) are specialised to making a small subset of everything rather than everything.

In a more realistic paradigm, yes something is recursively self-improving, but that something is "the world as a whole" and not any individual entity.

Returning to intelligence rather than factories, whether the ultra-X model is any more relevant here depends on whether intelligence is an enormously complex system or a fairly simple process. For example, you've perhaps read Scott Alexander's flurry of recent posts on Predictive Processing, a model of the human brain which Scott seems to believe is both correct and all-encompassing, in that it provides a simple general implementation of intelligence that requires almost no context-specific detail or optimisations. I would say that if he's right, recursive self-improvement quite probably can be localised to a single AI project, and all the FOOM-related fears are totally correct.

For the Hansonian model of AI progress to be right, this and all other attempts at a simple fully general learning machine must be mistaken: AI must turn out to be an amalgamation of many systems and subsystems and specialisations and little improvements and optimisations, not a single algorithm that can be described in full detail in just a few lines of code.

Expand full comment

On point 2, I can describe my interpretation of Robin's argument -- whether he would agree with that interpretation is another matter.

Following I.J. Good's example, we could imagine someone at the dawn of the industrial era defining an "ultraproductive factory" as "a factory that can produce any economic good". Since ultraproductive factories are themselves made entirely from economic goods, an ultraproductive factory can therefore be used to produce a second ultraproductive factory, which together can produce two more, then those four can produce eight, then sixteen, and so on. Clearly, whichever industrialist builds the first ultraproductive factory will quickly explode in capability to take over the world.

What is the problem with this argument? After all, given the definition of an ultraproductive factory, it's tautologically correct. The main problem, surely, is in assuming the relevance of this definition -- of taking 'fully general factories' as even remotely resembling how things are produced in an industrial economy.

It's probably theoretically possible to create a factory in which you create everything you need. Every time you need an intermediate product, you either create it from raw materials, or you determine what products you need to create to create it, and create those by recursively following the same process. The main impediment is that this is utterly totally uncompetitive with an industrial economy in which all the factories (or production mechanisms more generally) are specialised to making a small subset of everything rather than everything.

Under this more realistic paradigm, yes something is recursively self-improving, but that something is "the world as a whole" and not any individual entity.

Returning to intelligence rather than factories, whether the ultra-X model is any more relevant here depends on whether intelligence is an enormously complex system or a fairly simple process. For example, you've probably seen Scott Alexander's flurry of recent posts on Predictive Processing, a model of the human brain which Scott seems to think is both correct and all-encompassing, in that it provides a simple general implementation of intelligence that doesn't require much (or any?) context-specific detail. I would say that if he's right, recursive self-improvement quite probably can be localised to a single AI project, and all the FOOM-related fears are absolutely correct.

For the Hansonian model of AI progress to be right, this and all other attempts at a simple fully general learning process must be mistaken: AI must turn out to be an amalgamation of many systems and subsystems and specialisations and little improvements and optimisations, not a single algorithm that can be described in full detail in just a few lines of code.

Expand full comment

@John Smith: Maybe I'm still misunderstanding Robin's position, but if I'm not:

1 - Robin thinks we are very far from building an AGI.

He thinks recent progress in AI is limited to narrow special purpose systems for which they've been extensively trained (AlphaGo, image classifiers, etc.), and that we're not yet close to building general purpose AIs.

And he thinks that if/when we eventually get close to building an AGI, progress will not come suddenly with a breakthru but gradually, in fits and starts, with strength in some areas matched by severe weakness in other areas. This will allow multiple teams to catch up and compete, and permit people to learn how to control goals over an extended period of time. Therefore this won't lead to a runaway AI explosion and a singleton.

Which, if correct, means the worry about foom is, at best, very premature.

2 - He doesn't buy the recursive self-improvement argument.

This point I don't fully understand.

I find it very plausible that eventually (even if hundreds of years from now), an AI system will be able to, minimally, improve its own code (if only to make it faster).

And that will lead to a series of ever-more-powerful software versions. Whether these are "smarter" in an IQ or G sense or merely faster isn't important. (Per the arguments in Age of Em, a sufficiently fast ordinary intelligence is functionally equivalent to a "smarter" intelligence).

Eventually such an AI will be capable of earning money, and so (if allowed to by one of the many teams developing them) funding its own further expansion - hardware and resources as needed.

Recurse.

Exactly which part of this argument Robin disagrees with (if any), I don't understand.

(I hope Robin will correct my misunderstandings.)

Expand full comment

Sure it can, since it is universal it would just depends on the capacity of computation and memory available at the time of implementation, which is also universal.

Logic is: What limits human creativity and intelligence from growing our knowledge-stack faster is capacity. 7 billion people can create a lot more knowledge than 1 person. In the same way, whatever the capacity we have for computers when someone actually builds the sort of algorithm described, will determine the size of the effect.

If it happened today it would merely be a whimper, as even the strongest supercomputers aren't as fast as 1 human brain. But if Moore's law continues and we get to 2040s and a personal computer has equivalent capacity to all human beings combined, then such a knowledge-creating algorithm would mean so much more knowledge-creating that it in theory could get a head-start if it kept its findings a secret. (But not in a qualitatively different way than a group of human beings can do today)

My guess is, this algorithm will be built long before that time, it doesn't seem that hard. Maybe even already is built, since it wouldn't be impressive at all at current computational powers.

If I am correct, the way generalized AI will come into existence is a very calm one. It will at first be just a creativity-tool or decision-making tool, like adding a few more humans. And growing in use together with the growth of computation and memory over decades, being immersed in our knowledge and culture, and generally just being one of us. I would also guess that with the general trend of our fusing with technology via smartphones and stuff (not to mention some VR / neuralink scenario), we would during this time simply integrate these tools gradually and instead of Ems becoming something separate, we would be (one could argue we already are, and possibly remain) hybrids between Ems and human beings. And see the distinction between our virtual self and flesh-self as nothing more than a quaint perspective.

Expand full comment

If you already know and can describe a general intelligence algorithm, it can't be what is discovered by one team that allows it's computer to grow much much faster than all the others.

Expand full comment

Regarding the generalizability of intelligence.

David Deutsch's theory of people as "universal explainers" describe part of our intelligence (the interesting part) as being universal in the sense of being able to explain anything.

This does make sense to me. Yes, learning a specific piece of knowledge in geography doesn't translate to knowing knowledge in chemistry. But it is true that the same intelligence that both can discover, understand and explain geography also can explain chemistry. There is clearly some generalizability of human intelligence.

And it doesn't have to be any harder than this, if I were to conceptualize a simple algorithm structure.

Problem: Any problemWhat we have: A processor, memory and some mechanism of material transformation

Step 1: Creativity1. Random generation within constraints (relaxation: goals, prior beliefs)2. Selective retaining of randomly generated ideas based on algorithms for basic sense-making

Step 2: Critical thinking3. Critique new idea based on prior knowledge, remove those that doesn't4. Get critique from other external sources of knowledge

Step 3: Trial and error5. Test qualified idea, log whether it worked to update knowledge in a bayesian way6. Repeat until it works and new knowledge is created

To me this sort of method seems like what the human mind must be running. When I encounter something I haven't seen before, like the other day with a door without a handle, I seem to be running through something like this iteratively until I find a solution.

This seem to be to be the same mechanism I also run for figuring out a software problem, or an economics problem, or a construction problem, or a resource collection problem. The only thing I need is more knowledge in order to solve any problem. And the only thing I need to be able to generate solutions to any problem, is an algorithm like this, a processor, memory and some sort of method for material transformation.

Expand full comment

The assumption is common enough that I'd expect an author whose argument relied on disagreeing with it to mention that fact.

Expand full comment

That is a huge assumption. Seems the assumption should be made explicit when describing the multipolar scenario then.

Do you know if your assumption is widely shared in AI risk circles?

I myself have low credence in that assumption, mostly because of a general assymmetry: there are more ways to harm/destroy complex systems like human organisms than there are ways to prevent harm/destruction to them.

Expand full comment

As I'm pretty sure you can guess, I assume the usual case in history where one small part of the world cannot destroy the whole world

Expand full comment

Robin Hanson: "in a multipolar world, each superinteligence has only a small fraction of the world’s power. In that context there is little global risk from any one of them getting out of control of its creator or owner. The rest of the world limits its damage."

Is that a *definition* of a multipolar world? If not then what reason is there to think that a scenario where multiple superhuman AI grow in power in lockstep will not lead to a total destruction outcome? Is that based on belief 1 and/or 2 below or something else?

(1) for any total destruction method that one superhuman AI among others can design: other superhuman AIs will prevent the use of the method from having a total destruction outcome.

(2) for any method a superhuman AI among others can design that would destroy all competing AIs (but not itself): other superhuman AIs will have the same method *and* will reliably detect and reciprocate use of the method (MAD scenario) *and* no superhuman AI will choose total destruction via MAD.

Expand full comment

Given that Tegmark expressly rejects the estimation of likelihoods (or am I interpreting him correctly?), it particularly lacks sense to read off likelihood estimates based on pages devoted to specific subjects.

Expand full comment

I guess that makes sense if you're trying to criticize how the book was written, as opposed to trying to allay the author's actual concerns (and those of people like him). In other words, he may have just written more about foom because it's more fun to write about or he could think of more things to say about it or he expected the other concerns to be more obvious to his readers, not because he believes it's the only important concern.

Expand full comment

I don't have the book at the moment to review the Libertarian Utopia section, and the Descendants section seems mostly about value deception, not drift. But even if both property loss and value drift had been mentioned somewhere in the book, it seems crazy to describe the book as giving equal emphasis to those three considerations.

Expand full comment