Overcoming Bias

Hard Takeoff

Continuation of:  Recursive Self-Improvement

Constant natural selection pressure, operating on the genes of the hominid line, produced improvement in brains over time that seems to have been, roughly, linear or accelerating; the operation of constant human brains on a pool of knowledge seems to have produced returns that are, very roughly, exponential or superexponential.  (Robin proposes that human progress is well-characterized as a series of exponential modes with diminishing doubling times.)

Recursive self-improvement - an AI rewriting its own cognitive algorithms - identifies the object level of the AI with a force acting on the metacognitive level; it "closes the loop" or "folds the graph in on itself".  E.g. the difference between returns on a constant investment in a bond, and reinvesting the returns into purchasing further bonds, is the difference between the equations y = f(t) = m*t, and dy/dt = f(y) = m*y whose solution is the compound interest exponential, y = e^(m*t).

When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM.  An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely - far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology.  Our present life is no good indicator of things to come.

Or to try and compress it down to a slogan that fits on a T-Shirt - not that I'm saying this is a good idea - "Moore's Law is exponential now; it would be really odd if it stayed exponential with the improving computers doing the research."  I'm not saying you literally get dy/dt = e^y that goes to infinity after finite time - and hardware improvement is in some ways the least interesting factor here - but should we really see the same curve we do now?

RSI is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor.  The advent of human intelligence was a discontinuity with the past even without RSI...

Whither Manufacturing?

Back in the 70s many folks thought they knew what the future of computing looked like: everyone sharing time-slices of a few huge computers.  After all, they saw that CPU cycles, the main computing cost, were cheaper on bigger machines.  This analysis, however, ignored large administrative overheads in dealing with shared machines.  People eagerly grabbed personal computers (PCs) to avoid those overheads, even though PC CPU cycles were more expensive. 

Similarly, people seem to make lots of assumptions when they refer to "full-scale nanotechnology."  This phrase seems to elicit images of fridge sized home appliances that, when plugged in and stocked with a few "toner cartridges", makes anything a CAD system can describe, and so quickly and cheaply that only the most price-sensitive folks would consider making stuff any other way.  It seems people learned too much from the PC case, thinking everything must become personal and local.  (Note computing is now getting less local.)  But there is no general law of increasingly local production.

Recursive Self-Improvement

Followup to:  Life's Story Continues, Surprised by Brains, Cascades, Cycles, Insight, Recursion, Magic, Engelbart: Insufficiently Recursive, Total Nano Domination

I think that at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability - "AI go FOOM".  Just to be clear on the claim, "fast" means on a timescale of weeks or hours rather than years or decades; and "FOOM" means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology (that it gets by e.g. ordering custom proteins over the Internet with 72-hour turnaround time).  Not, "ooh, it's a little Einstein but it doesn't have any robot hands, how cute".

Most people who object to this scenario, object to the "fast" part. Robin Hanson objected to the "local" part.  I'll try to handle both, though not all in one shot today.

We are setting forth to analyze the developmental velocity of an Artificial Intelligence.  We'll break down this velocity into optimization slope, optimization resources, and optimization efficiency.  We'll need to understand cascades, cycles, insight and recursion; and we'll stratify our recursive levels into the metacognitive, cognitive, metaknowledge, knowledge, and object level.

Quick review:

Disappointment in the Future

This seems worth posting around now...  As I've previously observed, futuristic visions are produced as entertainment, sold today and consumed today.  A TV station interviewing an economic or diplomatic pundit doesn't bother to show what that pundit predicted three years ago and how the predictions turned out.  Why would they?  Futurism Isn't About Prediction.

But someone on the ImmInst forum actually went and compiled a list of Ray Kurzweil's predictions in 1999 for the years 2000-2009.  We're not out of 2009 yet, but right now it's not looking good...

· Individuals primarily use portable computers
· Portable computers have dramatically become lighter and thinner
· Personal computers are available in a wide range of sizes and shapes, and are commonly embedded in clothing and jewelry, like wrist watches, rings, earrings and other body ornaments
· Computers with a high-resolution visual interface range from rings and pins and credit cards up to the size of a thin book. People typically have at least a dozen computers on and around their bodies, which are networked, using body LANS (local area networks)
· These computers monitor body functions, provide automated identity to conduct financial transactions and allow entry into secure areas. They also provide directions for navigation, and a variety of other services.
· Most portable computers do not have keyboards

Stuck In Throat

Let me try again to summarize Eliezer's position, as I understand it, and what about it seems hard to swallow.  I take Eliezer as saying: 

Sometime in the next few decades a human-level AI will probably be made by having a stupid AI make itself smarter.  Such a process starts very slow and quiet, but eventually "fooms" very fast and then loud. It is likely to go from much stupider to much smarter than humans in less than a week.  While stupid, it can be rather invisible to the world.  Once smart, it can suddenly and without warning take over the world. 

The reason an AI can foom so much faster than its society is that an AI can change its basic mental architecture, and humans can't.  How long any one AI takes to do this depends crucially on its initial architecture.  Current architectures are so bad that an AI starting with them would take an eternity to foom.  Success will come from hard math-like (and Bayes-net-like) thinking that produces deep insights giving much better architectures.

A much smarter than human AI is basically impossible to contain or control; if it wants to it will take over the world, and then it will achieve whatever ends it has.  One should have little confidence that one knows what those ends are from its behavior as a much less than human AI (e.g., as part of some evolutionary competition).  Unless you have carefully proven that it wants what you think it wants, you have no idea what it wants. 

In such a situation, if one cannot prevent AI attempts by all others, then the only reasonable strategy is to try to be the first with a "friendly" AI, i.e., one where you really do know what it wants, and where what it wants is something carefully chosen to be as reasonable as possible. 

I don't disagree with this last paragraph.  But I do have trouble swallowing prior ones.  The hardest to believe I think is that the AI will get smart so very rapidly, with a growth rate (e.g., doubling in an hour) so far out of proportion to prior growth rates, to what prior trends would suggest, and to what most other AI researchers I've talked to think.  The key issues come from this timescale being so much shorter than team lead times and reaction times.  This is the key point on which I await Eliezer's more detailed arguments. 

Since I do accept that architectures can influence growth rates, I must also have trouble believing humans could find new AI architectures anytime soon that make this much difference.  Some other doubts: 

• Does a single "smarts" parameter really summarize most of the capability of diverse AIs?
• Could an AI's creators see what it wants by slowing down its growth as it approaches human level?
• Might faster brain emulations find it easier to track and manage an AI foom?

Singletons Rule OK

Reply to:  Total Tech Wars

How does one end up with a persistent disagreement between two rationalist-wannabes who are both aware of Aumann's Agreement Theorem and its implications?

Such a case is likely to turn around two axes: object-level incredulity ("no matter what AAT says, proposition X can't really be true") and meta-level distrust ("they're trying to be rational despite their emotional commitment, but are they really capable of that?").

So far, Robin and I have focused on the object level in trying to hash out our disagreement.  Technically, I can't speak for Robin; but at least in my own case, I've acted thus because I anticipate that a meta-level argument about trustworthiness wouldn't lead anywhere interesting.  Behind the scenes, I'm doing what I can to make sure my brain is actually capable of updating, and presumably Robin is doing the same.

(The linchpin of my own current effort in this area is to tell myself that I ought to be learning something while having this conversation, and that I shouldn't miss any scrap of original thought in it - the Incremental Update technique. Because I can genuinely believe that a conversation like this should produce new thoughts, I can turn that feeling into genuine attentiveness.)

Yesterday, Robin inveighed hard against what he called "total tech wars", and what I call "winner-take-all" scenarios:

Robin:  "If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury."

Robin and I both have emotional commitments and we both acknowledge the danger of that.  There's nothing irrational about feeling, per se; only failure to update is blameworthy.  But Robin seems to be very strongly against winner-take-all technological scenarios, and I don't understand why.

Among other things, I would like to ask if Robin has a Line of Retreat set up here - if, regardless of how he estimates the probabilities, he can visualize what he would do if a winner-take-all scenario were true.

Total Tech Wars

Eliezer Thursday:

Suppose ... the first state to develop working researchers-on-a-chip, only has a one-day lead time. ...  If there's already full-scale nanotechnology around when this happens ... in an hour ... the ems may be able to upgrade themselves to a hundred thousand times human speed, ... and in another hour, ...  get the factor up to a million times human speed, and start working on intelligence enhancement. ... One could, of course, voluntarily publish the improved-upload protocols to the world, and give everyone else a chance to join in.  But you'd have to trust that not a single one of your partners were holding back a trick that lets them run uploads at ten times your own maximum speed. 

Carl Shulman Saturday and Monday: 

I very much doubt that any U.S. or Chinese President who understood the issues would fail to nationalize a for-profit firm under those circumstances. ... It's also how a bunch of social democrats, or libertarians, or utilitarians, might run a project, knowing that a very likely alternative is the crack of a future dawn and burning the cosmic commons, with a lot of inequality in access to the future, and perhaps worse. Any state with a lead on bot development that can ensure the bot population is made up of nationalists or ideologues (who could monitor each other) could disarm the world's dictatorships, solve collective action problems ... [For] biological humans [to] retain their wealth as capital-holders in his scenario, ems must be obedient and controllable enough ... But if such control is feasible, then a controlled em population being used to aggressively create a global singleton is also feasible. 

Every new technology brings social disruption. While new techs (broadly conceived) tend to increase the total pie, some folks gain more than others, and some even lose overall.  The tech's inventors may gain intellectual property, it may fit better with some forms of capital than others, and those who first foresee its implications may profit from compatible investments.  So any new tech can be framed as a conflict, between opponents in a race or war.

Every conflict can be framed as a total war.  If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury.  All resources must be devoted to growing more resources and to fighting them in every possible way. 

Dreams of Autarky

Selections from my 1999 essay "Dreams of Autarky":

[Here is] an important common bias on "our" side, i.e., among those who expect specific very large changes. ... Futurists tend to expect an unrealistic degree of autarky, or independence, within future technological and social systems.  The cells in our bodies are largely-autonomous devices and manufacturing plants, producing most of what they need internally. ... Small tribes themselves were quite autonomous. ... Most people are not very aware of, and so have not fully to terms with their new inter-dependence.  For example, people are surprisingly willing to restrict trade between nations, not realizing how much their wealth depends on such trade. ... Futurists commonly neglect this interdependence ... they picture their future political and economic unit to be the largely self-sufficient small tribe of our evolutionary heritage.  ... [Here are] some examples. ...

Total Nano Domination

Followup to:  Engelbart: Insufficiently Recursive

The computer revolution had cascades and insights aplenty.  Computer tools are routinely used to create tools, from using a C compiler to write a Python interpreter, to using theorem-proving software to help design computer chips.  I would not yet rate computers as being very deeply recursive - I don't think they've improved our own thinking processes even so much as the Scientific Revolution - yet.  But some of the ways that computers are used to improve computers, verge on being repeatable (cyclic).

Yet no individual, no localized group, nor even country, managed to get a sustained advantage in computing power, compound the interest on cascades, and take over the world.  There was never a Manhattan moment when a computing advantage temporarily gave one country a supreme military advantage, like the US and its atomic bombs for that brief instant at the end of WW2.  In computing there was no equivalent of "We've just crossed the sharp threshold of criticality, and now our pile doubles its neutron output every two minutes, so we can produce lots of plutonium and you can't."

Will the development of nanotechnology go the same way as computers - a smooth, steady developmental curve spread across many countries, no one project taking into itself a substantial fraction of the world's whole progress?  Will it be more like the Manhattan Project, one country gaining a (temporary?) huge advantage at huge cost?  Or could a small group with an initial advantage cascade and outrun the world?

Engelbart: Insufficiently Recursive

Followup to:  Cascades, Cycles, Insight, Recursion, Magic
Reply to:  Engelbart As Ubertool?

When Robin originally suggested that Douglas Engelbart, best known as the inventor of the computer mouse, would have been a good candidate for taking over the world via compound interest on tools that make tools, my initial reaction was "What on Earth?  With a mouse?"

On reading the initial portions of Engelbart's "Augmenting Human Intellect: A Conceptual Framework", it became a lot clearer where Robin was coming from.

Sometimes it's hard to see through the eyes of the past.  Engelbart was a computer pioneer, and in the days when all these things were just getting started, he had a vision of using computers to systematically augment human intelligence.  That was what he thought computers were for.  That was the ideology lurking behind the mouse.  Something that makes its users smarter - now that sounds a bit more plausible as an UberTool.

Looking back at Engelbart's plans with benefit of hindsight, I see two major factors that stand out:

1. Engelbart committed the Classic Mistake of AI: underestimating how much cognitive work gets done by hidden algorithms running beneath the surface of introspection, and overestimating what you can do by fiddling with the visible control levers.
2. Engelbart anchored on the way that someone as intelligent as Engelbart would use computers, but there was only one of him - and due to point 1 above, he couldn't use computers to make other people as smart as him.