Tag Archives: AI

Stuck In Throat

By Robin Hanson · November 30, 2008 8:15 pm · Comments (45)

Let me try again to summarize Eliezer’s position, as I understand it, and what about it seems hard to swallow.  I take Eliezer as saying: 

Sometime in the next few decades a human-level AI will probably be made by having a stupid AI make itself smarter.  Such a process starts very slow and quiet, but eventually "fooms" very fast and then loud. It is likely to go from much stupider to much smarter than humans in less than a week.  While stupid, it can be rather invisible to the world.  Once smart, it can suddenly and without warning take over the world. 

The reason an AI can foom so much faster than its society is that an AI can change its basic mental architecture, and humans can’t.  How long any one AI takes to do this depends crucially on its initial architecture.  Current architectures are so bad that an AI starting with them would take an eternity to foom.  Success will come from hard math-like (and Bayes-net-like) thinking that produces deep insights giving much better architectures.

A much smarter than human AI is basically impossible to contain or control; if it wants to it will take over the world, and then it will achieve whatever ends it has.  One should have little confidence that one knows what those ends are from its behavior as a much less than human AI (e.g., as part of some evolutionary competition).  Unless you have carefully proven that it wants what you think it wants, you have no idea what it wants. 

In such a situation, if one cannot prevent AI attempts by all others, then the only reasonable strategy is to try to be the first with a "friendly" AI, i.e., one where you really do know what it wants, and where what it wants is something carefully chosen to be as reasonable as possible. 

I don’t disagree with this last paragraph.  But I do have trouble swallowing prior ones.  The hardest to believe I think is that the AI will get smart so very rapidly, with a growth rate (e.g., doubling in an hour) so far out of proportion to prior growth rates, to what prior trends would suggest, and to what most other AI researchers I’ve talked to think.  The key issues come from this timescale being so much shorter than team lead times and reaction times.  This is the key point on which I await Eliezer’s more detailed arguments. 

Since I do accept that architectures can influence growth rates, I must also have trouble believing humans could find new AI architectures anytime soon that make this much difference.  Some other doubts: 

  • Does a single "smarts" parameter really summarize most of the capability of diverse AIs?
  • Could an AI’s creators see what it wants by slowing down its growth as it approaches human level?
  • Might faster brain emulations find it easier to track and manage an AI foom?
GD Star Rating
loading...
Tagged as: ,

Total Tech Wars

By Robin Hanson · November 29, 2008 5:45 pm · Comments (25)

Eliezer Thursday:

Suppose … the first state to develop working researchers-on-a-chip, only has a one-day lead time. …  If there’s already full-scale nanotechnology around when this happens … in an hour … the ems may be able to upgrade themselves to a hundred thousand times human speed, … and in another hour, …  get the factor up to a million times human speed, and start working on intelligence enhancement. … One could, of course, voluntarily publish the improved-upload protocols to the world, and give everyone else a chance to join in.  But you’d have to trust that not a single one of your partners were holding back a trick that lets them run uploads at ten times your own maximum speed.

Carl Shulman Saturday and Monday:

I very much doubt that any U.S. or Chinese President who understood the issues would fail to nationalize a for-profit firm under those circumstances. … It’s also how a bunch of social democrats, or libertarians, or utilitarians, might run a project, knowing that a very likely alternative is the crack of a future dawn and burning the cosmic commons, with a lot of inequality in access to the future, and perhaps worse. Any state with a lead on bot development that can ensure the bot population is made up of nationalists or ideologues (who could monitor each other) could disarm the world’s dictatorships, solve collective action problems … [For] biological humans [to] retain their wealth as capital-holders in his scenario, ems must be obedient and controllable enough … But if such control is feasible, then a controlled em population being used to aggressively create a global singleton is also feasible.

Every new technology brings social disruption. While new techs (broadly conceived) tend to increase the total pie, some folks gain more than others, and some even lose overall.  The tech’s inventors may gain intellectual property, it may fit better with some forms of capital than others, and those who first foresee its implications may profit from compatible investments.  So any new tech can be framed as a conflict, between opponents in a race or war.

Every conflict can be framed as a total war. If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury.  All resources must be devoted to growing more resources and to fighting them in every possible way.

Continue reading "Total Tech Wars" »

GD Star Rating
loading...
Tagged as: , , , ,

Billion Dollar Bots

By James Miller · November 22, 2008 11:30 pm · Comments (15)

Robin presented a scenario in which whole brain emulations, or what he calls bots come into being.  Here is another:

Bots are created with hardware and software.  The higher the quality of one input the less you need of the other.  Hardware, especially with cloud computing, can be quickly allocated from one task to another.  So the first bot might run on hardware worth billions of dollars.

The first bot creators would receive tremendous prestige and a guaranteed place in the history books.  So once it becomes possible to create a bot many firms and rich individuals will be willing to create one even if doing so would cause them to suffer a large loss.

Imagine that some group has $300 million to spend on hardware and will use the money as soon as $300 million becomes enough to create a bot.  The best way to spend this money would not be to buy a $300 million computer but to rent $300 million of off-peak computing power.  If the group needed only 1,000 hours of computing power (which it need not buy all at once) to prove that it had created a bot then the group could have, roughly, $3 billion of hardware for the needed 1,000 hours.

It’s likely that the  first bot would run very slowly.  Perhaps it would take the bot 10 real seconds to think as much as a human does in one second.

Under my scenario the first bot would be wildly expensive.  But because of Moore’s law once the first bot was created everyone would expect that the cost of bots would eventually become low enough so that they would radically remake society.

Consequently, years before bots come to dominate the economy, many people will come to expect that within their lifetime bots will someday come to dominate the economy.   Bot expectations will radically change the world.

I suspect that after it becomes obvious that we could eventually create cheap bots world governments will devote trillions to bot Manhattan projects.  The expected benefits of winning the bot race will be so high that it would be in the self-interest of individual governments to not worry too much about bot friendliness.

The U.S. and Chinese militaries  might fall into a bot prisoners’ dilemma in which both militaries would prefer an outcome in which everyone slowed down bot development to ensure friendliness yet both nations were individually better off (regardless of what the other military did) taking huge chances on friendliness so as to increase the probability of their winning the bot race.

My hope is that the U.S. will have such a tremendous advantage over China that the Chinese don’t try to win the race and the U.S. military thinks it can afford to go slow.  But given China’s relatively high growth rate I doubt humanity will luck into this safe scenario.

GD Star Rating
loading...
Tagged as: ,

Emulations Go Foom

By Robin Hanson · November 22, 2008 1:15 am · Comments (74)

Let me consider the AI-foom issue by painting a (looong) picture of the AI scenario I understand best, whole brain emulations, which I’ll call “bots.”  Here goes.

When investors anticipate that a bot may be feasible soon, they will estimate their chances of creating bots of different levels of quality and cost, as a function of the date, funding, and strategy of their project.  A bot more expensive than any (speedup-adjusted) human wage is of little direct value, but exclusive rights to make a bot costing below most human wages would be worth many trillions of dollars.

It may well be socially cost-effective to start a bot-building project with a 1% chance of success when its cost falls to the trillion dollar level.  But not only would successful investors probably only gain a small fraction of this net social value, is unlikely any investor group able to direct a trillion could be convinced the project was feasible – there are just too many smart-looking idiots making crazy claims around.

But when the cost to try a 1% project fell below a billion dollars, dozens of groups would no doubt take a shot.  Even if they expected the first feasible bots to be very expensive, they might hope to bring that cost down quickly.  Even if copycats would likely profit more than they, such an enormous prize would still be very tempting.

The first priority for a bot project would be to create as much emulation fidelity as affordable, to achieve a functioning emulation, i.e., one you could talk to and so on.  Few investments today are allowed a decade of red ink, and so most bot projects would fail within a decade, their corpses warning others about what not to try.  Eventually, however, a project would succeed in making an emulation that is clearly sane and cooperative.

Continue reading "Emulations Go Foom" »

GD Star Rating
loading...
Tagged as: , ,

AI Go Foom

By Robin Hanson · November 19, 2008 11:00 pm · Comments (89)

It seems to me that it is up to [Eliezer] to show us how his analysis, using his abstractions, convinces him that, more likely than it might otherwise seem, hand-coded AI will come soon and in the form of a single suddenly super-powerful AI.

As this didn’t prod a response, I guess it is up to me to summarize Eliezer’s argument as best I can, so I can then respond.  Here goes:

A machine intelligence can directly rewrite its entire source code, and redesign its entire physical hardware.  While human brains can in principle modify themselves arbitrarily, in practice our limited understanding of ourselves means we mainly only change ourselves by thinking new thoughts.   All else equal this means that machine brains have an advantage in improving themselves. 

A mind without arbitrary capacity limits, that focuses on improving itself, can probably do so indefinitely.  The growth rate of its "intelligence" may be slow when it is dumb, but gets faster as it gets smarter.  This growth rate also depends on how many parts of itself it can usefully change.  So all else equal, the growth rate of a machine intelligence must be greater than the growth rate of a human brain. 

No matter what its initial disadvantage, a system with a faster growth rate eventually wins.  So if the growth rate advantage is large enough then yes a single computer could well go in a few days from less than human intelligence to so smart it could take over the world.  QED.

So Eliezer, is this close enough to be worth my response?  If not, could you suggest something closer?

GD Star Rating
loading...
Tagged as:

Setting The Stage

By Robin Hanson · November 18, 2008 11:30 pm · Comments (14)

As Eliezer and I begin to explore our differing views on singularity, perhaps I should summarize my current state of mind.   

We seem to agree that:

  1. Machine intelligence would be a development of almost unprecedented impact and risk, well worth considering now.
  2. Feasible approaches include direct hand-coding, based on a few big and lots of little insights, and emulations of real human brains. 
  3. Machine intelligence will more likely than not appear with a century, even if the progress rate to date does not strongly suggest the next few decades. 
  4. Many people say silly things here, and we do better to ignore them than to try to believe the opposite. 
  5. Math and deep insights (especially probability) can be powerful relative to trend-fitting and crude analogies. 
  6. Long term historical trends are suggestive of future events, but not strongly so.
  7. Some should be thinking about how to create "friendly" machine intelligences. 

We seem to disagree modestly about the relative chances of the emulation and direct-coding approaches; I think the first and he thinks the second is more likely to succeed first.  Our largest disagreement seems to be on the chances that a single hand-coded version will suddenly and without warning change from nearly powerless to overwhelmingly powerful; I’d put it as less than 1% and he seems to put it as over 10%. 

Continue reading "Setting The Stage" »

GD Star Rating
loading...
Tagged as: ,

Friendliness Factors

By Robin Hanson · November 16, 2008 12:15 pm · Comments (18)

Imagine several firms competing to make the next generation of some product, like a lawn mower or cell phone.  What factors influence variance in their product quality (relative to cost)?  That is, how much better will the best firm be relative to the average, second best, or worst?   Larger variance factors should make competitors worry more that this round of competition will be their last.  Here are a few factors:

  1. Resource Variance – the more competitors vary in resources, the more performance varies.
  2. Cumulative Advantage - the more prior wins help one win again, the more resources vary.
  3. Grab It First – If the cost to grab and defend a resource is much less than its value, the first to grab can gain a further advantage.
  4. Competitor Count – with more competitors, the best exceeds the second best less, but exceeds the average more. 
  5. Competitor Effort – the longer competitors work before their performance is scored, or the more resources they spend, the more scores vary.
  6. Lumpy Design – the more quality depends on a few crucial choices, relative to many small choices, the more quality varies.
  7. Interdependence – When firms need inputs from each other, winner gains are also supplier gains, reducing variance.   
  8. Info Leaks – the more info competitors can gain about others’ efforts, the more the best will be copied, reducing variance.
  9. Shared Standards – competitors sharing more standards and design features, in info, process, or product, can better understand and use info leaks. 
  10. Legal Barriers - may prevent competitors from sharing standards, info, inputs.
  11. Anti-Trust -  Social coordination may prevent too much winning by a few.
  12. Sharing Deals - If firms own big shares in each other, or form a coop, or just share values, may mind less if others win.  Lets tolerate more variance, but also share more info.
  13. Niche Density – When each competitor can adapt to a different niche, they may all survive.
  14. Quality Sensitivity – demand/success may be very sensitive, or not very sensitive, to quality.
  15. Network Effects – Users may prefer to use the same product regardless of its quality.
  16. [What factors am I missing?  Tell me and I'll extend the list.]

Some key innovations in history were associated with very high variance in competitor success.  For example, our form of life seems to have eliminated all trace of any other forms on Earth.  On the other hand, farming and industry innovations were associated with much less variance.  I attribute this mainly to info becoming much leakier, in part due to more shared standards, which seems to bode well for our future. 

If you worry that one competitor will severely dominate all others in the next really big innovation, forcing you to worry about its "friendliness," you should want to promote factors that reduce success variance.  (Though if you cared mainly about the winning performance level, you’d want more variance.)

GD Star Rating
loading...
Tagged as:

Friendly Teams

By Robin Hanson · November 15, 2008 12:15 pm · Comments (18)

Wednesday I described UberTool, an imaginary firm planning to push a set of tools through a rapid mutual-improvement burst until they were in a position to basically “take over the world.”  I asked when such a plan could be reasonable.

Thursday I noted that Doug Engelbart understood in ’62 that computers were the most powerful invention of his century, and could enable especially-mutually-improving tools.  He understood lots of detail about what those tools would look like long before others, and oversaw a skilled team focused on his tools-improving-tools plan.  That team pioneered graphic user interfaces and networked computers, and in ’68 introduced the world to the mouse, videoconferencing, email, and the web.

I asked if this wasn’t ideal for an UberTool scenario, where a small part of an old growth mode “takes over” most of the world via having a head start on a new faster growth mode.  Just as humans displaced chimps, farmers displaced hunters, and industry displaced farming, would a group with this much of a head start on such a general better tech have a decent shot at displacing industry folks?  And if so, shouldn’t the rest of the world have worried about how “friendly” they were?

In fact, while Engelbart’s ideas had important legacies, his team didn’t come remotely close to displacing much of anything.  He lost most of his funding in the early 1970s, and his team dispersed.  Even though Engelbart understood key elements of tools that today greatly improve team productivity, his team’s tools did not seem to have enabled them to be radically productive, even at the task of improving their tools.

It is not so much that Engelbart missed a few key insights about what computer productivity tools would look like.  I doubt if it would have made much difference had he traveled in time to see a demo of modern tools.  The point is that most tools require lots more than a few key insights to be effective – they also require thousands of small insights that usually accumulate from a large community of tool builders and users.

Small teams have at times suddenly acquired disproportionate power, and I’m sure their associates who anticipated this possibility used the usual human ways to consider that team’s “friendliness.”  But I can’t recall a time when such sudden small team power came from an UberTool scenario of rapidly mutually improving tools.

Some say we should worry that a small team of AI minds, or even a single mind, will find a way to rapidly improve themselves and take over the world.  But what makes that scenario reasonable if the UberTool scenario is not?

GD Star Rating
loading...
Tagged as:

Engelbart As UberTool?

By Robin Hanson · November 13, 2008 6:00 am · Comments (16)

Yesterday I described UberTool, an imaginary company planning to push a set of tools through a mutual-improvement process; their team would improve those tools, and then use those improved versions to improve them further, and so on through a rapid burst until they were in a position to basically "take over the world."  I asked what it would take to convince you their plan was reasonable, and got lots of thoughtful answers.

Douglas Engelbart is the person I know who came closest to enacting such a UberTool plan.  His seminal 1962 paper, "Augmenting Human Intellect: A Conceptual Framework" proposed using computers to create such a rapidly improving tool set.  He understood not just that computer tools were especially open to mutual improvement, but also a lot about what those tools would look like.  Wikipedia:

[Engelbart] is best known for inventing the computer mouse; as a pioneer of human-computer interaction whose team developed hypertext, networked computers, and precursors to graphical user interfaces.

Doug led a team who developed a rich set of tools including a working hypertext publishing system.  His 1968 "Mother of all Demos" to 1000 computer professionals in San Francisco 

featured the first computer mouse the public had ever seen, as well as introducing interactive text, video conferencing, teleconferencing, email and hypertext [= the web].

Now to his credit, Doug never suggested that his team, even if better funded, might advance so far so fast as to "take over the world."  But he did think it could go far (his Bootstrap Institute still pursues his vision), and it is worth pondering just how far it was reasonable to expect Doug’s group could go.   

To review, soon after the most powerful invention of his century appeared, Doug Engelbart understood what few others did — not just that computers could enable fantastic especially-mutually-improving tools, but lots of detail about what those tools would look like.  Doug correctly saw that computer tools have many synergies, offering tighter than usual loops of self-improvement.  He envisioned a rapidly self-improving team focused on developing tools to help them develop better tools, and then actually oversaw a skilled team pursuing his vision for many years.  This team created working systems embodying dramatically-prescient features, and wowed the computer world with a dramatic demo. 

Wasn’t this a perfect storm for a tool takeoff scenario?  What odds would have been reasonable to assign to Doug’s team "taking over the world"?

GD Star Rating
loading...
Tagged as:

Fund UberTool?

By Robin Hanson · November 12, 2008 7:00 am · Comments (22)

Some companies specialize in making or servicing tools, and some even specialize in redesigning and inventing tools.  All these tool companies use tools themselves.  Let us say that tool type A "aids" tool type B if tools of type A are used when improving tools of type B.  The aiding graph can have cycles, such as when A aids B aids C aids D aids A. 

Such tool aid cycles contribute to progress and growth.  Sometimes a set of tool types will stumble into conditions especially favorable for mutual improvement.  When the aiding cycles are short and the aiding relations are strong, a set of tools may improve together especially quickly.  Such favorable storms of mutual improvement usually run out quickly, however, and in all of human history no more than three storms have had a large and sustained enough impact to substantially change world economic growth rates. 

Imagine you are a venture capitalist reviewing a proposed business plan.  UberTool Corp has identified a candidate set of mutually aiding tools, and plans to spend a millions pushing those tools through a mutual improvement storm.  While UberTool may sell some minor patents along the way, UberTool will keep its main improvements to itself and focus on developing tools that improve the productivity of its team of tool developers. 

In fact, UberTool thinks that its tool set is so fantastically capable of mutual improvement, and that improved versions of its tools would be so fantastically valuable and broadly applicable, UberTool does not plan to stop their closed self-improvement process until they are in a position to suddenly burst out and basically "take over the world."  That is, at that point their costs would be so low they could enter and dominate most industries.   

Now given such enormous potential gains, even a very tiny probability that UberTool could do what they planned might enticed you to invest in them.  But even so, just what exactly would it take to convince you UberTool had even such a tiny chance of achieving such incredible gains?

GD Star Rating
loading...
Tagged as: