Tag Archives: AI

Is The City-ularity Near?

The land around New York City is worth a lot.  A 2008 analysis estimated prices for land, not counting buildings etc., for most (~80%?) of the nearby area (2750 square miles, = a 52 mile square).  The total New York area land value (total land times ave price) was 5.5T$ (trillion) in 2002 and 28T$ in 2006.

The Economist said that in 2002 all developed nation real estate was worth 62T$.  Since raw land value is on average about a third of total real estate value, that puts New York area real estate at over 30% of all developed nation real estate in 2002!  Whatever the exact number, clearly this agglomeration contains vast value.

New York land is valuable mainly because of how it is organized.  People want to be there because they want to interact with other people they expect to be there, and they expect those interactions to be quite mutually beneficial.  If you could take any other 50 mile square (of which Earth has 72,000), and create that same expectation of mutual value from interactions, you could get people to come there, make buildings, etc., and sell that land for many trillions of dollars of profit.

Yet the organization of New York was mostly set long ago based on old tech (e.g., horses, cars, typewriters).  Worse, no one really understands at a deep level how it is organized or why that works so well.  Different people understand different parts, in mostly crude empirical ways.

So what will happen when super-duper smarties wrinkle their brows so hard that out pops a deep math theory of cities, explaining clearly how city value is produced?  What if they apply their theory to designing a city structure that takes best advantage of our most advanced techs, of 7gen phones, twitter-pedias, flying Segways, solar panels, gene-mod pigeons, and super-fluffy cupcakes?  Making each city aspect more efficient makes the city more attractive, increasing the gains from making other aspects more efficient, in a grand spiral of bigger gains.

Once they convince the world of the vast value in their super-stupendous city design, won’t everyone flock there and pay mucho trillions for the privilege? Couldn’t they leverage this lead into better theories enabling better designs giving far more trillions, and then spend all that on a super-designed war machine based on those same super insights, and turn us all into down dour super-slaves?  So isn’t the very mostest importantest cause ever to make sure that we, the friendly freedom fighters, find this super deep city theory first?

Well, no, it isn’t.  We don’t believe in a city-ularity because we don’t believe in a super-city theory found in a big brain flash of insight.  What makes cities work well is mostly getting lots of details right.  Sure new-tech-based cities designs can work better, but gradual tech gains mean no city is suddenly vastly better than others.  Each change has costs to be weighed against hoped-for gains.  Sure costs of change might be lower when making a whole new city from scratch, but for that to work you have to be damn sure you know which changes are actually good ideas.

For similar reasons, I’m skeptical of a blank-slate AI mind-design singularity.  Sure if there were a super mind theory that allowed vast mental efficiency gains all at once, but there isn’t.  Minds are vast complex structures full of parts that depend intricately on each other, much like the citizens of a city.  Minds, like cities, best improve gradually, because you just never know enough to manage a vast redesign of something with such complex inter-dependent adaptations.

What Core Argument?

People keep asking me to return to the core of the argument, but, well, there's just not much there.  Let's review, again.  Eliezer suggests someone soon may come up with a seed AI architecture allowing a single AI to within roughly a week grow from unimportant to strong enough to take over the world.  I'd guess we are talking over 20 orders of magnitude growth in its capability, or 60 doublings.  

This amazing growth rate sustained over such a large magnitude range is far beyond what the vast majority of AI researchers, growth economists, or most any other specialists would estimate.  It is also far beyond estimates suggested by the usual choices of historical analogs or trends.  Eliezer says the right reference set has two other elements, the origin of life and the origin of human minds, but why should we accept this reference?  He also has a math story to suggest this high average growth, but I've said:

I also find Eliezer's growth math unpersuasive. Usually dozens of relevant factors are co-evolving, with several loops of all else equal X growth speeds Y growth speeds etc. Yet usually it all adds up to exponential growth, with rare jumps to faster growth rates. Sure if you pick two things that plausibly speed each other and leave everything else out including diminishing returns your math can suggest accelerating growth to infinity, but for a real foom that loop needs to be real strong, much stronger than contrary muting effects.

Eliezer has some story about how chimp vs. human brain sizes shows that mind design doesn't suffer diminishing returns or low-hanging-fruit-first slowdowns, but I have yet to comprehend this argument.  Eliezer says it is a myth that chip developers need the latest chips to improve chips as fast as they do, so there aren't really diminishing returns there, but chip expert Jed Harris seems to disagree.

Continue Reading "What Core Argument?" »

Two Visions Of Heritage

Eliezer and I seem to disagree on our heritage.

I see our main heritage from the past as all the innovations embodied in the design of biological cells/bodies, of human minds, and of the processes/habits of our hunting, farming, and industrial economies.  These innovations are mostly steadily accumulating modular "content" within our architectures, produced via competitive processes and implicitly containing both beliefs and values.  Architectures also change at times as well.

Since older heritage levels grow more slowly, we switch when possible to rely on newer heritage levels.  For example, we once replaced hunting processes with farming processes, and within the next century we may switch from bio to industrial mental hardware, becoming ems.  We would then rely far less on bio and hunting/farm heritages, though still lots on mind and industry heritages.  Later we could make AIs by transferring mind content to new mind architectures.  As our heritages continued to accumulate, our beliefs and values should continue to change. 

I see the heritage we will pass to the future as mostly avoiding disasters to preserve and add to these accumulated contents.  We might get lucky and pass on an architectural change or two as well.  As ems we can avoid our bio death heritage, allowing some of us to continue on as ancients living on the margins of far future worlds, personally becoming a heritage to the future.

Continue Reading "Two Visions Of Heritage" »

Are AIs Homo Economicus?

Eliezer yesterday

If I had to pinpoint a single thing that strikes me as "disagree-able" about the way Robin frames his analyses, it’s that there are a lot of opaque agents running around, little black boxes assumed to be similar to humans, but there are more of them and they’re less expensive to build/teach/run.  … The core of my argument has to do with what happens when you pry open the black boxes that are your economic agents, and start fiddling with their brain designs, and leave the tiny human dot in mind design space.

Lots of folks complain about economists; believers in peak oil, the gold standard, recycling, electric cars, rent control, minimum wages, tariffs, and bans on all sorts of things complain about contrary economic analyzes.  Since compared to most social scientists economists use relatively stark mathy models, the usual complaint is that our models neglect relevant factors, and make false assumptions. 

But of course we must neglect most everything, and make false assumptions, to have tractable models; the question in each context is what neglected factors and false assumptions would most mislead us. 

It is odd to hear complaints that economic models assume too much humanity; the usual complaint is the opposite.  Unless physicists have reasons to assume otherwise, they usually assume masses are at points, structures are rigid, surfaces are frictionless, and densities are uniform.  Similarly, unless economists have reasons to be more realistic in a context, they usually assume people are identical, risk-neutral, live forever, have selfish material stable desires, know everything, make no mental mistakes, and perfectly enforce every deal.  Products usually last one period or forever, are identical or infinitely varied, etc.

Continue Reading "Are AIs Homo Economicus?" »

Wrapping Up

This Friendly AI discussion has taken more time than I planned or have.  So let me start to wrap up. 

On small scales we humans evolved to cooperate via various pair and group bonding mechanisms.  But these mechanisms aren’t of much use on today’s evolutionarily-unprecedented large scales.  Yet we do in fact cooperate on the largest scales.  We do this because we are risk averse, because our values mainly conflict on resource use which conflicts destroy, and because we have the intelligence and institutions to enforce win-win deals via property rights, etc. 

I raise my kids because they share my values.  I teach other kids because I’m paid to.  Folks raise horses because others pay them for horses, expecting horses to cooperate as slaves.  You might expect your pit bulls to cooperate, but we should only let you raise pit bulls if you can pay enough damages if they hurt your neighbors. 

In my preferred em (whole brain emulation) scenario, people would only authorize making em copies using borrowed or rented brains/bodies when they expected those copies to have lives worth living.  With property rights enforced, both sides would expect to benefit more when copying was allowed.  Ems would not exterminate humans mainly because that would threaten the institutions ems use to keep peace with each other. 

Similarly, we expect AI developers to plan to benefit from AI cooperation, via either direct control, indirect control such as via property rights institutions, or such creatures having cooperative values.  As with pit bulls, developers should have to show an ability, perhaps via insurance, to pay plausible hurt amounts if their creations hurt others.  To the extent they or their insurers fear such hurt, they would test for various hurt scenarios, slowing development as needed in support.  To the extent they feared inequality from some developers succeeding first, they could exchange shares, or share certain kinds of info.  Naturally-occurring info-leaks, and shared sources, both encouraged by shared standards, would limit this inequality.

Continue Reading "Wrapping Up" »

Friendly Projects vs. Products

I’m a big board game fan, and my favorite these days is Imperial.   Imperial looks superficially like the classic strategy-intense war game Diplomacy, but with a crucial difference:  instead of playing a nation trying to win WWI, you play a banker trying to make money from that situation.  If a nation you control (by having loaned it the most) is threatened by another nation, you might indeed fight a war, but you might instead just buy control of that nation.  This is a great way to mute conflicts in a modern economy: have conflicting groups buy shares in each other.

For projects to create new creatures, such as ems or AIs, there are two distinct friendliness issues: 

Project Friendliness  Will the race make winners and losers, and how will winners treat losers? While any race might be treated as part of a total war on several sides, usually the inequality created by the race is moderate and tolerable.  For larger inequalities, projects can explicitly join together, agree to cooperate in weaker ways such as by sharing information, or they can buy shares in each other.  Naturally arising info leaks and shared standards may also reduce inequality even without intentional cooperation.  The main reason for failure here would seem to be the sorts of distrust that plague all human cooperation.

Product Friendliness  Will the creatures cooperate with or rebel against their creators?  Folks running a project have reasonably strong incentives to avoid this problem.  Of course for the case of extremely destructive creatures the project might internalize more of the gains from cooperative creatures than they do the losses from rebellious creatures.  So there might be some grounds for wider regulation.  But the main reason for failure here would seem to be poor judgment, thinking you had your creatures more surely under control than in fact you did. 

It hasn’t been that clear to me which of these is the main concern re "friendly AI." 

Added:  Since Eliezer says product friendliness is his main concern, let me note that the main problem there is the tails of the distribution of bias among project leaders.  If all projects agreed the problem was very serious they would take near appropriate caution to isolate their creatures, test creature values, and slow creature development enough to track progress sufficiently.  Designing and advertising a solution is one approach to reducing this bias, but it need not need the best approach; perhaps institutions like prediction markets that aggregate info and congeal a believable consensus would be more effective. 

I Heart CYC

Eliezer Tuesday:

EURISKO may still be the most sophisticated self-improving AI ever built – in the 1980s, by Douglas Lenat before he started wasting his life on Cyc.  … EURISKO lacked what I called "insight" – that is, the type of abstract knowledge that lets humans fly through the search space. 

I commented:

You ignore that Lenat has his own theory which he gives as the reason he’s been pursuing CYC. You should at least explain why you think his theory wrong; I find his theory quite plausible.

Eliezer replied only:

Artificial Addition, The Nature of Logic, Truly Part of You, Words as Mental Paintbrush Handles, Detached Lever Fallacy

The main relevant points from these Eliezer posts seem to be that AI researchers wasted time on messy ad-hoc non-monotonic logics, while elegant mathy Bayes nets approaches work much better, that it is much better to know how to generate specific knowledge from general principles than to just be told lots of specific knowledge, and that our minds have lots of hidden machinery behind the words we use; words as "detached levers" won’t work.  But I doubt Lenat or CYC folks disagree with any of these points.

The lesson Lenat took from EURISKO is that architecture is overrated;  AIs learn slowly now mainly because they know so little.  So we need to explicitly code knowledge by hand until we have enough to build systems effective at asking  questions, reading, and learning for themselves.  Prior AI researchers were too comfortable starting every project over from scratch; they needed to join to create larger integrated knowledge bases.  This still seems to me a reasonable view, and anyone who thinks Lenat created the best AI system ever should consider seriously the lesson he thinks he learned.

Continue Reading "I Heart CYC" »

Stuck In Throat

Let me try again to summarize Eliezer’s position, as I understand it, and what about it seems hard to swallow.  I take Eliezer as saying: 

Sometime in the next few decades a human-level AI will probably be made by having a stupid AI make itself smarter.  Such a process starts very slow and quiet, but eventually "fooms" very fast and then loud. It is likely to go from much stupider to much smarter than humans in less than a week.  While stupid, it can be rather invisible to the world.  Once smart, it can suddenly and without warning take over the world. 

The reason an AI can foom so much faster than its society is that an AI can change its basic mental architecture, and humans can’t.  How long any one AI takes to do this depends crucially on its initial architecture.  Current architectures are so bad that an AI starting with them would take an eternity to foom.  Success will come from hard math-like (and Bayes-net-like) thinking that produces deep insights giving much better architectures.

A much smarter than human AI is basically impossible to contain or control; if it wants to it will take over the world, and then it will achieve whatever ends it has.  One should have little confidence that one knows what those ends are from its behavior as a much less than human AI (e.g., as part of some evolutionary competition).  Unless you have carefully proven that it wants what you think it wants, you have no idea what it wants. 

In such a situation, if one cannot prevent AI attempts by all others, then the only reasonable strategy is to try to be the first with a "friendly" AI, i.e., one where you really do know what it wants, and where what it wants is something carefully chosen to be as reasonable as possible. 

I don’t disagree with this last paragraph.  But I do have trouble swallowing prior ones.  The hardest to believe I think is that the AI will get smart so very rapidly, with a growth rate (e.g., doubling in an hour) so far out of proportion to prior growth rates, to what prior trends would suggest, and to what most other AI researchers I’ve talked to think.  The key issues come from this timescale being so much shorter than team lead times and reaction times.  This is the key point on which I await Eliezer’s more detailed arguments. 

Since I do accept that architectures can influence growth rates, I must also have trouble believing humans could find new AI architectures anytime soon that make this much difference.  Some other doubts: 

  • Does a single "smarts" parameter really summarize most of the capability of diverse AIs?
  • Could an AI’s creators see what it wants by slowing down its growth as it approaches human level?
  • Might faster brain emulations find it easier to track and manage an AI foom?

Total Tech Wars

Eliezer Thursday:

Suppose … the first state to develop working researchers-on-a-chip, only has a one-day lead time. …  If there’s already full-scale nanotechnology around when this happens … in an hour … the ems may be able to upgrade themselves to a hundred thousand times human speed, … and in another hour, …  get the factor up to a million times human speed, and start working on intelligence enhancement. … One could, of course, voluntarily publish the improved-upload protocols to the world, and give everyone else a chance to join in.  But you’d have to trust that not a single one of your partners were holding back a trick that lets them run uploads at ten times your own maximum speed. 

Carl Shulman Saturday and Monday

I very much doubt that any U.S. or Chinese President who understood the issues would fail to nationalize a for-profit firm under those circumstances. … It’s also how a bunch of social democrats, or libertarians, or utilitarians, might run a project, knowing that a very likely alternative is the crack of a future dawn and burning the cosmic commons, with a lot of inequality in access to the future, and perhaps worse. Any state with a lead on bot development that can ensure the bot population is made up of nationalists or ideologues (who could monitor each other) could disarm the world’s dictatorships, solve collective action problems … [For] biological humans [to] retain their wealth as capital-holders in his scenario, ems must be obedient and controllable enough … But if such control is feasible, then a controlled em population being used to aggressively create a global singleton is also feasible. 

Every new technology brings social disruption. While new techs (broadly conceived) tend to increase the total pie, some folks gain more than others, and some even lose overall.  The tech’s inventors may gain intellectual property, it may fit better with some forms of capital than others, and those who first foresee its implications may profit from compatible investments.  So any new tech can be framed as a conflict, between opponents in a race or war.

Every conflict can be framed as a total war.  If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury.  All resources must be devoted to growing more resources and to fighting them in every possible way. 

Continue Reading "Total Tech Wars" »

Billion Dollar Bots

Robin presented a scenario  in which whole brain emulations, or what he calls bots come into being.  Here is another:

Bots are created with hardware and software.  The higher the quality of one input the less you need of the other.  Hardware, especially with cloud computing, can be quickly allocated from one task to another.  So the first bot might run on hardware worth billions of dollars.

The first bot creators would receive tremendous prestige and a guaranteed place in the history books.  So once it becomes possible to create a bot many firms and rich individuals will be willing to create one even if doing so would cause them to suffer a large loss.

Imagine that some group has $300 million to spend on hardware and will use the money as soon as $300 million becomes enough to create a bot.  The best way to spend this money would not be to buy a $300 million computer but to rent $300 million of off-peak computing power.  If the group needed only 1,000 hours of computing power (which it need not buy all at once) to prove that it had created a bot then the group could have, roughly, $3 billion of hardware for the needed 1,000 hours.

It’s likely that the  first bot would run very slowly.  Perhaps it would take the bot 10 real seconds to think as much as a human does in one second.

Under my scenario the first bot would be wildly expensive.  But because of Moore’s law once the first bot was created everyone would expect that the cost of bots would eventually become low enough so that they would radically remake society.

Consequently, years before bots come to dominate the economy, many people will come to expect that within their lifetime bots will someday come to dominate the economy.   Bot expectations will radically change the world. 

I suspect that after it becomes obvious that we could eventually create cheap bots world governments will devote trillions to bot Manhattan projects.  The expected benefits of winning the bot race will be so high that it would be in the self-interest of individual governments to not worry too much about bot friendliness.

The U.S. and Chinese militaries  might fall into a bot prisoners’ dilemma in which both militaries would prefer an outcome in which everyone slowed down bot development to ensure friendliness yet both nations were individually better off (regardless of what the other military did) taking huge chances on friendliness so as to increase the probability of their winning the bot race. 

My hope is that the U.S. will have such a tremendous advantage over China that the Chinese don’t try to win the race and the U.S. military thinks it can afford to go slow.  But given China’s relatively high growth rate I doubt humanity will luck into this safe scenario.