An Outside View of AI Control

I’ve written much on my skepticism of local AI foom (= intelligence explosion). Recently I said that foom offers the main justification I understand for AI risk efforts now, as well as being the main choice of my Twitter followers in a survey. It was the main argument offered by Eliezer Yudkowsky in our debates here at this blog, by Nick Bostrom in his book Superintelligence, and by Max Tegmark in his recent book Life 3.0 (though he denied so in his reply here).

However, some privately complained to me that I haven’t addressed those with non-foom-based AI concerns. So in this post I’ll consider AI control in the context of a prototypical non-em non-foom mostly-peaceful outside-view AI scenario. In a future post, I’ll try to connect this to specific posts by others on AI risk.

An AI scenario is where software does most all jobs; humans may work for fun, but they add little value. In a non-em scenario, ems are never feasible. As foom scenarios are driven by AI innovations that are very lumpy in time and organization, in non-foom scenarios innovation lumpiness is distributed more like it is in our world. In a mostly-peaceful scenario, peaceful technologies of production matter much more than do technologies of war and theft. And as an outside view guesses that future events are like similar past events, I’ll relate future AI control problems to similar past problems.

Conway’s law of software says that the structure of software tends to reflect the structure of the social organizations that make it. This suggests that the needs of software to have particular modularity structures for particular problems is usually weaker than the needs of organizations to maintain familiar structures of communication and authority. So a world where software does most job tasks could retain a recognizable clumping of tasks into jobs, divisions, firms, professions, industries, cities, and nations.

Today, most lumpiness in firms, industries, cities, etc. is not due to lumpiness in innovation, but instead due to various scale and scope economies, and network effects. Innovation may be modestly lumpy at the scale of particular industries, but not at the level of entire economic sectors. Most innovation comes from many small contributions; big lumps contain only a small fraction of total value.

While innovation is often faster in some areas than in others, most social rates of change tend to be near the doubling time of the economy. An AI world allows for faster growth, as it isn’t held back by slow growing humans; I’ve estimated that it might double monthly. But our first guess should be that social rates of change speed up together; we’ll need specific reasons to expect specific changes, relative to today, in relative rates of change.

Today humans tend to get job specific training in the firms where they work, and general training elsewhere. Similarly, in an AI world specific software may be written in organizations near where it is used, while more general tools are written in more distant organizations.

Today most tasks are done by human brains, which come in a standard-sized unit capable of both doing specific tasks and also related meta-tasks, such as figuring out how to do tasks better, deciding which specific tasks to do when and how, and regrouping how tasks are clumped within organizations. So we tend to first automate tasks that can be done by much smaller units. And while managers and others often specialize in meta-tasks, line workers also do many meta tasks themselves. In contrast, in an AI world meta-tasks tend to be separated more from other tasks, and so done by software that is more different.

In such a division of tasks, most tasks have a relatively narrow scope. For narrow tasks, the main risks are of doing tasks badly, and of hostile agents taking control of key resources. So most control efforts focus on such problems. For narrow tasks there is little extra risk from such tasks being done very well, even if one doesn’t understand how that happens. (War tech is of course an exception; victims can suffer more when war is done well.) The control risks on which AI risk folks focus, of very effective efforts misdirected due to unclear goals, are mainly concentrated in tasks with very wide scopes, such as in investment, management, law, and governance. These are mostly very-meta-tasks.

The future problem of keeping control of advanced software is similar to the past problems of keeping control both of physical devices, and of social organizations. As the tasks we assign to physical devices tend to be narrow, we mostly focus there on specific control failure scenarios. The main risks there are losing control to hostile agents, and doing tasks badly, rather than doing them very well. The main people to get hurt when control is lost are those who rely on such devices, or who are closely connected to such people.

Humans started out long ago organized into small informal bands, and later had to learn to deal with the new organizations and institutions of rulers, command heirarchies, markets, family clans, large scale cultures, networks of talking experts, legal systems, firms, guilds, religions, clubs, and government agencies. Such organizations are often given relatively broad tasks. And even if not taken over by hostile agents, they can drift out of control. For example, organizations may on the surface seem responsive and useful, while increasingly functioning mainly to entrench and perpetuate themselves.

When social organizations get out of control in this way, the people who initiated and then participated in them are the main folks to get hurt. So such initiators and participants thus have incentives to figure out how to avoid such control losses, and this has long been a big focus of organization innovation efforts.

Innovation in control mechanisms has long been an important part of innovation in devices and organizations. People sometimes try to develop better control mechanisms in the abstract, before they’ve seen real systems. They also sometimes experiment in the context of small test versions. But most control innovations come in response to seeing real behaviors associated with typical real versions. The main reason that it becomes harder to implement innovations later is that design features often become entrenched. But if control is important enough, it can be worth paying large costs of change to implement better control features.

Humans one hundred thousand years ago might have tried to think carefully about how to control rulers with simple command hierarchies, and people one thousand years ago might have tried to think carefully about how to control complex firms and government agencies. But the value of such early efforts would have been quite limited, and it wasn’t at all too late to work on such problems after such systems appeared. In peacetime, control failures mainly threatened those who initiated and participated in such organizations, not the world as a whole.

In the AI scenario of this post, the vast majority of advanced future software does tasks of such narrow scopes that their control risks are more like those for physical devices, relative to new social organizations. So people deploying new advanced software will know to focus extra control efforts on software doing wider scope meta-tasks. Furthermore, the main people harmed by failures to control AI assigned to meta-tasks will be those associated with the organizations that do such meta-tasks.

For example, customers who let an AI tell them whom to date may suffer from bad dates. Investors in firms that let AI manage key firm decisions might lose their investments. And citizens who let AI tell their police who to put in jail may suffer in jail, or from undiscouraged crime. But such risks are mostly local, not global risks.

Of course for a long time now, coordination scales have been slowly increasingly worldwide. So over time “local” effects become increasingly larger scale effects. This is a modest reason for everyone to slowly get more concerned about “local” problems elsewhere.

Today is a very early date to be working on AI risk; I’ve estimated that without ems it is several centuries away. We are now pretty ignorant about most aspects of how advanced software will be used and arranged. So it is hard to learn much useful today about how to control future advanced software. We can learn to better control the software we have now, and later on we should expect innovations in software control to speed up roughly as fast as do innovations in making software more effective. Even if control innovations by humans don’t speed up as fast, advanced software will itself be made of many parts, and some parts will want to keep control over other parts.

The mechanisms by which humans today maintain control over organizations include law, property rights, constitutions, command hierarchies, and even democracy. Such broad mechanisms are effective, entrenched and robust enough that future advanced software systems and organizations will almost surely continue to use variations within these broad categories keep control over each other. So humans can reasonably hope to be at least modestly protected in the short run if they can share the use of such systems with advanced software. For example, if law protects software from stealing from software, it may also protect humans from such theft.

Of course humans have long suffered from events like wars and revolutions, events that create risks of harm and loss of control. And the rate of such events can scale with the main rates of change in the economy, which go inversely as the economic doubling time. So a much faster changing future AI economy can have faster rates of such risky events. It seems a robust phenomenon that when the world speeds up, those who do not speed up with it face larger subjective risks if they do not ally with sped-up protectors.

Having humans create and become ems is one reasonable approach to creating sped-up allies for humans. Humans will no doubt also try to place advanced software in such an ally role. Once software is powerful, then attempts by humans to control such software are probably mostly based on copying the general lessons and approaches that advanced software discovers for how to maintain control over advanced software. Humans may also learn extra lessons that are specific to the human control problem, and some of those lessons may come from our era, long before any of this plays out.

But in the sort of AI scenario I’ve described in this post, I find it very hard to see such early efforts as the do-or-die priority that some seem to suggest. Outside of a foom scenario, control failures threaten to cause local, not global losses (though on increasingly larger scales).

From this view, those tempted to spend resources now on studying AI control should consider two reasonable alternatives. The first alternative is to just save more now to grow resources to be used later, when we understand more. The second alternative is to work to innovate with our general control institutions, to make them more robust, and thus better able to handle larger coordination scales, and whatever other problems the future may hold. (E.g., futarchy.)

Okay, this is how I see the AI control problem in a non-em non-foom mostly-peaceful outside-view AI scenario. But clearly others disagree with me; what am I missing?

Added 4 Oct: 

In the context of foom, the usual AI concern is a total loss of control of the one super AI, whose goals quickly drift to a random point in the space of possible goals. Humans are then robustly exterminated. As the AI is so smart and inscrutable, any small loss of control is said to open the door to such extreme failure. I have presumed that those who tell me to look at non-foom AI risk are focused on similar failure scenarios.

Today most social systems suffer from agency costs, and larger costs (in % terms) for larger systems. But these mostly take the form of modestly increasing costs. It isn’t that you can’t reliably use these systems to do the things that you want. You just have to pay more. That extra cost mostly isn’t a transfer accumulating in someone else’s account. Instead there is just waste that goes to no one, and there are more cushy jobs and roles where people can comfortably sit as parasites. Over time, even though agency costs take a bigger cut, total costs get lower and humans get more of what they want.

When I say that in my prototypical non-foom AI scenario, AI will still pay agency costs but the AI control problem seems mostly manageable, I mean that very competent future social and software systems will suffer from waste and parasites as do current systems, but that humans can still reliably use such systems to get what they want. Not only are humans not exterminated, they get more than before of what they want.

GD Star Rating
loading...
Tagged as: , ,
Trackback URL:
  • J

    “An AI scenario is where software does most all jobs; humans may work for fun, but they add little value.”

    I get cranky about preoccupation with this spot in possibility space: the idea that technology, in a complete reversal of the last 300 years of industrial revolution, is (or will soon be) destroying wealth and employment. That position appears in fast-talking videos like the one from CGPgrey that sound reasonable but actually provide really awful arguments for the position. (Eg., his evidence is food ordering kiosks and assembly line robots, which have been around for decades or even centuries).

    I’m extra suspicious because the meme appeared at a time when government got especially interested in having more control over the internet, and needed to assign blame to someone else for unemployment and stagnation woes.

    For AI taking jobs it’s even worse, because unlike robots that have progressed gradually and can’t self-improve, an AI that can program can be asked to program a better AI. So there are lots of reasons to expect a very steep cliff once AIs start getting to the point where they can write programs and organize bureaucracies.

    I know you understand this stuff at least as well as I do, so I guess what I’m saying is to resist the urge to favor “automation doing most jobs in an otherwise familiar future”, as it’s a small patch on the side of a very steep mountain that we’re unlikely to spend much time occupying, yet it’s a great excuse for politicians to deflect blame for unemployment without addressing the complete upheaval that would actually happen if AIs started displacing programmers.

    • http://overcomingbias.com RobinHanson

      I’m trying to describe here what happens after the “steep cliff” of faster growth. You seem to disagree, but you don’t say clearly what you think happens different from what I’ve described.

  • Brian Slesinsky

    For non-AI machinery, instead of talking about a task being done “too well” we talk about the possibility of positive feedback and chain reactions. Positive feedback might result in an airplane crash, a train wreck or an explosion at a chemical plant, but we wouldn’t call that a job being done “too well”.

    We might also take a biological point of view and talk about the spread of viruses and other diseases. And indeed, computer viruses are common, and there are famous examples of viruses getting out of control and doing their job “too well” – that is, spreading further and faster than their creator intended.

    It seems likely that near-term AI risks will mostly be minor variations of ordinary computer security risks, but scaled up. This is quite scary enough. Every year there are bigger news stories where insufficient computer security plays an important role.

    Computer network attacks are often global in the sense that they affect computers in many parts of the world. So far they’ve only affected a small minority of computers at a time, but even so, we are often talking about millions of people and computers affected. It doesn’t seem all that far-fetched that something close to a global network outage could occur. (A security update mechanism in the wrong hands is essentially a botnet.)

    • http://overcomingbias.com RobinHanson

      Your concern about damaging computer viruses isn’t the concern of the AI risk folks that I’m trying to respond to here.

  • Charlene Cobleigh Soreff

    >For narrow tasks there is little extra risk from having such tasks done
    very very well, even if one doesn’t understand how that happens.

    Nope.

    There are a broad range of task where the purchaser of a service
    isn’t the _target_ of the task, and the target of the task can very
    well be damaged when the task is done well:

    Consider anti-personnel munitions
    or surveillance,
    or deceptive advertising.

    • http://overcomingbias.com RobinHanson

      Thanks; I clarified the issue of war-related tech in the post.

      • Charlene Cobleigh Soreff

        Many Thanks!

  • Robert Koslover

    I think this is a prescient essay; I expect future events to confirm your predictions. I do have a minor quibble about one of your examples. You say that “War tech is of course an exception; victims can suffer more when war is done well.” That’s actually not true. The war-fighters I know (and yes, I happen to know some) are generally pretty dedicated to winning wars via killing the smallest number of people they actually need to kill, and even doing the least amount of property damage needed, to ensure victory. Note also that recent wars fought with precision weapon technologies have led to far fewer casualties than past wars fought with “dumb” weapons. But actually, this is a general rule historically; it is not just a consequence of modern technology. Consider, in particular, the teachings of Sun Tzu, in his famous treatise, the Art of War (~500 BC): “The supreme art of war is to subdue the enemy without fighting.” This principle remains the goal of most professional warriors, in most countries, including even most of those hostile to us. Many wars, of course, have been extremely bloody and brutal. But victory — not brutality, destruction, and pointlessly spreading misery all around — is the goal of the professional warrior. Long, bloody, horrible, miserable, brutal wars (and yes, there have been very many) are wars that should be considered as done “poorly,” not “well.”

    • http://overcomingbias.com RobinHanson

      I just meant to indicate a possibility. Of course the harm caused by war tech depends a lot on the intentions of those running each side in the war.

    • Silent Cal

      Worth noting, but I think this is more of a wording issue. The point in context is that war AI is fairly likely to have programmed goals such that achieving them ‘too well’ would be harmful, unless its authors solve a value alignment problem.

      Real-world paper clip manufacturers would also not consider turning the world into clips to be a job well done, but that’s essentially what we mean by talking about a paper-clip-making AI doing ‘too well’.

  • https://llordoftherealm.wordpress.com/ Lord

    It certainly is difficult to anticipate what may come to pass and do much now about it, but it may be worth considering, one approach being advanced simulations for prediction. By the time this could happen, local would be global though and fast speeds would rapidly propagate through the system and loss of control may no longer be recoverable. It may be more benevolent dictator than sky net and we may be welcoming our robot overlords, after all would we want control if it meant being worse off?

  • Pingback: Rational Feed – deluks917

  • Michael Vassar

    If history teaches one thing, I’d say its that social control mechanisms usually go irrevocably out of control. A second thing might be that there’s a very long term trend towards unipolarity.

    • Fleshy506

      Not sure what you mean by “social control mechanisms usually go irrevocably out of control,” but I’m curious. The specific examples of social control mechanisms Robin gives are “law, property rights, constitutions, command hierarchies, and even democracy.” In what sense have these or other social control mechanisms gone out of control, and why irrevocably? Are you talking about the sort of thing people mean when they complain about there being too many lawyers, regulators, and/or bureaucrats gumming up the works? (I can think of more sinister examples, like totalitarian regimes, but those don’t seem so irrevocable.)

      Also, regarding the trend towards unipolarity, are you proposing out of control social control mechanisms as a manifestation/mechanism of that? Are there any plausible alternative paths humanity could go down?

      • Michael Vassar

        Law, property rights, command and control mechanisms, heirarchy and democracy all converge on degenerate states of social control where they become self evident lost purposes, eventually leading to the society being overthrown from outside by less civilized humans. Movement towards unipolarity makes the cycle take longer as there are fewer people around to overthrow it.

        It’s probably possible to do better if we are ever able to construct a system on a decent foundation of knowledge of human behavior, but I think it’s more likely to happen by constructing desired genomes, since those are probably more linear and thus more understandable.

  • http://praxtime.com/ Nathan Taylor

    For the non-em advanced AI scenario, we could build AIs that are socially similar to humans. Those AIs could reasonably build upon the foundational controls used by humans today: law, rights, democracies, etc. And there should be every incentive to build such AIs, as we want AI to be able to communicate and understand humans. This is the mostly-peaceful scenario. em or non-em won’t be that different. Agree this seems most likely.

    Now. There is an AI risk scenario where we build AIs which do not have human style social instincts. In human terms: psychopaths. I mean this not in the sense of crazy, but in the sense of having no innate craving to obey social norms, associate with others, have social/cultural learning. In the natural world, I always think of the octopus. Born of an egg, never really social, but highly intelligent. In social terms: alone and amoral. In the non-foom scenario, it seems like it should be possible to co-evolve ways to control. But it’s less clearly a given.

    As the gap grows between AIs and humans, the weight of interaction importance will shift to AI-AI interactions and away from AI-human interactions. If AIs start socially similar to humans, that’s fine. If AIs are never socially human-like, then AI-AI interaction may risk a phase shift to become a darwinian winner take all, prisoner’s dilemma race for resources.

    The point here is the form of AI control I see as most likely and our best bet, is to make AI similar enough to humans in terms of AI social “instincts” so they can start from existing human forms of control: law, rights, government. Furthermore, AI fears, both foom and non-foom, are similar, in they are fears of amoral AI. Even in the non-foom scenario, once the gap between AIs and normal humans gets large enough, AIs will primarily need to focus on interacting with other god-like AIs (in the sense of running at speeds far faster than humans if nothing else).

    So the steelman argument for non-foom AI risk seems to be the case where AIs are amoral, where they over time become godlike against humans, but we fail to co-evolve control suitable for such entities. (where I favor/expect social “instincts” in AIs). Then one of these AIs goes darwinian/malthusian for resources. Either by cloning itself endlessly, or sucking all resources into a single centralized entity. At that point we’re in the Bostrum scenario. This cluelessness seems unlikely given our current human fascination with god-like AIs. But it’s clearly a very foom-like result, albeit one brought on by gradual cumulative capability growth of AI to the point where foom-like bad things could happen with any single AI. And it could move from local to global quickly since at that point the human AI gap is massive.

    • http://overcomingbias.com RobinHanson

      Humans already are Darwinian/Malthusian. Familiar social institutions do not collapse when dealing with psychopaths. AI that is competent will be competent at dealing with its social environment, which includes familiar social institutions. It doesn’t need special “instincts” beyond competence.

      • http://praxtime.com/ Nathan Taylor

        My underlying thought here is taken from the common criticism of group selection by evolutionary theorists. Where groupish behavior is constantly undermined by cheaters in the group. So human cooperation ultimately is built on social cooperation instincts (gossip/policing/punishing against norm violations). And going down a level, our bodies are constantly fighting off cancers (defectors) that evolve.

        That is to say, I don’t think a society of pyshcopaths/purely calculating actors can exist. The more competent the psychopaths, the faster their society would end with all defecting on all in the prisoners dilemma. Tit-for-tat is not enough for the center to hold. I’m not sure of your view. But sounds like you believe a society can exist whose members are rationally very competent in social skills, but do not have an innate baseline program (“instinct”) to abide certain group beneficial norms. I’m skeptical of that being possible. Sorry if I’m confused on your position.

        Unfortunately examples in the animal world aren’t too helpful, because of course all cooperative animal groups use instinct for this, not rational calculation. But to the extent this applies, I’d say suggestively instinct for group behavior is hard to evolve or retain. Humans arguably boot strapped it from our unusual selective pressure that came from social learning (aka Henrich). Where the groupish traits got pulled along by selective pressure for social learning.

  • davidmanheim

    I’m not qualified to do more than weigh arguments on each side of the question of time scales, but what approximate probability distribution would you used for the arrival date of human-level AI, (conditional on not having ems or not)?

    The reason I ask is that many AI risk people I have spoken to agree AI is likely many decades away, but because our confidence in that estimate is low, and the distribution of technical advances is hard to predict, the probability it is much sooner (20-30 years away) is non-negligible.

    • davidmanheim

      (This is plausibly “what you’re missing” – a 5% chance of near-term AI would make the value of near-term risk mitigation very high.)

    • http://overcomingbias.com RobinHanson

      It isn’t enough to argue that it might be soon. You also have to argue that there’s something we can do now that’s more useful than saving resources for when we know more, or making our general institutions of control more robust.

  • Pingback: Overcoming Bias : Reply to Christiano on AI Risk

  • https://dndebattbetyg.wordpress.com/ Stefan Schubert

    Effective altruists have talked about “narrow” vs “broad” interventions to reduce existential risk and shape the long-term future, where narrow interventions target specific risks and include, e.g., technical AI safety work, and where broad interventions target multiple risks and include, e.g., institutional reform.

    You here suggest some broad interventions:

    “From this view, those tempted to spend resources now on studying AI control should consider two reasonable alternatives. The first alternative is to just save more now to grow resources to be used later, when we understand more. The second alternative is to work to innovate with our general control institutions, to make them more robust, and thus better able to handle larger coordination scales, and whatever other problems the future may hold. (E.g., futarchy.)”

    These are interesting ideas. I’d be very interested if you had further thoughts on what broad interventions to pursue.

  • Paul Christiano

    > Today is a very early date to be working on AI risk; I’ve estimated that without ems it is several centuries away.

    If I thought AI were very likely to be > a century away, then I wouldn’t work on AI risk. I expect most people interested in safety don’t believe your estimates.

    I see good arguments that ML-similar-to-today has a >20% chance of scaling up to human-level performance, and if it does then work done in the context of existing ML has a good chance of being applicable to the earliest powerful AI systems. If this is the case, then it’s a good deal, since it seems quite easy to make progress on such problems relative to their importance. In the future spending will be radically larger, and it will be much harder to have a similar impact.

    (You lean a lot on your timeline estimation methodology, but haven’t really made the argument precise or engaged with some typical criticisms, and I don’t currently consider it nearly as convincing as other lines of argument. The predictions also already look like they are in bad empirical shape—400 years for early visual processing in humans?)

    • http://overcomingbias.com RobinHanson

      This post is not about the time estimate; is there nothing else in it you find relevant? I don’t know of any posted criticisms of my time estimates to engage. (In general, if you want me to engage something, TELL ME IT EXISTS.)

      • Paul Christiano

        I responded to that paragraph because it seemed like the main substantive argument against working on control now. I agree with most of the rest of the claims in the post, but don’t see you as making much of an argument against working on control.

        I don’t see why this post shows that law enforcement and government will clearly be able to utilize AI without control problems. You say things like:

        > Such broad mechanisms are effective, entrenched and robust enough that future advanced software systems and organizations will almost surely continue to use variations within these broad categories keep control over each other. So humans can reasonably hope to be at least modestly protected in the short run if they can share the use of such systems with advanced software. For example, if law protects software from stealing from software, it may also protect humans from such theft.

        Which does not really speak to most AI risk advocates’ concerns (we are concerned about humans realizing much of the value in the universe over the long term).

        You say:

        > In peacetime, control failures mainly threatened those who initiated and participated in such organizations, not the world as a whole.

        and

        > Outside of a foom scenario, control failures threaten to cause local, not global losses (though on increasingly larger scales).

        But don’t provide much argument—if government and law enforcement cannot effectively apply AI to protect human interests because of technological limitations, then that “local” problem will arise in *every* locality, i.e. it will be a global problem.

        You say:

        > such initiators and participants thus have incentives to figure out how to avoid such control losses, and this has long been a big focus of organization innovation efforts

        With which I agree; future people will have incentives to deal with these problems. But the argument for working on control now is that if there is a problem with small investment today and large investment in the future, additional investment today can be much more highly leveraged than additional investment in the future (owing to serial dependencies and communication latency).

        I guess you give the relevant example:

        > And citizens who let AI tell their police who to put in jail may suffer in jail, or from undiscouraged crime. But such risks are mostly local, not global risks.

        But you seem to have basically agreed that police need to use AI to enforce the law. So citizens everywhere will face a decision between allowing AI to be involved in law enforcement, and tolerating ineffective law enforcement.

        You point to an analogy with other control tasks:

        > The future problem of keeping control of advanced software is similar to the past problems of keeping control both of physical devices, and of social organizations.

        That’s an analogy I’ve considered at some length, and it doesn’t make me feel much more optimistic. You don’t say much about why it should make us feel more optimistic. It’s not convincing to just say that two things look similar, and one isn’t catastrophic. One data point does not make a strong inductive trend (and the interpretation is not even clear, agency costs today are quite large and mainly acceptable collectively because they are paid to other humans and easily dominated by wages anyway, before we even get into large differences in cognitive ability which are one of the main sources of concern in the AI case).

  • Yosarian2

    One possible concern in this kind of situation is what Scott Alexander described as an “ascended economy”. That would be an economy where several corporations owned and run by AI’s with a fully automated workforce start to just trade with each other directly in ways that don’t involve humans at all. In that case they could just continue to expand economically and consume more and more resources without there being any humans capable of putting any control mechanisms on them.

    http://slatestarcodex.com/2016/05/30/ascended-economy/

    • http://overcomingbias.com RobinHanson

      That’s just a generic concern about all competition.

      • Yosarian2

        But capitalism thus far has been done in such a way that while maybe not everyone benefits (externalities or imbalances of information or whatever), that at least some people benefit. An AI driven economy could turn into something that literally benefits no one, especially if the AI’s do not have moral value of their own. Over the long run the entire AI economy could turn into the equivalent of an AI paperclipper, using up all available resources while producing nothing of true value.

      • http://overcomingbias.com RobinHanson

        Again, that’s just a generic concern; it has nothing to do with AI in particular.

  • Will Pearson

    Hey Robin, I’m interested in a system that changes the method of control of programs inside a computer system.

    Currently you can see your management of the resources inside a computer as akin to a command economy. You install new programs which take up as much hard disk space, a resource, as they want. You have to delete files it creates. They also take up as much memory as they want, unless you put specific bounds on them.

    This puts a strain on you, the computer user, to the number of programs that you want to manage and also how quickly they can be updated as you need to get to know how to use the new features.

    So I am interested in creating systems that use a economy based on feedback from the user to manage normal computer programs. I think this could radically change the speed of program change, especially when novelty is added into the system (there was no previous way of telling what good novelty was vs bad, but the economy decides).

    I’m not sure this will work, but I think a change in the management of computational resources will have a large impact on the efficiency how software is managed which will impact the world a lot.

    This sort of thing seems required for true general AI and also in need of making sure that it works well.

    I have a website about my approach and what I hope to do with it, if you are interested.

  • Joe

    Robin, how well do you think the “mostly peaceful” assumption holds up here? I ask because in your em scenario, you give a sizeable chance of human extermination per objective year of em era (30%, IIRC?). Won’t this be equally true in a multipolar non-em AI future? Perhaps moreso, if the AIs are more able to develop new separate institutions due to not being as tied to a human past?

    If so, then under the further assumption of complex slowly-accumulating AI that takes centuries to reach human level (rather than Christiano’s prosaic AGI model), perhaps our best bet is to hurry ems, so that this transition to an AI future happens when we’re running at the same speed as the AIs and thus can better integrate into their institutions.

    (Note that this assumes a framework in which all that matters is our own personal survival and happiness – which is probably not right but is nonetheless useful as a partially correct model.)

    • http://overcomingbias.com RobinHanson

      I assume “mostly peaceful” here mainly to separate scenarios according to factors that are most relevant to each one. We’d focus on different factors in a war scenario.

  • David Krueger

    “In the context of foom, the usual AI concern is a total loss of control of the one super AI, whose goals quickly drift to a random point in the space of possible goals.”

    That seems very wrong to me. The concern is not about goals drifting; it is about them being relentlessly pursued. What am I missing?

    • Riothamus

      I may be mistaken, but I think the drift here means ‘relative to what we believed them to be’.

      There isn’t much to distinguish something that is directly random from something that is bounded in its randomness but the bound is unknown.