Tag Archives: Ems

New Yorker on Age of Em

Joshua Rothman, in The New Yorker, on Age of Em:

It may be, too, that we should look with some trepidation toward the transitional period—that strange era in which our real-world ways will be disrupted by the introduction of new and bizarre simulated life forms. In “The Age of Em,” a nonfiction work of social-science speculation published earlier this year, the economist and futurist Robin Hanson describes a time in which researchers haven’t yet cracked artificial intelligence but have learned to copy themselves into their computers, creating “ems,” or emulated people, who quickly come to outnumber the real ones. Unlike Bostrom, who supposes that our descendants will create simulated worlds for curiosity’s sake, Hanson sees the business case for simulating people: instead of struggling to find a team of programmers, a company will be able to hire a single, brilliant em and then replicate her a million times. An enterprising em might gladly replicate herself to work many jobs at once; after she completes a job, a copied em might choose to delete herself, or “end.” (An em contemplating ending won’t ask “Do I want to die?,” Hanson writes, since other copies will live on; instead, she’ll ask, “Do I want to remember this?”) An em might be copied right after a vacation, so that whenever she is pasted into the simulated workplace, she is cheerful, rested, and ready to work. She might also be run on computer hardware that is more powerful than a human brain, and so think (and live) at a speed millions or even trillions of times faster than an ordinary human being.

Hanson doesn’t think that ems must necessarily live unhappy lives. On the contrary, they may thrive, fall in love, and find fulfillment in their competitive, flexible, high-speed world. Non-simulated people, meanwhile, may retire on the proceeds from their investments in the accelerated and increasingly autonomous em economy—a pleasant vantage point from which to observe the twilight of non-emulated civilization. Many people have imagined that technology will free us from the burden of work; if Hanson is right, that freedom could come through the virtualization of the human race.

This was in an article about the simulation argument. Two years ago I compared em and sim conversations, noting that in both cases many discuss using them as fiction settings, the chances that they are true, clues for inferring if they are true, and what they imply for identity, consciousness, physics, etc. But few discuss social consequences, such as how to live in a simulation or what a em world is like socially.

Oddly to me, Rothman didn’t go that direction; he didn’t even mention my (or anyone’s) analysis of how to live in a simulation.

Oh, and running trillions of times faster than humans is quite a bit faster than I’ve guessed; I’ve said maybe millions of times faster.

GD Star Rating
Tagged as: ,

Reply to Jones on Ems

In response to Richard Jones’ book review, I said:

So according to Jones, we can’t trust anthropologists to describe foragers they’ve met, we can’t trust economics when tech changes society, and familiar design principles fail for understanding brains and tiny chemical systems. Apparently only his field, physics, can be trusted well outside current experience. In reply, I say I’d rather rely on experts in each field, relative to his generic skepticism. Brain scientists see familiar design principles as applying to brains, even when designed by evolution, economists see economics as applying to past and distant societies with different tech, and anthropologists think they can understand cultures they visit.

Jones complained on twitter that I “prefer to argue from authority rather than engage with their substance.” I replied “There can’t be much specific response to generic skepticism,” to which he replied, “Well, there’s more than 4000 words of quite technical argument on the mind uploading question in the post I reference.” He’s right that he wrote 4400 words. But let me explain why I see them more as generic skepticism than technical argument.

For context, note that there are whole fields of biological engineering, wherein standard engineering principles are used to understand the engineering of biological systems. These include the design of many specific systems with organisms, such as lungs, blood, muscles, bone, and skin, and also specific subsystems within cells, and also standard behaviors, such as gait rhythms and foraging patterns. Standard design principles are also used to understand why cells are split into different modules that perform distinct functions, instead of having each cell try to contribute to all functions, and why only a few degrees of freedom for each cell matters for that cell’s contribution to its system. Such design principles can also be used to understand why systems are abstract, in the sense of as having only one main type of muscle, for creating forces used for many purposes, one main type of blood system, to move most everything around, or only one main fast signal system, for sending signals of many types.

Our models of the function of many key organs have in fact often enabled us to create functional replacements for them. In addition, we already have good models of, and successful physical emulations of, key parts of the brain’s input and out, such, as input from eyes and ears, and output to arms and legs.

Okay, now here are Jones’ key words:

This separation between the physical and the digital in an integrated circuit isn’t an accident or something pre-ordained – it happens because we’ve designed it to be that way. For those of us who don’t accept the idea of intelligent design in biology, that’s not true for brains. There is no clean “digital abstraction layer” in a brain – why should there be, unless someone designed it that way?

But evolution does design, and its designs do respect standard design principles. Evolution has gained by using both abstraction and modularity. Organs exist. Humans may be better in some ways than evolution at searching large design spaces, but biology definitely designs.

In a brain, for example, the digital is continually remodelling the physical – we see changes in connectivity and changes in synaptic strength as a consequence of the information being processed, changes, that as we see, are the manifestation of substantial physical changes, at the molecular level, in the neurons and synapses.

We have programmable logic devices, such as FPGAs, which can do exactly this.

Underlying all these phenomena are processes of macromolecular shape change in response to a changing local environment. .. This emphasizes that the fundamental unit of biological information processing is not the neuron or the synapse, it’s the molecule.

But you could make that same sort of argument about all organs, such as bones, muscles, lungs, blood, etc., and say we also can’t understand or emulate them without measuring and modeling them them in molecular detail. Similarly for the brain input/output systems that we have already emulated.

Determining the location and connectivity of individual neurons .. is necessary, but far from sufficient condition for specifying the informational state of the brain. .. The molecular basis of biological computation means that it isn’t deterministic, it’s stochastic, it’s random.

Randomness is quite easy to emulate, and most who see ems as possible expect to need brain scans with substantial chemical, in addition to spatial, resolution.

And that’s it, that is Jones’ “technical” critique. Since biological systems are made by evolution human design principles don’t apply, and since they are made of molecules one can’t emulate them without measuring and modeling at the molecular level. Never mind that we have actually seen design principles apply, and emulated while ignoring molecules. That’s what I call “generic skepticism”.

In contrast, I say brains are signal processing systems, and applying standard design principles to such systems tells us:

To manage its intended input-output relation, a signal processor simply must be designed to minimize the coupling between its designed input, output, and internal channels, and all of its other “extra” physical degrees of freedom. ..  To emulate a biological signal processor, one need only identify its key internal signal dimensions and their internal mappings – how input signals are mapped to output signals for each part of the system. These key dimensions are typically a tiny fraction of its physical degrees of freedom. Reproducing such dimensions and mappings with sufficient accuracy will reproduce the function of the system. This is proven daily by the 200,000 people with artificial ears, and will be proven soon when artificial eyes are fielded.

GD Star Rating
Tagged as: , ,

Caplan on Age of Em

As many have noted, ours in an era of ideological polarization. On topics where there are strong emotions, we tend to gravitate to extremes, and are less interest in intermediate positions. Which is a problem for my book; while most see it as too weird, others mostly see it as not weird enough. A  tech futurism minority expects to soon see very rapid progress in artificial intelligence and machine learning, and so see brain emulations as too slow and inefficient compared to the super-intelligence they foresee. And the majority to whom that seems pretty crazy also seem brain emulations as similarly crazy; they don’t care much if ems seem a bit less crazy.

At least future tech enthusiasts who think my book not weird enough are willing to write reviews to say so. But those who think my book too weird mostly stay silent; I’ve heard privately of many who were going to cover the book before they fully realized what it is about. So I thank my college Bryan Caplan for being willing to say what others won’t, in his critical review. His review is long, with ten criticisms. This response will also be long, going point by point.

Six of his ten objections seem to be mainly about my language. (His review is indented, and often contains book quotes; my replies are not.) Continue reading "Caplan on Age of Em" »

GD Star Rating
Tagged as: ,

Age of Em Criticism

My book’s topic seems to me so obviously important that I figure a reader’s main question must be whether he can trust me to actually know something on it. As a result, potential readers should be especially interested to hear criticisms; where do reviewers think my book gets it wrong? And as the book draws on many disciplines, readers should be especially interested in expert criticism, i.e., reviewers who find fault in an area they know well. Let us consider the reviews so far.

Three reviews so far can be seen as “main stream media.” At the Financial Times, journalist Sarah O’Connor calls the book “alluring” and “fascinating”, but notes that not everyone will accept the premise that ems are possible or “that current economic and social theories will hold in this strange new world.” However, the closest she gets to direct criticism is:

Some of the forecasts seem old-fashioned, like the notion that male ems will prefer females with “signs of nurturing inclinations and fertility, such as youthful good looks” while females will prefer males with “signs of wealth and status”.

At the Guardian, journalist Zoe Williams uses the book to direct readers to her critical question: “In a world without work, how do we distribute resources?” At Reason, journalist Ronald Bailey calls the book “fascinating”, and summarizes it in detail, but doesn’t otherwise evaluate it, other than to note that “other futurists have projected other pathways” that the future might take.

There are 2.5 reviews at widely read blogs. Economist Tyler Cowen likes the book, but cares less about its official topic than its indirect uses, such as a “Straussian commentary on the world we actually live in” and “A reminder of how strange everything is.” Economist Bryan Caplan has posted half of a review, on “What’s Right in Robin Hanson’s The Age of Em”; his other shoe has yet to drop.

Psychiatrist Scott Alexander really likes and highly recommends the book, though he worries that it is not weird enough, and he thinks I overstate my case on prior futurist accuracy. Alexander assigns low moral value to the scenario I describe, even though he sees it as full of happy complex creatures. He fears it will get even worse, leading to ems who are only ever focused on their particular work task, with no mind-wandering, breaks from work, or socializing.

There are also five reviews at other blogs. (There are also three reviews at Goodreads, and one more at Amazon, which don’t mention author expertise or offer field-specific criticisms.)

Futurist and computational neuroscientist Anders Sandberg calls the book a “very rich synthesis of many ideas with a high density of fascinating arguments,” but warns “most readers will disagree with large parts of it” and “many elements presented as uncontroversial will be highly controversial.” He himself only complains that I put in “too little effort bolstering the plausibility” of the basic idea of an emulation, a topic to which he has devoted in much effort.

Education reformer Neerav Kingsland calls the book “worth reading” though he would have rather I had written more fiction. He questions our ability to foresee the results of changes this big, and he questions my prediction of low wages: “Perhaps it would become taboo to replicate yourself, akin to teenage pregnancy?”

Private investor Peter McCluskey calls the book “quite valuable” though he notes my key assumptions could end up being wrong. He wishes I would have estimated wages relative to suspense more precisely, though he felt I was borderline overconfident overall, and thought I devoted too much attention to topics like swearing, relative to topics like democracy.

Economist Peter St Onge says “The pacing is fast, chock-full of interesting ideas to play with .. Hanson has done a fantastic job.” But he sees me as “too pessimistic” because the cost to run an em is very low compared to the cost to maintain a human today, and he just can’t see marginal product of human-like labor falling that low, no matter how many workers there are.

Physicist Richard Jones, in contrast, to the above nine reviewers, criticizes just about everything but my physics. He has long criticized Eric Drexler’s efforts to apply principles of mechanical engineering to tiny chemical systems. On Age of Em, he says:

Mind uploading .. will not be possible any time soon .. The brain .. is not the product of design, it is the product of evolution, and for this reason we can’t expect there to be such a digital abstraction layer. .. It would need to incorporate a molecularly accurate model of brain development and plasticity. .. His argument is that our understanding of human nature and the operations of human societies .. is now sufficiently robust that .. meaningful predictions can be made about the character of the resulting post-human societies. I don’t find this enormously convincing. .. Hanson often is simply unable to make firm predictions; this is commendably even-handed, but somewhat undermines his broader argument. .. How do we know what forager values actually were? Very few forager societies survived in any form into historical times, .. and what we know about their values is mediated by the biases of the anthropologists and ethnographers that recorded them.

So according to Jones, we can’t trust anthropologists to describe foragers they’ve met, we can’t trust economics when tech changes society, and familiar design principles fail for understanding brains and tiny chemical systems. Apparently only his field, physics, can be trusted well outside current experience. In reply, I say I’d rather rely on experts in each field, relative to his generic skepticism. Brain scientists see familiar design principles as applying to brains, even when designed by evolution, economists see economics as applying to past and distant societies with different tech, and anthropologists think they can understand cultures they visit.

Regarding O’Connor concerns on old-fashioned mate preferences I cited a literature on that, and regarding Alexander’s zero-leisure fears the book cites a literature on max productivity breaks and vacations. Regarding Kingsland and St Onge hopes for high wages, I’ll note that though most of history before the industrial era, taboos against having kids didn’t prevent marginal productivity from typically being very low.

So far I’d say that reviews give readers reasons to suspect my emphasis is at times off, but not strong reasons to fear that Age of Em is so wrong as to be not worth reading. But more reviews are yet to come.

GD Star Rating
Tagged as: ,

Alexander on Age of Em

If I ever have an executioner, I want him to be Scott Alexander. Alexander has such a winning way with words that I and his many fans enjoy him even when we disagree. I’d hardly notice my destination as his pleasing patter entranced me while we took the long way around to the gallows.

So I am honored that Alexander wrote a long review of Age of Em (9K words, 6% as long as the book), wherein he not only likes and recommends it, he also accepts pretty much all its claims within its main focus. That is, I present my book as being expert on the topic of what would actually happen if cheap ems were our next huge social change. Where Alexander disagrees is on two auxiliary topics, which I mention but on which I claim less expertise, namely how likely is this key scenario assumption, and how valuable is the resulting civilization I describe.

On the subject of value, Alexander leans forager (i.e., liberal) on the forager vs. farmer scale. He dislikes civilization evolving away from the behaviors and values of our forager ancestors, and today he partly blames this on capitalism. He doesn’t see our increase in numbers, comfort, and lifespan as sufficient compensation. (I think he’d like the book Against Civilization.) He says:

[Nick Land’s Ascended Economy] seems to me the natural end of the economic system. Right now it needs humans only as laborers, investors, and consumers. But robot laborers are potentially more efficient, companies based around algorithmic trading are already pushing out human investors, and most consumers already aren’t individuals – they’re companies and governments and organizations. At each step you can gain efficiency by eliminating humans, until finally humans aren’t involved anywhere. .. The Age of Em is an economy in the early stages of such a transformation. Instead of being able to replace everything with literal robots, it replaces them with humans who have had some aspects of their humanity stripped away. Biological bodies. The desire and ability to have children normally. ..

I envision a spectrum between the current world of humans and Nick Land’s Ascended Economy. Somewhere on the spectrum we have ems who get leisure time. A little further on the spectrum we have ems who don’t get leisure time. But we can go further. .. I expect [greatly reduced sex desire] would happen about ten minutes after the advent of the Age of Em .. Combine that with the stimulant use mentioned above, and you can have people who will never have nor want to have any thought about anything other than working on the precise task at which they are supposed to be working at any given time. ..

I see almost no interesting difference between an em world with full use of these tweaks and an Ascended Economy world. Yes, there are things that look vaguely human in outline laboring in the one and not the other, but it’s not like there will be different thought processes or different results. I’m not even sure what it would mean for the ems to be conscious in a world like this – they’re not doing anything interesting with the consciousness. .. If we get ems after all, I expect them to be lobotomized and drugged until they become effectively inhuman, cogs in the Ascended Economy that would no more fall in love than an automobile would eat hay and whinny.

Alexander seems to strongly endorse the usual forager value of leisure over work, so much so that he can’t see people focused on their work as human, conscious, or of any moral value. Creatures only seem valuable to him to the extent that they have sex, leisure time, minds wandering away from work, and desires to do things other than work.

This seems ironic because Scott Alexander is one of the most human and productive workers I know. He has a full time job as a psychiatrist, an especially demanding job, and in addition finds time to write frequent long careful analyses of many topics. I find it hard to see where he has that much time for leisure, and doubt he would in fact be substantially more productive overall if he took drugs to make him forget sex, mentally wander less, and focus more on his immediate tasks. He is exactly the sort of person an em economy would want many copies of, pretty much just as he is. Yet if we are to believe him, he only sees value in his brief leisure hours.

I see Alexander as having too little respect for the functionality of human behaviors and mind design. Yes, maximally competitive em-era behaviors and minds won’t be exactly like current ones. But that doesn’t necessarily mean one wants to throw out most existing behaviors and brain modules wholesale and start over from scratch. As these behaviors and modules all arose because they helped our ancestors be more competitive in some prior context, it makes more sense to try to repair, reform, and repurpose them.

For example, the robust productivity gains observed from workers who take breaks don’t seem to depend much on worker motivation. Breaks aren’t just about motivation; they are a deeply entrenched part of being productive. Similarly, wandering minds may take away from the current immediate task, but they help one to search for hidden problems and opportunities. Also, workers today who focus on just doing immediate tasks often lose out to others who attend more to building and managing social relations, as well as office politics. Love and sex can be very helpful in forming and maintaining relations.

Of course I’m not trying to offer any long term assurances, and it is quite reasonable to worry about what we will lose along with what we will gain. But since today most of the people we most respect and celebrate tend to be workaholics, I just can’t buy the claim that most of us today can’t find value in similarly productive and work-focused ems. And I just can’t see thoughtless workers being the most productive in the early em era of my book.

GD Star Rating
Tagged as: , ,

Rating Ems vs AIs

Everyone without exception believes his own native customs, and the religion he was brought up in, to be the best (Herodotus 440BC).

I’ve given about sixty talks so far on the subject of my book The Age of Em. A common response is to compare my scenario to one where instead of ems, it is non-emulation-based software that can first replace humans on most all jobs. While some want to argue about which tech may come first, most prefer to evaluate which tech they want to come first.

Most who compare to ems to non-em-AI seem to prefer the latter. Some say they are concerned because they see ems as having a lower quality of life than we do today (more on that below). But honestly I mostly hear about humans losing status. Even though both meat humans and ems can both be seen as our descendants, people identify more with meat as “us” and see ems as “them.” So they lament meat no longer being the top dog in-charge center-of-attention.

The two scenarios have many similarities. In both scenarios, meat humans must all retire, and robots take over managing the complex details of this new world, which humans are too slow, distant, and stupid to manage. The world economy can grow very fast, letting meat get collectively very rich, and which meat soon starves depends mostly on how well meat insures and shares among themselves. But it is hard to offer much assurance of long run stability, as the world can plausibly change so fast.

Ems, however, seem more threatening to status than other kinds of sophisticated capable machinery. You can more vividly imagine ems more clearly winning the traditional contests whereby humans compete for status, and then afterward acting superior, such as by laughing at meat humans. In contrast, other machines can be so alien that we may not be tempted to make status comparisons with them.

If, in contrast, your complaint about the em world is that ems have a lower quality of life, then you have to either care about something more like an average quality of life, or you have to argue that the em quality of life is below some sort of “zero”, i.e., the minimum required for a life to be worth living (or having existed). And this seems to me a hard case to make.

Oh I can see you thinking that em lives aren’t as good as yours; pretty much all cultures find ways to see their culture as superior. But unless you argue that em lives are much worse than the typical human life in history, then either you must say the typical human life was not worth living, or you must accept em lives as worth living. And if you claim that the main human lives that have been worth living are those in your culture, I’ll shake my head at your incredible cultural arrogance.

(Yes, some like Nick Bostrom in Superintelligence, focus on which scenario reduces existential risk. But even he at one point says “On balance, it looks like the risk of an AI transition would be reduced if whole brain emulation comes before AI,” and in the end he can’t seem to rank these choices.)

GD Star Rating
Tagged as: , ,

Problem, No Solution Taboo?

Three years ago I described the “What if Failure Taboo”:

A simple moral principle: when a future change is framed as a problem which we might hope our political system to solve, then the only acceptable reason to talk about the consequences of failing to solve that problem is to scare folks into trying harder to solve it. If you instead assume that politics will fail to solve the problem, and analyze the consequences of that in more detail, not to scare people but to work out how to live in that scenario, you are seen as expressing disloyalty to the system and hostility toward those who will suffer from that failure.

I suggested this could be an issue with my book Age of Em:

All of which seems bad news for my book, which mostly just accepts the “robots take over, humans lose wages and get sidelined” scenario and analyzes its consequences. No matter how good my reasons for thinking politics will fail to prevent this, many will react as did Nikola Danaylov, with outrage at my hostility toward the poor suffering losers.

This week I talked on my book to a sharp lively group organized by Azeem Azhar (author of the futurist newsletter Exponential View), and learned that this taboo may be worse than I thought. I tried to present the situation as something that you might consider to be a problem, but that while my analysis should enable better problem solving, I’ve personally focused on just describing this situation. Mixing up normative and positive discussions risks the positive being overshadowed by the normative, and positive claims seeming less reliable when mixed up with more disputable normative claims.

Even with this reframing, several people saw me as still violating the key taboo. Apparently it isn’t just taboo to assume that we’ll fail to solve a problem; it can also be taboo to merely describe a problem without recommending a solution. At least when the problem intersects with many strong feelings and moral norms. To many, neutral analysis just seems cold and uncaring, and suspiciously like evil.

GD Star Rating
Tagged as: ,

FT Reviews Age of Em

The first MSM review is in for Age of Em. It is by Sarah O’Connor (really!), in the Financial Times. She “gets” me:

Plenty of futurists and science fiction writers have toyed with the idea that the brains of particular humans could one day be scanned and “uploaded” into artificial hardware but Prof Hanson’s take is different. His aim is to use standard theories from the physical, human and social sciences to make forecasts about how this technological breakthrough would really change our world. ..

Some of the most profound questions .. are the questions on which Prof Hanson is most tentative and brief because he has set out deliberately to focus on the predictable and to avoid being “creative or contrarian”. This is reasonable enough and the book succeeds on its own terms. Still, it does leave a hole. Perhaps it will fall to the science fiction writers to fill it.

Yes, mine are first steps, and I hope others will further develop this scenario. O’Connor does have a few words of mild criticism:

The book is crammed full of such fascinating visions of an imagined future. Still, some readers will share criticisms the author says he has encountered. For any of this to seem plausible, one has to believe that we will invent the ability and be willing to scan and copy human brains. Not everyone will accept this.

One also has to believe that current economic and social theories will hold in this strange new world; that the “unknown unknowns” are not so great as to make any predictions impossible. Certainly, some of the forecasts seem old-fashioned, like the notion that male ems will prefer females with “signs of nurturing inclinations and fertility, such as youthful good looks” while females will prefer males with “signs of wealth and status”.

Yes, we can’t be at all certain of this scenario. But if it is worth having a hundred books on future scenarios, it is worth having books that analyze scenarios with only a 1% chance of happening.

Yes, you have to think that social scientists, like physicists, engineers, and computer scientists, actually know some generalities not hopelessly tied to the details of our particular time and culture. Such as our standard results on robust “old-fashioned” sex differences in long-term mate preferences:

Several standard sex differences replicated across cultures, including womens greater valuation of social status and mens greater valuation of physical attractiveness. (more; see also, also, also)

Even if such differences were weaker in nations with greater gender parity, they’d still remain. (But in fact such differences actually seem stronger there.) Yes, such differences may be in part caused by culture. But whatever their cause, they seem pervasive and robust enough to make them likely to continue on in an em world.

GD Star Rating
Tagged as:

The Labor-From-Factories Explosion

As I’ve discussed before, including in my book, the history of humanity so far can be roughly summarized as a sequence of three exponential growth modes: foragers with culture started a few million years ago, farming started about ten thousand years ago, and industry starting a few hundred years ago. Doubling times got progressively shorter: a quarter million years, then a millennia, and now fifteen years. Each time the transition lasted less than a previously doubling time, and roughly similar numbers of humans have lived during each era.

Before humans, animal brains brains grew exponentially, but even more slowly, doubling about every thirty million years, starting about a half billion years ago. And before that, genomes seem to have doubled exponentially about every half billion years, starting about ten billion years ago.

What if the number of doublings in the current mode, and in the mode that follows it, are comparable to the number of doublings in the last few modes? What if the sharpness of the next transition is comparable to the sharpness if the last few transitions, and what if the factor by which the doubling time changes next time is comparable to the last few factors. Given these assumptions, the next transition will happen sometime in roughly the next century. Within a period of five years, the economy will be doubling every month or faster. And that new mode will only last a year or so before something else changes.

To summarize, usually in history we see relatively steady exponential growth. But five times so far, steady growth has been disturbed by a rapid transition to a much faster rate of growth. It isn’t crazy to think that this might happen again.

Plausibly, new faster exponential modes appear when a feedback loop that was previously limited and blocked becomes is unlocked and strong. And so one way to think about what might cause the next faster mode after ours is to look for plausible feedback loops. However, if there thousands of possible factors that matter for growth and progress, then there are literally millions of possible feedback loops.

For example, denser cities should innovate more, and more innovation can find better ways to make buildings taller, and thus increase city density. More better tutorial videos make it easier to learn varied skills, and some of those skills help to make more better tutorial videos. We can go all day making up stories like these.

But as we have only ever seen maybe five of these transitions in all of history, powerful feedback loops whose unlocking causes a huge growth rate jump must be extremely rare. The vast majority of feedback loops do not create such a huge jump when unlocked. So just because you can imagine a currently locked feedback loop does not make unlocking it likely to cause the next great change.

Many people lately have fixated on one particular possible feedback loop: an “intelligence explosion.”  The more intelligence a creature is, the more it is able to change creatures like itself to become more intelligent. But if you mean something more specific than “mental goodness” by “intelligence”, then this remains only one of thousands of possibilities. So you need strong additional arguments to see this feedback loop as more likely than all the others. And the mere fact that you can imagine this feedback being positive is not remotely enough.

It turns out that we already know of an upcoming transition of a magnitude similar to the previous transitions, scheduled to arrive roughly when prior trends led us to expect a new transition. This explosion is due to labor-from-factories.

Today we can grow physical capital very fast in factories, usually doubling capital on a scale ranging from a few weeks to a few months, but we grow human workers much more slowly. Since capital isn’t useful without more workers, we are forced to grow today mainly via innovation. But if in the future we find a way to make substitutes for almost all human workers in factories, the economy can grow much faster. This is called an AK model, and standard growth theory says it is plausible that this could let the economy double every month or so.

So if it is plausible that artificial intelligence as capable as humans will appear in the next century or so, then we already know what will cause the next great jump to a faster growth mode. Unless of course some other rare powerful feedback loop is unlocked before then. But if an intelligence explosion isn’t  possible until you have machines at least as smart as humans, then that scenario won’t happen until after labor-from-factories. And even then it is far from obvious that feedback can cause one of the few rare big growth rate jumps.

GD Star Rating
Tagged as: , , ,

Is Forgotten Party Death?

Consider 3 possible space-time experience trajectories:

ForgottenPartyToSpurIn A, a person takes a drug at the start of a party which causes them to not remember that party the next day or anytime later. In C, an em splits off a short term “spur” copy which does a short term task and then ends.

Scenario B can be seen as like A, except that right after taking the drug the person quickly moves to a distant party location. Call this B1.  Alternatively, B can be seen as like C, except that the em original is archived and inactive while the spur works. Call this B2.

In my talks on Age of Em, I’ve heard many object to scenario C, but few object to A. In scenario A, few think they’d be stressed near the end of the party, thinking they are about to “die.” Yet many see scenario C as a “death,” and claim the em spur would refuse to do their assigned task, and instead fight fiercely to keep going.

Scenario B is designed to be intermediate between A and B, and so to force a choice between conflicting intuitions. Would you really see B2 as “death” but B1 as no big deal, even though they have the same space time structure of experiences? If not, I think you should admit either that A is “death”, or that C is not. Or explain what matters besides the space-time structure of experience.

I confidently predict that ems see all of these as no big deal, because a competitive em world selects for ems who are more productive, and a willingness to create short term spurs is quite productive in the em world.

Note that this issue engages similar space vs time morality intuitions as these three prior posts.

GD Star Rating
Tagged as: , ,