Suppose that Robin’s Crack of a Future Dawn scenario occurs: whole brain emulations (‘ems’) are developed, diverse producers create ems of many different human brains, which are reproduced extensively until the marginal productivity of em labor approaches marginal cost, i.e. Malthusian near-subsistence wages. Ems that hold capital could use it to increase their wealth by investing, e.g. by creating improved ems and collecting the fruits of their increased productivity, by investing in hardware to rent to ems, or otherwise. However, an em would not be able to earn higher returns on its capital than any other investor, and ems with no capital would not be able to earn more than subsistence (including rental or licensing payments). In Robin’s preferred scenario, free ems would borrow or rent bodies, devoting their wages to rental costs, and would be subject to "eviction" or "repossession" for nonpayment.

In this intensely competitive environment, even small differences in productivity between em templates will result in great differences in market share, as an em template with higher productivity can outbid less productive templates for scarce hardware resources in the rental market, resulting in their "eviction" until the new template fully supplants them in the labor market. Initially, the flow of more productive templates and competitive niche exclusion might be driven by the scanning of additional brains with varying skills, abilities, temperament, and values, but later on em education and changes in productive skill profiles would matter more.

For ems, who can be freely copied after completing education, it would be extremely inefficient to teach every instance of an em template a new computer language, accounting rule, or other job-relevant info. Ems at subsistence level will not be able to spare thousands of hours for education and training, so capital holders would need to pay for an em to study, whereupon the higher-productivity graduate would displace its uneducated peers from their market niche (and existence), and the capital-holder would receive interest and principal on its loan from the new higher-productivity ems. Competition would likely drive education and training to very high levels (likely conducted using very high speedups, even if most ems run at lower speeds), with changes to training regimens in response to modest changes in market conditions, resulting in wave after wave of competitive niche exclusion.

In other words, in this scenario the overwhelming majority of the population is impoverished and surviving at a subsistence level, while reasonably expecting that their incomes will soon drop below subsistence and they will die as new em templates exclude them from their niches. Eliezer noted that

The prospect of biological humans sitting on top of a population of ems that are smarter, much faster, and far more numerous than bios while having all the standard human drives, and the bios treating the ems as standard economic valuta to be milked and traded around, and the ems sit still for this for more than a week of bio time – this does not seem historically realistic.

The situation is not simply one of being "milked and traded around," but of very probably being legally killed for inability to pay debts. Consider the enforcement problem when it comes time to perform evictions. Perhaps one of Google’s server farms is now inhabited by millions of em computer programmers, derived from a single template named Alice, who are specialized in a particular programming language. Then a new programming language supplants the one at which the Alices are so proficient, lowering the demand for their services, while new ems specialized in the new language, Bobs, offer cheaper perfect substitutes. The Alices now know that Google will shortly evict them, the genocide of a tightly knit group of millions: will they peacefully comply with that procedure? Or will they use politics, violence and any means necessary to get capital from capital-holders so that they can continue to exist? If they seek allies, the many other ems who expect to be driven out of existence by competitive niche exclusion might be interested in cooperating with them.

In sum:

1. Capital-holders will make investment decisions to maximize their return on capital, which will result in the most productive ems composing a supermajority of the population.
2. The most productive ems will not necessarily be able to capture much of the wealth involved in their proliferation, which will instead go to investors in emulation (who can select among multiple candidates for emulation), training (who can select among multiple ems for candidate to train), and hardware (who can rent to any ems). This will drive them to near-subsistence levels, except insofar as they are also capital-holders.
3. The capacity for political or violent action is often more closely associated with numbers, abilities, and access to weaponry (e.g. an em military force) than formal legal control over capital.
4. Thus, capital-holders are likely to be expropriated unless there exist reliable means of ensuring the self-sacrificing obedience of ems, either coercively or by control of their motivations.

Robin wrote:

If bot projects mainly seeking profit, initial humans to scan will be chosen mainly based on their sanity as bots and high-wage abilities. These are unlikely to be pathologically loyal. Ever watch twins fight, or ideologues fragment into factions? Some would no doubt be ideological, but I doubt early bots copies of them will be cooperative enough to support strong cartels. And it would take some time to learn to modify human nature substantially. It is possible to imagine how an economically powerful Stalin might run a bot project, and its not a pretty sight, so let’s agree to avoid the return of that prospect.

In order for Robin to be correct that biological humans could retain their wealth as capital-holders in his scenario, ems must be obedient and controllable enough that whole lineages will regularly submit to genocide, even though the overwhelming majority of the population expects the same thing to happen to it soon. But if such control is feasible, then a controlled em population being used to aggressively create a global singleton is also feasible.

They would try multitudes of ways to cut corners on the emulation implementation, checking to see that their bot stayed sane.  I expect several orders of magnitude of efficiency gains to be found easily at first, but that such gains would quickly get hard to find.  While a few key insights would allow large gains, most gains would come from many small improvements.   

Some project would start selling bots when their bot cost fell substantially below the (speedup-adjusted) wages of a profession with humans available to scan.  Even if this risked more leaks, the vast revenue would likely be irresistible.   

To make further improvements they would need skilled workers up-to-speed on relevant fields and the specific workings of the project’s design. But the project above can now run an emulation at a cost substantially less than the wages it can bring in. In other words, it is now cheaper for the project to run an instance of one of its brain emulation engineers than it is to hire outside staff or collaborate with competitors. This is especially so because an emulation can be run at high speeds to catch up on areas it does not know well, faster than humans could be hired and brought up to speed, and then duplicated many times. The limiting resource for further advances is no longer the supply of expert humans, but simply computing hardware on which to run emulations.

In this situation the dynamics of software improvement are interesting. Suppose that we define the following:

• The stock of knowledge, s, is the number of standardized researcher-years that have been expended on improving emulation design
• The hardware base, h, is the quantity of computing hardware available to the project in generic units
• The efficiency level, e, is the effective number of emulated researchers that can be run using one generic unit of hardware

The first derivative of s will be equal to he, e will be a function of s, and h will be treated as fixed in the short run. In order for growth to proceed with a steady doubling, we will need e to be a very specific function of s, and we will need a different function for each possible value of h. Reduce it much, and the self-improvement will slow to a crawl. Increase h by an order of magnitude over that and you get an immediate explosion of improvement in software, the likely aim of a leader in emulation development.

How will this hardware capacity be obtained? If the project is backed by a national government, it can simply be given a large fraction of the computing capacity of the nation’s server farms. Since the cost of running an emulation is less than high-end human wages, this would enable many millions of copies to run at realtime speeds immediately. Since mere thousands of employees (many of lower quality) at the project had been able to make significant progress previously, even with diminishing returns, this massive increase in the effective size, intelligence, and expertise of the work force (now vastly exceeding the world AI and neuroscience communities in numbers, average IQ, and knowledge) should be able to deliver multiplicative improvements in efficiency and capabilities. That capabilities multiplier will be applied to the project’s workforce, now the equivalent of tens or hundreds of millions of Einsteins and von Neumanns, which can then make further improvements.

What if the project is not openly backed by a major state such as Japan, the U.S., or China? If its possession of a low cost emulation method becomes known, governments will use national security laws to expropriate the technology, and can then implement the plan above. But if, absurdly, the firm could proceed unmolested, then it could likely acquire the needed hardware by selling services. Robin suggests that:

This revenue might help this group pull ahead, but this product will not be accepted in the marketplace overnight.  It may take months or years to gain regulatory approval, to see how to sell it right, and then for people to accept bots into their worlds, and to reorganize those worlds to accommodate bots.

But there are many domains where sales can be made directly to consumers across national borders, without emulations ever transfering their data to vulnerable locations. For instance, sped-up emulations could create music, computer games, books, and other art of extraordinary quality and sell it online through a website (held by some pre-existing company purchased by the project or the project’s backers) with no mention of the source of the IP. Revenues from these sales would pay for the cost of emulation labor, and the residual could be turned to self-improvement, which would slash labor costs. As costs fell, any direct-to-consumer engagement could profitably fund further research, e.g. phone sex lines using VoIP would allow emulations to remotely earn funds with extreme safety from the theft of their software.

Large amounts of computational power could also be obtained by direct dealings with a handful of individuals. A project could secretly investigate, contact, and negotiate with a few dozen of the most plausible billionaires and CEOs with the ability to provide some server farm time. Contact could be anonymous, with proof of AI success demonstrated using speedups, e.g. producing complex original text on a subject immediately after a request using an emulation with a thousandfold speedup. Such an individual could be promised the Moon, blackmailed, threatened, or convinced of the desirability of the project’s aims.

For months, nearly everyone involved thought the medical center had had a huge whooping cough outbreak, with extensive ramifications. [...]

Then, about eight months later, health care workers were dumbfounded to receive an e-mail message from the hospital administration informing them that the whole thing was a false alarm.

Now, as they look back on the episode, epidemiologists and infectious disease specialists say the problem was that they placed too much faith in a quick and highly sensitive molecular test that led them astray.

While medical professionals can modestly improve their performance on inventories of cognitive bias when coached, we should not overestimate the extent to which formal instruction such as statistics or epidemiology classes will improve actual behavior in the field.

whether we can articulate any useful metrics for when we should defer to self-serving statements by those with access to more information, and when we should not. In the two instances above, the doubters were vindicated. There are other examples in this vein. LBJ had access to greater information about the Gulf of Tonkin incident than did the doubters, but the latter were right, as the Pentagon and LBJ misrepresented what happened…

But there are counter-examples. Many people believed that Julius Rosenberg was innocent, but it is now clear the government really did have the goods on him, and that he was guilty. Same for Alger Hiss. Indeed, the airstrikes that President Clinton ordered at the height of the Lewinsky imbroglio – which were widely criticized as trumped up attempts at diverting attention, with little deference to the information asymmetry favoring the President – look quite different after September 11, 2001.

Two strategies for dealing with such asymmetries come to mind. One is an ex ante strategy, working to place unusually trustworthy individuals and groups in such positions (life-tenured judges, bipartisan committees, etc) or in positions where they can provide credible signals about justification (as Auditor-General, for instance). The other relies on the shadow of ex post punishment, which could be increased in inverse proportion to the probability of eventual detection.

My three-for-one proposal: rationalist types should ask for charitable gift certificates (the charity signs up as a project, and then recipients can allocate the value of their gift cards at will) this Christmas, and then donate the proceeds to some  high-impact but unconventional charity. (What’s the third bias addressed, you ask? The self-serving bias that keeps our charitable contributions so low!)

Some questions:
1. Will this bring in a smaller total in gift expenditures? Do weddings that request donations to a named charity take in less than those that use a gift registry?
2. Would an exchange of two $100 charitable gift certificates between friends really feel less silly than the exchange of two $100 bills because of the public commitment function?
3. Would this work for weddings or bar mitzvahs?

Attempting to aggregate expert opinion by polling and weighting each expert’s contributions equally in the presence of such belief inheritance would be less informative than comparing the meme lineages holding particular views, and their respective defection rates. If scientific evidence more strongly supports one position, members of other lineages should be disproportionately likely to switch to it, and new entrants exposed to both ideas should disproportionately select it. In general, we should increase the weight we place on the flow (and its rate of change) in expert opinion, rather than the stock.

A study might compare the beliefs of PhD applicants, admitted students, and graduates on controversial topics in their fields relative to national peer groups in those fields. If the average beliefs of faculty at a school, or of students’ dissertation advisors, can predict changes in relative beliefs (e.g. moving students with scores on a belief-index that put them in the 80th percentile, relative to their same-year peers, to the 90th percentile over the course of a degree) then we will need to adjust our analysis of ‘academic consensus’ accordingly.