Em Software Engineering Bleg

Many software engineers read this blog, and I’d love to include a section on software engineering in my book on ems. But as my software engineering expertise is limited, I ask you, dear software engineer readers, for help.

“Ems” are future brain emulations. I’m writing a book on em social implications. Ems would substitute for human workers, and once ems were common ems would do almost all work, including software engineering. What I seek are reasonable guesses on the tools and patterns of work of em software engineers – how their tools and work patterns would differ from those today, and how those would vary with time and along some key dimensions.

Here are some reasonable premises to work from:

  1. Software would be a bigger part of the economy, and a bigger industry overall. So it could support more specialization and pay more fixed costs.
  2. Progress would have been made in the design of tools, languages, hardware, etc. But there’d still be far to go to automate all tasks; more income would still go to rent ems than to rent other software.
  3. After an initial transition where em wages fall greatly relative to human wages, em hardware costs would thereafter fall about as fast as non-em computer hardware costs. So the relative cost to rent ems and other computer hardware would stay about the same over time. This is in stark contrast to today when hardware costs fall fast relative to human wages.
  4. Hardware speed will not rise as fast as hardware costs fall. Thus the cost advantage of parallel software would continue to rise.
  5. Emulating brains is a much more parallel task than are most software tasks today.
  6. Ems would typically run about a thousand times human mind speed, but would vary over a wide range of speeds. Ems in software product development races would run much faster.
  7. It would be possible to save a copy of an em engineer who just wrote some software, a copy available to answer questions about it, or to modify it.
  8. Em software engineers could sketch out a software design, and then split into many temporary copies who each work on a different part of the design, and talk with each other to negotiate boundary issues. (I don’t assume one could merge the copies afterward.)
  9. Most ems are crammed into a few dense cities. Toward em city centers, computing hardware is more expensive, and maximum hardware speeds are lower. Away from city centers, there are longer communication delays.

Again, the key question is: how would em software tools and work patterns differ from today’s, and how would they vary with time, application, software engineer speed, and city location?

To give you an idea of the kind of conclusions one might be tempted to draw, here are some recent suggestions of François-René Rideau:

Amdahl’s Law applies to ems. So does Gustafson’s Law, and programming em’s will thus use their skills to develop greater artifacts than is currently possible.

Now, if you can afford to simulate a brain, memory and massive parallelism are practically free. Fast ems will thus use programming languages that are very terse and abstract, yet extremely efficient in a parallel setting. Much more like APL than like Java. Since running the code is expensive, bugs that waste programmer latency will be much more expensive than now. In some ways more like the old days of batch processing with punch cards than the current days of fancy interactive graphical interfaces — yet in other ways, more modern and powerful than what we use now. Any static or dynamic check that can be done in parallel with respectively developing or running the code will be done — the Lisp machine will be back, except it will also sport fancy static type systems. Low-level data corruption will be unthinkable; and even what we currently think of as high-level might be low-level to fast em programmers: declarative meta-programming will be the norm, with the computer searching through large spaces of programs for solutions to meta-level constraints — machine time is much cheaper than brain time, as long as it can be parallelized. Programmers will be very parsimonious in the syntax and semantics of their programming languages and other programs; they will favor both high-falluting abstraction and ruthless efficiency over any kind of fanciness. If you don’t grok both category theory and code bumming, you won’t be the template for the em programmer of the future. Instead imagine zillions of Edward Kmett’s programming in parallel.

At high-speed, though, latency becomes a huge bottleneck of social programming, even for these geniuses — and interplanetary travel will only make that worse. Bug fixes and new features will take forever to be published then accepted by everyone, and every team will have to develop in parallel its own redundant changes to common libraries: what to us are simple library changes to fast ems might be as expensive as agreeing on standard document is to us. (more)

Can you improve on that? If so, let’s hear it! Since I seek robust predictions, I’ll be especially interested to hear of implications that you mostly agree on. And for this post, I’m much less interested in comments from those without computer expertise.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • IMASBA

    This may be a silly question but since EMs are computer programs themselves shouldn’t it be very easy to make them intuitively understand some optimized future programming language and have them write code with simple thoughts (similar to how biological humans intuitively understand tactile experience, ie. building stuff with our hands)?

    • http://overcomingbias.com RobinHanson

      No it wouldn’t be any easier for ems to understand code.

      • IMASBA

        Fair enough, it would be more of a thing for AIs that were built/developed from the ground up, as others are commenting, but then again how would you implement an EM without translating brain activity into code (them running on code at some level should make it easy to add some module that translates conscious thought into code)?

      • http://don.geddis.org/ Don Geddis

        Humanity could simulate a brain (= create an EM), without having much of any idea how it works. And thus without much of any ability to change or enhance it.

        After all, we can already interface to organic human brains. (E.g. build artificial limbs that are somewhat controlled by thought. Or: http://www.hongkiat.com/blog/brain-controlled-gadgets/ ) Yet the idea that you would “translate thought into code” with anything like the efficiency of using fingers on a keyboard, isn’t even in the realm of realistic imagination.

      • IMASBA

        The gadgets you linked to don’t use the standard mental commands our limbs would use. They rely on a person training to create signals that are coherent enough to be picked up by the gadget’s sensors (you might have to imagine the color blue very strongly to make the gadget move forward but it varies from person to person what exactly you have to be thinking about to create a certain signal, at first you just sort of have to try out different things to see what creates a reaction). With enough training people can master this but I doubt it’ll ever be as accurate as the mental commands we’re born with, I could be wrong though. people have been trained to adapt to upside down visual input very successfully. Are EMs supposed to permanently interact with the world in such a way? If they are it could be that they’ll never be as good with a robot hand as we our with our own human hands. Now why can’t we just create more sensitive gadgets? In practice it’s impossible to connect all the nerve endings to an artificial limb in the same way they would in a real limb, it would be complicated surgery on a microscopic level and many times over. Why can’t we just create more sensitive mind reading gadgets? Because in practice it’s impossible to read signals with high resolution (with only small numbers of neurons and synapses per pixel). But in a computer simulation you could do those things: you could have virtual limbs that feel real and you could monitor the brain activity in such a high resolution at some output point that you could guess the words the EM is trying to say. If you can directly readout the intentions of the vocal and motory areas you could combine that with intelligent software that can guess the algorithm the EM is trying to describe and if you increase signal resolution you might very well be able to read the EMs conscious thoughts directly after a period of calibration and if the EM is cooperating (which it would be when writing software). Our knowledge of the brain definitely has to be improved before attempting to create an EM, for example we would have to know, and very accurately, how brain activity restructures the brain. We’d also need to know much more about how memories are stored and retrieved (since there are signs that the internal structure of cells in the brain plays a role and because we can’t simulate every atom of every neuron and synaps we really need to know exactly how detailed the simulation have to be for memories to be preserved), having that knowledge would already allow artificial uploading of memories (or, more efficiently, allow coupling to a central databse) and make every EM knowledgeable in everything (though of course they can still differ in recall speed, making connections and creativity).

        So to make EMs we need a fair bit more understanding of the brain than we do now and that understanding would probably enough for mind reading and memory uploading. of course we still might not know how a brain becomes conscious and what the role of every individual cell is, if that’s what you meant by “not having much of any idea how it works” then I agree, but the point is we only need to scratch the surface to be able to do some spectacular sci-fi things with EMs.

      • http://don.geddis.org/ Don Geddis

        Sure, we’re not yet ready to create EMs, so we’d need to know more about the brain. But what we mostly need to know more about how individual neurons work. Your hypothesis, that the science/technology needed to create EMs “would probably be enough for mind reading and memory uploading”, is almost certainly wrong. The whole point of the EM scenario, is that we might be able to construct human-level computers, WITHOUT needing to figure out how thinking works. (The alternative FOOM scenario is deliberately created AIs, instead of blindly copying human brains.)

        Meanwhile, your description of human innate abilities also seems confused. “the standard mental commands our limbs would use … ever be as accurate as the mental commands we’re born with.” Have you ever spent much time around infants? They have essentially no mental control of their bodies. You seem to be confusing your current mental mastery of your body, with some idea that it must be a qualitatively different kind of mental control. But in fact the “rely on a person training to create signals that are coherent enough” is actually the exact same way that infants learn to control their limbs. Their brains throw out a bunch of neuron static, once in awhile they get lucky that some random pattern moves part of their body, and with a lot of practice and pattern recognition, they eventually train reliable subconscious control of their limbs. And, it’s not just babies: adults do the same thing, when learning to play sports or drive a car. It’s true that these natural applications typically have higher neural bandwidth than the current “mind control” technology. But that’s just a quantitative difference in sensitivity, not a qualitative difference in the process itself.

      • IMASBA

        “The whole point of the EM scenario, is that we might be able to construct human-level computers, WITHOUT needing to figure out how thinking works.”

        I didn’t say we need to understand how thinking works, just what the brain activity that accompanies conscious thoughts looks like. There are already experiments where visual images can be read out with a limited degree of accuracy after the device has been calibrated to a person (the person does not need any training). With EMs this could become much more detailed because you could actually measure the brain activity with as much resolution as you want to. In fact you’d already need a high resolution to make speech possible between EMs.

        More knowledge of neurons and their interactions (we would also need to know how connections change) would probably already tell you a lot about memory.

        “You seem to be confusing your current mental mastery of your body, with some idea that it must be a qualitatively different kind of mental control. But in fact the “rely on a person training to create signals that are coherent enough” is actually the exact same way that infants learn to control their limbs.”

        I’d think that human muscles are hardwired to use the most intuitive and efficient brain activity and EMs would be based on adults who are already entrenched. Then again, you might be right, I’m not knowledgeable enough to definitively say an adult mind could not adapt well enough.

      • foldl

        It’s easier to understand if it never has to be forgotten.

        Code is always harder to understand the more you differ from the one who wrote it. This includes looking at your own code from past-you.

        If backup/restore/storage of ems is cheap I’d expect an em to be saved to version control with every commit, so it could pick up wherever it left off whenever requirements change or bugs are found.

        So I’d expect it to be just as likely that instead of the glittering towers of abstraction, the ems would instead tend their own row in an otherwise-incomprehensible ball of spaghetti code.

      • foldl

        Beaten by posts above, but also adding software paid for by maintenance contract in the form of rental.

    • endril

      That’s some Matrix-y thinking. Ems aren’t any more attuned to code than biological humans just because they’re instantiated in transistors and not proteins.

    • Oscar Cunningham

      Ems are copies of biological brains, and so have our intuitions rather than the ones that an intelligence designed for computers might have.

  • ShardPhoenix

    As a software engineer I’d say something like, “You’re making too many assumptions here about how this thing is going to work. Let’s just get a simple prototype working first.”

  • ShardPhoenix

    At any rate it’s difficult to predict what will happen with programming languages (even assuming such things will still exist) because as far as I can tell existing programming languages aren’t yet on an efficiency frontier. That is, it’s not just a matter of tradeoffs of development speed vs. runtime speed vs. safety and so on, but that we’ll probably see languages that are more or less strictly better than existing ones be developed over the next decade or so. So we can’t really say what the future tradeoffs will be yet.

  • Doug

    (I’ve made a few, relatively disjoint observations, that I split into multiple posts for the purpose of threading)

    You should consider Fred Brooks’ observation in the Mythical Man Month: 1 programmer working for 10 months can produce much more working software than 10 programmers working together for 1 month. Software development is a very poor parallel task. The reason large software teams exist today is because of the time constraints of having a single developers work on large projects. A modern OS contains on the order of 100 million lines of code (LoC), whereas a very skilled developer might be able to write and debug 50 thousand LoC/yr. A single developer would require millennia of subjective time, so we sacrifice worker efficiency to speed up the software development cycle.

    As team size scales the efficiency drops off. In a thousand person team, that metric probably falls to 5 thousand LoC/yr (as well as final quality greatly decreasing). Things would definitely move faster in the EM era, but the global software development cycle would not be thousands of time faster. One would expect even giant software projects, to be done by single EM programmers, or small teams, over enormously long periods of subjective time. Since the costs of parallelizing software development is so high, one would expect that software engineer EMs would pay up to run on the fastest, highest-cost hardware. For the same development constraints, using faster EM speedups would allow smaller teams to develop with more subjective time.

    • http://overcomingbias.com RobinHanson

      Yes the cost of coordinating programmers would tempt one to run fewer of them faster. On the other hand, the faster the programmers the more sluggish the hardware will feel to them. Is it clear that the tradeoff here favors fast programmers?

      • David Condon

        That sounds like something you could model.

      • Doug

        Hmm, not sure if sluggish hardware is the problem that you think it is. If the EM is running on an expensive 10X faster processor shouldn’t his developer tools be just as sped up as his mind. It’s true there are some development process delays that are latency bound, not compute bound: reading data off disk, network requests to StackOverflow, waiting to accumulate random entropy, etc.

        But it’s not like the EM is going to sit around and wait for a packet off the network. An IO-blocked EM process would simply suspend until the data arrived. The subjective aeons that would pass for a running EM, would feel like an instant for a suspended one. More importantly the owner doesn’t pay for the fast, expensive EM simulator to idle for long blocks. (There’s a minor cost to context switching, but it seems assuredly trivial).

        The server farm can simply run another process while the EM is suspended. So I don’t see how sluggishness imposes any substantial difference in compute cost between fast and slow EMs.

      • David Condon

        Trivial to you may not be trivial to the em. If the em is currently using 90% of processing power, would suspension still feel instantaneous? The em is sharing processing power with the developer tools, remember? The developer tools would only be fast enough to record the em’s work most of the time. Doubling the number of ems would be like halving keyboard input latency from the em’s perspective. Imagine working at a keyboard that only recorded 15 characters per minute, and compiling a program took an hour while you waited to be suspended and then unsuspended.

      • http://overcomingbias.com RobinHanson

        The issue is that emulating brains is a much more parallel task, allowing it to be sped up more than more serial tasks. That is what makes the hardware seem sluggish on those serial tasks.

      • James Babcock

        To the extent that these tools are slow, it is entirely due to technical debt: things which could be made fast if someone put in the time, but which aren’t worth it in today’s world. If there were a serious effort to make devtools fast, including exploiting the parallelism, then that would leave a ludicrous amount of headroom to speed up the ems – a 10^6x speedup programmer could work on 2014 hardware with only mild annoyance.

        (I write software development tools in my spare time.)

      • http://overcomingbias.com RobinHanson

        I agree there isn’t an issue for tools that can be made very parallel. The issue is tools with big serial parts.

      • Doug

        I see. In that case I think that the returns to smaller, faster EM teams is high enough that it will significantly push the equilibrium to light weight tools. Even if light weight tools are inferior, they’re probably much less inferior to managing large software development teams.

        That probably means modern IDEs and GUI environments are out in favor of small text editors like vim and command line environments. Compiling in most languages is a huge sequential cost, so you’ll probably see a very heavy emphasis on developing languages like Go, that are designed to compile very fast. High-performance compilers and debuggers may quickly become the most valuable software in the world.

        Network requests looking up questions on the Internet is a also huge serialized cost, so either EM instances will be pre-trained to have memorized a lot of things relevant to the project. Or StackOverflow will be very locally cached near EM processors.

      • James Babcock

        I don’t think there are any widely-used tools with inherently serial parts (as opposed to being serial just because parallelizing takes work).

      • http://overcomingbias.com RobinHanson

        Others have suggested compilers and debuggers as inherently serial.

      • James Babcock

        I say, coming from a fairly detailed knowledge of compiler architecture, that compilers for most programming languages are not inherently serial. (Some programming languages are exceptions, but they are exceptions for bad reasons, not good ones.)

        Debuggers can be exactly as parallel as the thing that is being debugged, and no more.

      • lump1

        My thought was similar to Doug’s: If an EM has the resources to run at 10X, it surely has the resources to also run its workstation at 10X. But if this is so, wouldn’t the really important software problems be assigned to small all-star teams of elite programmers who are accelerated to 1000X, or whatever the hardware can bear? They could do 10 years of coding in just 3 days. And would not the same logic apply to elite teams of scientists, engineers, architects, etc., who are working on unparalellizable problems of global importance? Since lab experiments could not be correspondingly accelerated, their work would have to be mostly conceptual, but lots of such work would be extremely valuable. I guess a life at 1000x could get boring and lonely, since the slow world around you would produce only 3 days of novelty in ten subjective years. Maybe there would be elite communes of government-sponsored hyperspeed hotshots hanging out together and cranking out code, weapons designs, movie scripts, blueprints, etc. That would make a neat short story.

  • Punchcard Pete

    It seems to me that what this does is to dramatically decrease the cost of programmer time relative to the cost of computing resources (CPU time, memory and storage and bandwidth requirements). This is directly opposed to the sustained trend since the invention of the general purpose computer, where improvements in hardware have far outpaced all other improvements, making the time and ability of the programmer increasingly the limiting resource. Pretty much all improvements in the art and science of programming have been aimed not at making software better but at making it easier and cheaper for a greater number of more average programmers to write software of acceptable quality more quickly. Thus, even basic innovations such as the use of high-level computer languages and the promotion of code reuse through shared libraries (and in fact the use of operating systems at all) tend to dramatically reduce the difficulty and cost of developing acceptable software, not the quality of the resulting software as measured by the use of computing resources. This is obviously an excellent trade-off in the current world. In the proposed EM world, on the other hand, it would seem that computing resources (which would otherwise be available for running more EMs) would instantly become the limiting factor and not the availability of highly skilled and intelligent programmer.

    I suggest that the best model for this world that we have is the very early days of computing. In those days, when computing resources in environments such as Colossus at Bletchley Park or the computer on the Apollo LEM were the scarce limiting factor, software was lovingly custom-crafted specifically for the intended application by teams of extremely skilled programmers. They used machine-specific instructions instead of a general purpose language, and didn’t reuse any code at all, nor did they consume computing resources with an ‘operating system’ designed to abstract away common tasks, but only at an acceptable level of efficiency. In the same way, I believe that in the EM world, each application would be individually crafted by a team of master programmers entirely in machine code. In fact, important applications would almost certainly not use general-purpose hardware but would be co-designed as custom hardware and software to work optimally together for the purpose. (Unimportant applications that would not operate enough to repay such optimization would consist of just telling an EM what to do). Everything we currently know and teach regarding software engineering would be entirely counter-productive in such an environment.

    It is obvious that the application that would best repay any increase in efficiency and that therefore would be most carefully and continually optimized would be the the EM platform itself. I think this makes this a dynamically unstable situation that I would expect to collapse almost immediately into a Singularity as the continual exponential improvement of the EM platform resulted in superhuman intelligences whose activities our semi-evolved ape brains are entirely incapable of predicting.

    But for a brief vanishing time until that happened, the economic effect of EMs would be as though tailors were suddenly made relatively much, much less expensive relative to cloth. Any suits that were made would be beautifully custom tailored by experts for the wearer, and off-the-rack mass-produced clothing would be rendered entirely obsolete.

    Thanks for the interesting thought experiment!

    • http://overcomingbias.com RobinHanson

      I agree that a fall in programmer cost relative to computer hardware costs moves us away from abstraction and code reuse, and toward lower level context specific coding. But I’m not so sure just how far that move will be.

    • http://overcomingbias.com RobinHanson

      I agree lots of effort would go into optimizing the emulation process. But I expect strongly diminishing returns to that, at least until we understand brain design at a deep level, even then I don’t see that big a change.

  • Doug

    One of the overriding maxims that guides software development decisions today is that developer time is much more expensive than CPU time. Many existing programs and systems are quite capable of running using much less computing resources. The reason they don’t is because optimizing them would cost more in developer costs than it would save in computing costs. This biases existing software tools to use high levels of abstraction. A concrete example in garbage collection. Manual memory management is faster if done properly, but automated garbage collection requires much less effort from the programmer.

    EM wages shifts the relative balance. Developer costs fall relative to computing costs. One would expect that the EM future would do much more work with lower-level software tools that run closer to the bare metal. That would reverse the current decades long trend of software systems using increasingly higher and layered levels of abstraction. Another consideration is that developer system tools would “crowd” the same computing resources that the developer minds are consuming.

    A particularly intense development tool would mean a slower EM speedup. So we’d also expect to see a particular push towards very optimized and resource-sparse editors, compilers, linkers, debuggers, etc. Alternatively we might see more programming done in a more batch manner, where large segments of code are moved towards far-away cheap processors, rather than the current model of high interactivity.

    • http://overcomingbias.com RobinHanson

      As I said to Pete, I agree that one would move away from abstraction to more intense attention to efficiency. But I don’t see why editors, debuggers, etc. need to be that sparse if even luxurious ones are still vastly cheaper than running an em mind.

      • Doug

        On further reflection, I think you’re right. It’s hard to imagine any scenario where these tools take anything more than a trivial fraction of the cost of EM sims. Even today many programmers use tools that are programmed to run on 1980s hardware.

  • Doug

    Generally developers who’ve worked on a codebase have much higher efficiency when revisiting that codebase in some manner. This can apply to debugging or extending the codebase, but the much more common scenario is interfacing with it in another project. Probably a thousand times more developers will call an OS’s system libraries, than will actually modify the OS code in anyway.

    Software guidelines highly emphasize clean, simple, modular and abstracted external interfaces. Even within a project, sub-modules are expected to be designed in a way that’s decoupled as much as possible. Most developers building on top of an OS, database or server, have very little knowledge about the system’s internal workings. Therefore great care is taken to making downstream applications able to interact in a largely transparent manner.

    In an EM world, the minds of developers who’ve worked on upstream systems can be snapped and copied. Anyone who wants to build an IPhone app can acquire a copy of an EM that worked as a core iOS developer. The need to build modular interfaces is greatly reduced. Modularity is not free, it costs both in developer time as more “glue” has to be applied when interfacing between systems, as well as reduced opportunities for computational optimization. One would expect EM software environments to be more tightly coupled than is the norm today.

    • http://overcomingbias.com RobinHanson

      Yes, the option to used saved copies of those who wrote key software modules should make it easier to usefully break modularity abstractions.

  • Joshua Brulé

    If making em copies is sufficiently easy, I’d imagine a sort of “fork-join” processing happening with an em as (s)he works.

    If I were an em writing software, I’d think about what functions I needed next; “Okay, I’m going to need something to parse this data… I’ll turn it into this format… and then depending on where it’s of type X or Y I’ll need to process it using method A or B.” Split into 4 copies, write the functions.

    If I could write the functions sufficiently quickly, I don’t even think I’d mind if it was just a “random” copy of me who persisted. I think I’d prefer to merge the copies of “myself” back together, but if I can brake down the current part of the problem into sufficiently small sub-problems, I don’t think I can get too upset about a copy of myself getting deleted when it’s only diverged from the “main” copy for less than an hour or two.

    The effect from my surviving self’s point of view would be that while I remembered working on this one function and then, as I finished it up, I found the other functions I was thinking about before I started ready for me.

    There would be a very strong preference for programming languages that supported most of the code being written without side-effects. Pure functions are a lot easier to combine while having no memory of what went into producing the original.

    • http://overcomingbias.com RobinHanson

      Yes splitting copies to write related parts makes sense. But others have argued for less modularity, not more, because em labor is cheaper.

      • Joshua Brulé

        I think the software itself might get very specialized, compared to today’s software and a lot less reusable across domains. For example, I’ve noticed that math software packages like Sage or SciPy like to use the browser as a “notebook” since there isn’t enough programmer time to build a custom front-end from scratch (like the custom front-end that Mathematica or Matlab has). With ems, labor would be cheap enough to make a custom front-end (with features that wouldn’t be useful for any other projects) worth it.

        But I’m confident the internal design of software will have to be quite modular. In code with a lot of uncontrolled side-effects, there’s too many things that can go wrong when combining them into the larger system so the ems responsible for each component will have to coordinate with each other. It’s worth adding a little more development time for each component of the system if it they can all be built in parallel.

    • Doug

      Your assumption seems to be that an EM when faced with a large problem, would split into multiple copies and tackle each sub-problem concurrently. Why wouldn’t the EM simply work sequentially? Each approach takes the same amount of simulator processing time, which is the constrained resource. Unless you’re running under extreme real-time deadlines, there’s no advantage to concurrent splitting. And there’s a major disadvantage in that you don’t have a single final mind that understands the entire system.

      • http://overcomingbias.com RobinHanson

        The disadvantages to running faster sequentially are a) you might be in a development race and sequential takes longer where there is a maximum device speed, b) you use up the limited lifespan of the em before its mind gets too fragile to be useful, c) serial software tools get sluggish when you run faster.

      • Doug

        They’re all valid considerations. You’ve convinced me that c) is a serious issue, but as I explained in another comment I think it will be mostly easier to squeeze efficiency out of faster tools than larger software teams. Given the magnitude of EM speedups, I think a) is unlikely in all but the most gigantic projects. In an EM world software could be developed almost arbitrarily quickly, given enough resources. If the marginal constraining factor is cost not time that favors sequential.

        I’m no where near qualified to comment on b). But my understanding is that near all brain aging is low-level biological: plaque buildup, DNA damage, neurotransmitter depletion. oxidative stress, cell death, etc. I know you largely model EMs as black-boxes simulations. But the above mechanisms seem totally unrelated to replicating the brain’s mental processes with fidelity. It seems a simulated brain could run for orders of magnitude longer subjective time than a physical one.

      • wassname

        neural networks become over-specialised and loose adaptability after working on something too long. I believe he is reffering to this effect for ems in b)

      • Peter David Jones

        ANN’s? Human brains? If human brain don’t foolish why would a functional duplicate?

  • https://kev.inburke.com kevinburke

    There are 10x differences between industry programmers in time to complete a given task, size of the completed program, defect rate, and defects caught during code review. Many studies have replicated this finding, one of the studies that shows this is Sackman, Erickson & Grant, “Exploratory Experimental Studies Comparing Online/Offline Programming Performance”, 1968, and available here: https://kev.inburke.com/docs/sackman_erickson_grant_productivity.pdf. Another is Curtis, “Substantiating Programmer Variability”, 1981, available here: https://kev.inburke.com/docs/curtis_substantiating_variability.pdf. It’s easy to imagine that the more efficient programmers will be used as a baseline.

    (Discussing the existence of 10x differences in programmer ability is currently controversial in the industry).

    18-36% of errors made by programmers today are clerical errors (misspelled variable names, syntax errors, incorrect copy/paste) (Card, “A software technology evaluation program”, 1987, and Weiss, 1975 (not sure of the paper name)). It is likely that an em would make fewer errors in this area.

    One of the most powerful techniques for finding/avoiding errors is a formal design inspection, first described by Michael Fagan at IBM (http://ssltest.cs.umd.edu/class/fall2006/cmsc838p/VandV/fagan.pdf). The process involves asking a number of people to review the proposed changes to a piece code, or the architecture for a new feature, and try to find errors. It’s easy to imagine finding a segment of the population that’s particularly good at design inspection or code review and cloning that em so every program can have an excellent review process. (Also important to note that people find different errors when reviewing code; there’s a study that shows this but I don’t have it at the moment. I am assuming ems will have the same limitation, so you would want to have multiple ems look at a given piece of code.).

    For more in this area you may want to read “Code Complete” by Steve McConnell and pay attention to areas where a study is cited (these are marked in the book by a “Hard Facts” icon or similar in the margin). Capers Jones has also done some good research in this area.

    • http://overcomingbias.com RobinHanson

      I agree the ability to select and copy the best ems would make ems better suited for all parts of the software development process. But I don’t see that this favors some parts over others, so I don’t see why we’d predict a change in the mix of tasks.

      • https://kev.inburke.com kevinburke

        In other fields, like manufacturing, robots are preferred to humans for some tasks because they can “focus” better and complete mundane tasks with a lower error rate than humans.

        I was assuming the same effect would apply, and that a machine would not lose focus when programming, or make typographical errors.

      • Doug

        Since EMs are likely to be brain scans of the greatest hackers alive, one consideration when generalizing empirical results is whether very talented programmers are relatively more prone to some categories of mistakes more than the median?

      • Wilbur

        There are significant portions of the development process that are not practiced today, such as formal machine-verifiable specification and proof (ala Coq or Agda style proof assistants), outside of very isolated niches. This is in some part because of cost, in some part due to the lack of an established workforce capable of performing these tasks at all, and for any given human, there exists a sufficiently complex decision matrix that they will refuse to explore (we routinely overlook implementation edge-cases and test cases due to this). If your definition of an em includes the ability to absorb, retain, and leverage theory with ease and exhaustively execute tasks regardless of mundanity or complexity, advanced academic concepts like algorithmic proof can and will become mandatory practices.

    • John_Maxwell_IV

      Do you know of any studies supporting the 10x idea that have been done in the past 30 years?

    • a_programmer

      Christ, that Curtis thing is the world’s stupidest fucking paper. It provides only very weak evidence of programmer variability. All it shows is that different programmers happened to take different amounts of time on a certain relatively easy (debugging, not even programming) task. But what we don’t know is: would the same programmers that were fast on this task be fast on other tasks?

      As a programmer, I’m inclined to believe in programmer variability — but after reading Curtis, I’m actually less convinced; if that’s the best evidence there is on variability, then maybe it’s made-up.

  • Dan Browne

    My main issue with EMs isn’t the technical ability to create them. It’s how you cajole them to do work for you for less wages than a human version of them would demand. Or am I missing something?

    • lump1

      It’s not that they would get smaller wages, it’s that “paying rent” on the server which runs them would eat up almost the entirety. If you don’t pay it, you’re deleted, and the freed cycles will run a copy of an EM who is less of a deadbeat, maybe a copy of Elon Musk or Mozart. But even they will be struggling to earn enough to stay alive.

      • Dan Browne

        I get the concept, but there’s no guarantee they’ll be able to make EMs that are so scared of being deleted that they’d prefer to be slaves. I myself would rather be dead than be a slave. An EM copy of me would have to be heavily modified to not feel that way and who knows if such modifications would make the modded EM be efficient.

      • lump1

        Yeah, I think you’re getting it. There would be lots of ems, but none would be recognizably like you (and me) in their psychology. The ones willing to sacrifice everything to merely live, who choose the most lucrative jobs and work themselves to the edge of insanity, those (and their copies) would be the only survivors.

      • IMASBA

        But who says the rest will go quietly? Why not perform some sort of equivalent to a suicide bombing one nanosecond before the powers that be were going to delete you? If the premise is that EMs can be forcefully deleted then violence is still possible in the EM world.

      • Dan Browne

        Not necessarily. I know precisely how I would react in such a scenario. I would “pretend” to cooperate and instead collaborate with any copies of myself in a conspiracy to get revenge on my “owner”.

      • http://overcomingbias.com RobinHanson

        Standard labor econ predicts near subsistence wages. And those have been the norm through pretty much all of animal history, 99.5% of human history, and still hold in much of the world today. This isn’t some strange theoretical possibility that only applies to weird personalities.

      • IMASBA

        Actually foragers lived on more than subsistence wages. They successfully practiced population control and had a lot of leisure time (which is a form of wealth). They were sometimes growing too fast when colonizing a new area but typically recognized this in time and prevented a malthusian catastrophe. Many predatory animals have a lot of leisure time as well (felines are a great example).

        The modern world is again seeing a return to effective population control and that is a fundamental shift in human development that won’t be undone just like that. EMs are different in the sense that procreation is simple and very fast so that effective population control will be difficult, even impossible without a central government, that is something specific to EMs and the resulting extreme malthusian pressure will also be worse than what we’ve typically had in the real world, the few historical exceptions ended badly and show that most personalities won’t just sit around and accept the situation (Viking age, conquest of the new world, the incessant civil wars in modern Africa, religious violence in the Middle East and India).

      • http://overcomingbias.com RobinHanson

        I say “near subsistence” not exact subsistence. Foragers and animals are typically near subsistence. Think of ems being roughly that near.

      • IMASBA

        I don’t pretend to know what EM crime and civil war would look like but what’s certain is that the non-compliant EMs wouldn’t just quietly let themselves be carried off to the proverbial slaughterhouse. Of course they would use every method at their disposal to stave off their own deletion. Who would even try to stop that or determine what constitutes a “crime” when it’s just an EM-eat-EM world with no government?

      • Dan Browne

        I’m talking just about supply and demand and commodities vs non commodities. In software development for example, although management would like it to be this way, developers are not cookie cutter. From that perspective each “good” developer has a monopoly on the market and is able to extract better than subsistence. So what I’m saying is this: everybody would want to make copies of the “good” developers. Every market place is composed of a bid and an ask price and a trade is only done if the bid and ask coincide. Obviously if supply is incredibly high then the bid is way lower than the ask and the ask will have to come down. But I only see that happening in commoditized situations and human beings (especially high end knowledge workers) are NOT commodities and they (the workers) know it. From that perspective, why would a high end worker who is EM’d and wakes up (as an EM) in a virtual environment suddenly decide that they need the job of “do what I say for no pay if you want to live”. I’m not sure that they would, unless the bidder was able to convince them that they the EM had no bargaining power. And the only to convince them that they have no bargaining power is to explain the situation. But then, once the situation was explained, I’m not sure the EM would cooperate. There are tons of psych experiments showing that humans don’t behave as precisely rational agents when offered an “unfair” trade. Economics predicts that in a sharing scenario, where $1 is to be shared between two actors, and if a deal cannot be reached both actors get zero dollars, that most people would prefer to have zero dollars if the other person is perceived to be presenting an “unfair” share. Whereas rational actors would accept even $0.01 because they are still better off. So… is an EM copy of a human suddenly going to behave like a rational economic unit ? (And especially one who has self-perceived bargainin power). I think not.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Standard labor econ predicts near subsistence wages.

        What stops wages from dropping below (intermediate-term) subsistence when labor is extremely readily replaced?

        Also, reiterating Dan Browne’s point, what distinguishes the em-world from most of human history is the sharp decline in welfare em personalities experience.

      • IMASBA

        “what distinguishes the em-world from most of human history is the sharp decline in welfare em personalities experience.”

        Yes, people can get used to being poor if it has always been that way, but when they remember better times that takes a huge psychological toll. The recent situation in Greece is a prime example, the country is still relatively rich but it has declined and people lost their certanties which has made the population much more pessimistic and depressed than the population of fast-growing and developing Chile, even though Chile is exactly as rich as Greece in absolute terms.

  • lump1

    Speculation from a non-coder: Since such a big portion of global CPU cycles will be running ems on this scenario, even a tiny streamlining of em code efficiency will have a huge economic value. The most economically motivated coding question would be: “Which aspects of brain function don’t have to be emulated, or could be fudged in some efficient way, without the fudge being offset by harm to the em’s economic productivity?” All such fudges will be widely implemented, with ems turning out to be psychologically quite unlike humans.

    Advertising and virus-writing will merge into a single activity. Ad-viruses will infect ems so as to create or stoke desires that were non-operant. Ad-antivirus will be a large industry.

    In an em world, a running computer is the equivalent of a human’s home, sustenance and entertainment. So I expect that the portion of the global economy going into building and running CPUs will be at least as much as the sum of housing, agriculture, medicine and entertainment today. Even seemingly trifling improvements in code or hardware efficiency will have huge global impacts, just because the running of code is darn close to everything that the world does anymore.

    Another guess: the most common computing hardware will be much more error prone in the future, because one thing we will learn from brains is how to get adequate results from sloppy processing units. The sloppiness will be more than compensated by the increased efficiency of the imprecise processors. Code written for such CPUs cannot be written efficiently in the familiar languages of today. Of course there will be compilers, but the code that needs to be efficient – the bottleneck stuff – will be written in some fuzzy counterpart to assembler.

    • http://overcomingbias.com RobinHanson

      I agree ems would run on fault-tolerant hardware, with matching software tools, and it would be very highly optimized. I don’t see that needing to create inhuman psychology however.

  • Kevin

    I’m a software engineering manager so that’s my context here. If I could have a copy of even just 100 software engineers that I hand picked to be highly skilled for a diversity of skills, and the copies ran at 1000x speed, I think the resulting team would have more software engineering capacity than all current software engineering companies combined. This is just assuming only 100 such ems exist. The entire industry would move massively forward, and the sorts of problems we would face would be different. I don’t think questions like “what programming language would they use” would even matter.

    I don’t think point #8 will actually be a big problem. You wouldn’t really need to communicate much between ems except for the largest projects, like an operating system or a search engine. Just because a generalist engineer working at 1000x speed could build quite a lot.

    Point #9 also doesn’t seem like a big deal, since you wouldn’t need much communication. Also you can just videoconference to normals and milliseconds of latency to other ems doesn’t seem like a problem.

    It seems like the cost of software engineering would essentially go to zero. That means the cost of software-related projects would be totally driven by product issues like distribution or design. Projects that are heavily software-engineering bound could have singularity-like progress. Like, search engines. Projects that have some non-software-engineering component would be limited strictly by their non-software-engineering components due to Amdahl’s law. For example, social applications would be limited by considerations like, there are a thousand other social applications launching every day, how do we get attention to ours. Applications like e-commerce could be easily customized for every single product sold; the limit would be a product manage rdescribing what you want to happen.

    In this scenario it seems likely that one corporate entity would manage to monopolize all software engineering, because the first entity to create an em capable of this will have many opportunities to enforce lockin. For example imagine one of Google or Apple being that company, using this power to ensure their mobile platform won, and subsequently leveraging that. Cloning applications would get much easier in this world because it requires minimal non-software-engineer input, so the business importance of other things like control of distribution rises.

    • http://overcomingbias.com RobinHanson

      I agree that software wouldn’t set project time-limits, though they could still set budget limits. I don’t see as much locking as you do.

      • Kevin

        Maybe a better question is whether one corporate entity would monopolize all ems. It seems to me like there would be an em monopoly, because the natural monopoly that goes to the first inventor could be quickly extended by using the ems to improve the software engineering that creates the em itself.

      • http://overcomingbias.com RobinHanson

        Most products allow the first developer to do first use and improvement. Doesn’t usually add up to natural monopoly.

      • http://don.geddis.org/ Don Geddis

        You’re suggesting the EM version of FOOM / the singularity. Robin has already extensively discussed that scenario. E.g. http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate

        (FOOM is far more likely with designed artifacts like AIs, than with black-box copies like EMs.)

  • arch1

    I’m surprised to see no discussion here of moral constraints. In our current world, we tend to avoid, and actively attempt to prevent, certain behaviors because we believe they are bad for the well being of conscious creatures. Murder, slavery, prospects of cloning for organ-harvesting, etc. But 43 comments in, I don’t see any such considerations (wrt either ems *or* humans) entering the picture. Did I skim too quickly? Miss a memo?

    • http://overcomingbias.com RobinHanson

      The issue here is to predict what will happen, not what should happen. And few discussions of software engineering tools and work habits in any context mention morality.

      • arch1

        Robin, your 2nd sentence seems like a non sequitir. Wrt your 1st sentence – if you really are trying to predict what will happen, then don’t you need to factor in moral considerations that may (as in the present time) strongly constrain what happens? Or alternatively explain why such constraints won’t be operational in future?
        Put another way, the approach you describe, applied to the present day from (say) a time 300 years in the past, would unnecessarily systematically err from what we see around us, and in extremely consequential ways.

      • arch1

        I guess I should attempt to respond directly to your 2nd sentence “…few discussions of software engineering tools and work habits in any context mention morality.” Just for starters:
        Few discussions of software engineering tools and work habits describe the creation and destruction of conscious creatures as phases of the SW development lifecycle. Or have as potential side effects the obsolescence or destruction of humanity.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Arch1 is referring to the moral norms that will constrain developments, not those that should constrain them.

        I think the candid response would be that you deny that morals constrain developments. (The economic base determines the ideological superstructure, including morality.)

      • arch1

        That’s right, thanks for clarifying.

    • oldoddjobs

      Guess we’re just not as moral as you, arch1. Thanks for caring.

      • arch1

        (begin attempted humorous remark) Maybe all I *really* care about is whether Robin has long term archiving enabled (end attempted humorous remark)

  • John_Maxwell_IV

    Some programmers are better than others, and in the competition for hiring the best programmers, some organizations consistently do better than others. That’s why using Gmail is a nicer experience than logging in to your bank or university backend. In an em world where minds could be duplicated, you could have the world’s best programmers working on what would normally be considered boring projects.

    “Programmers will be very parsimonious in the syntax and semantics of their programming languages and other programs; they will favor both high-falluting abstraction and ruthless efficiency over any kind of fanciness.”

    I’m not sure I agree. The reason modern programmers don’t trend towards maximal terseness is that terseness trades off against readability and maintainability. And writing terse code doesn’t necessarily take less time. It’s possible that the readability and maintainability will not be as important if you can boot up a saved version of the person who wrote the code, but I doubt their importance will go to zero.

    “At high-speed, though, latency becomes a huge bottleneck of social programming, even for these geniuses”

    Well yeah, if bandwidth becomes a significant factor relative to the speed minds are running at, LOTS of stuff will change, and not just in software development. That’s like taking us back to the pre-internet days, where sending information around the world was costly and slow. In the pre-internet days, we had shrinkwrapped software packages like Microsoft Encarta: developed by local teams, carefully tested, and then shipped to the rest of the world. If sending info was slow but not costly, you might see a kind of hybrid between modern SAAS and shrinkwrapped software development cultures.

  • a_programmer

    In an EM world, there might be much less of a call for software. Facebook? You don’t have time for that, because you’re poor, because everyone’s poor. Microsoft Word? Your bosses will just give you a cheap, shitty word processor because maybe it takes you a little longer, but hey, your time is cheap relative to the cost of Word (especially if the (relative) slowing speed of serial computation leads to more expensive software development). Banking software? Well, I guess owners of capital will see more gains, so that’s probably still a thing. Computer vision? Why not just hire some poor EMs to do it? Robotics? For what, manufacturing more computers? Serving rich people? If the future is poor, then there are plenty of poor people to do that (plus half the fun of being rich is having poor people wait on you). Is the reason for the recent boom in computing that it’s one of the last areas that are making money?

    Will there even be general-purpose computers? Or will we need a license to write software (the way we now need such a license on iOS and some versions of Windows)? If so, will licensing costs increase the costs of programming? Or will the companies that build computers stifle innovation for fear of upsetting the status quo? And won’t that fear be even greater in a poorer world?

    The difficulty of predicting this is the difficulty in predicting where computing will go plus the difficulty of thinking about the future generally. I’m not sure that any prediction is going to be meaningful.

    • http://overcomingbias.com RobinHanson

      If ems are cheap then software written by ems is cheap. And a larger em economy can divide the fixed costs of writing software among more customers. So software should be plenty cheap enough for ems to afford it.

  • glebm

    If you can copy, emulate, and fork brains, wouldn’t flesh & blood humans quickly become obsolete and extinct, as they are comparatively slow, unreliable, inefficient, and mortal?

    • Joshua Brulé

      Most likely yes.

      There may persist small colonies of flesh-and-blood humans, though. I’d expect that the larger em world would treat them like we treat, say, the Amish today. Ems wouldn’t make any effort to kill the meatspace humans off, or even steal their stuff, but also wouldn’t take meatspace humans very seriously.

  • Geoff Brown

    I think the most fundamental change to software tools in this scenario will be a focus shift from tactical economic activities to the optimization of experimentation, discovery and simulation methods.

    I think this would be the nexus of the battle between EM’s. This could even lead to evolutionary differentiation between EM’s.
    Any real advances beyond speed will still be constrained by the experimental environments the EM’s can create. Especially as the multivariate complexities of the simulations grow exponential or slower yet if the carbon/hydrogen system can’t be modelled accurately, then discovery will still have a analog speed limit.

    I think a core focus of EMs might be to either minimize serendipity or quantify the optimal randomness required to create sufficient serendipity to create answers to primary research questions.

  • Pingback: Overcoming Bias : Em Software Results

  • DavidGross

    Software geek for 20+ years. BS in CS with minor in AI. My gut reaction on reading this post, FWIW:

    Creating an em strikes me as a Very Hard Problem. My money is on it not happening, period. That said, if it were to happen, the various problems solved along the way to making an em a reality would make the idea of using an em to do software engineering ridiculous. Instead, you’d branch off some of these intermediate innovations to make software-engineering-specific AI applications that wouldn’t be very em-like. Asking how you’d use an em to do software engineering is like asking how much hay you’d need to feed a horseless carriage.

    FWIW: I’m currently working with a company that deals with cutting-edge Hard Problems involving huge amounts of data distributed all over the place and used by tons of people and managed by an ecosystem of a gazillion feisty bits of difficult software that nobody really understands in the aggregate. A lot of their work these days is going in to trying to make the status of this ecosystem more easily processable by the human neurological system through creative data visualization techniques… in other words, trying to finesse the boundary between human brain data-processing ability and computer big-data.