The Question

We humans have been writing “code” for many decades now, and as “software eats the world” we will write a lot more. In addition, we can also think of the structures within each human brain as “code”, code that will also shape the future.

Today the code in our heads (and bodies) is stuck there, but eventually we will find ways to move this code to artificial hardware. At which point we can create the world of brain emulations that is the subject of my first book, Age of Em. From that point on, these two categories of code, and their descendant variations, will have near equal access to artificial hardware, and so will compete on relatively equal terms to take on many code roles. System designers will have to choose which kind of code to use to control each particular system.

When designers choose between different types of code, they must ask themselves: which kinds of code are more cost-effective in which kinds of applications? In a competitive future world, the answer to this question may be the main factor that decides the fraction of resources devoted to running human-like minds. So to help us envision such a competitive future, we should also ask: where will different kinds of code work better? (Yes, non-competitive futures may be possible, but harder to arrange than many imagine.)

To think about which kinds of code win where, we need a basic theory that explains their key fundamental differences. You might have thought that much has been written on this, but alas I can’t find much. I do sometimes come across people who think it obvious that human brain code can’t possibly compete well anywhere, though they rarely explain their reasoning much. As this claim isn’t obvious to me, I’ve been trying to think about this key question of which kinds of code wins where. In the following, I’ll outline what I’ve come up with. But I still hope someone will point me to useful analyses that I’ve missed.

In the following, I will first summarize a few simple differences between human brain code and other code, then offer a deeper account of these differences, then suggest an empirical test of this account, and finally consider what these differences suggest for which kinds of code will be more cost-effective where.

Differences

The code in our heads is the product of learning over our lifetimes, inside a biological brain system that has evolved for eons. Though brain code was designed mainly for old problems and environments, it represents an enormous investment into a search for useful code. (Even if some parts seem simple, the whole system is not.) In contrast, the artificial code that we’ve been writing started from almost nothing a few decades ago.

Our brain code seems to come in a big tangled package that cannot easily be broken into smaller units that can usefully function independently. While it has identifiable parts, connections are dense everywhere; brains seem less modular than artificial code. Relatedly, brains seem much more robust to local damage, perhaps in part via having more redundancy. Brains seem designed for the case where communication is relatively fast and cheap within a brain, but between brains it is far more expensive, slow, and unreliable.

The code in our head does not take much advantage of many distinctions that we often use in artificial code. In our artificial systems, we gain many advantages by separating hardware from software, learning from doing, memory from processing, and memory addresses from content. But it seems that evolution just couldn’t find a way to represent and act on such distinctions.

Artificial code seems to “rot” more quickly. That is, as we adapt it to changing conditions, it becomes more fragile and harder to usefully change. As a result, most of the code we now use is not an edited version of the first code that accomplished each task. Instead, we repeatedly re-write similar systems over from scratch. In contrast, while the aspects of brain code that we learn over a lifetime do seem to slowly rot, in that we become less mentally flexible with age, we see little evidence of rot in the older evolved brain systems that we use to learn.

Our brain code is designed for hardware with many parallel but slow computing units, while our artificial code has so far mostly been designed for fewer fast and more sequential units. That is, brains calculate many things all at once slowly, while most artificial code calculates one thing at a time.

Our brains do some pre-determined tasks quickly in parallel. Such as simultaneously recognizing both visual and auditory signs of a predator. However, when we humans work on the tasks that most display our versatile generality, the sort of tasks that are most likely to matter in the future, our brains mostly function both slowly and sequentially. That is, we accomplish such tasks by proceeding step by step, and at each step our whole brain works in parallel by adding up many small contributions to that step.

Even so, the power and generality that often results from this process is truly stunning, being far beyond anything we know how to achieve with artificial code, no matter how much hardware we use and how many man-years we devote to writing it. This generality is why humans brains still earn the vast majority of “wages” in our economy. Artificial code is quite useful but gets paid much less in the aggregate, and in this sense is still far less useful than brain code. (“AI” software, i.e., artificial software intended to more effectively mimic brain abilities, earns a much smaller fraction of aggregate wages.)

While the code in our heads resulted largely from simple variation and selection of whole organisms, human brains use more directed processes to generate the artificial code. For example, one common procedure is to repeatedly have a brain imagine the results of particular small sections of code being executed in particular contexts, and search for edits to code such that imagined executions produce desired outcomes. This is commonly interleaved with less frequent actual execution of trial code on test cases. This process works better with more modular code expressed in terms of logical concepts that have sharp boundaries and implications, as it is easier for our brains to predict what happens in such contexts. A few parts of artificial code are generated via statistical analysis of large datasets.

When we have invested lately in having a kind of code actually do a task, such investments tend to give an advantage to that kind of code in continuing to do that task. Also, when each type of code can more easily connect to, or coordinate with, other code of its type, each type of code gains an advantage at a task when more of the tasks that it must coordinate with are also done by the same type of code. Thus each type of code has momentum, in continuing on where it was, and it naturally clumps together, especially in the most highly clumped sections of the network of tasks.

Easy Implications

So what do these various considerations imply about which kinds of code win where? We can identify some weak “all else equal” tendencies.

Brain code has at least a small temporary advantage on tasks that have been done by brain code lately, and that must coordinate with many other tasks done by brain code. The fact that that brain code requires an entire brain when being general suggests that artificial code is more cost-effective on small simple problems where brains do not have special abilities. At least when hardware costs to run code are important relative to costs to write code. Artificial code also seems much cheaper to run when a relatively simple sequential algorithm is sufficient, as brain code uses a whole brain to execute simple sequential computations. Artificial code also has advantages when lots of fast precise communication is desired across scopes larger than a brain.

The fact that brain code was designed for old problems suggests that it has advantages on old and long-lasting problems, relative to new problems. On the one hand, brain code being old and old systems being less flexible suggests using artificial code when great adaptability is required beyond the range for which brains were designed. On the other hand, the fact that artificial code rots more quickly suggests that artificial code has advantages when problems or contexts change quickly, which would force new code soon in any case, but disadvantages for stable long-lasting tasks.

In addition to these relatively simple and shallow implications, I have found a somewhat deeper perspective that seems useful. Let me explain.

Code Principles

Two key principles for managing code are: abstraction and modularity. With abstraction, you cut redundancy, via doing the same tasks but replacing many similar systems with a smaller number of more abstracted systems. Abstracted systems are smaller, in a code-length sense, though they may cost more in hardware to execute their code. By avoiding redundancy, abstracted systems make more efficient use of efforts to modify them.

With modularity, you try to limit the dependencies between subsystems, so that changes to one subsystem force fewer changes to other subsystems. Better ways to integrate and organize systems, which better “carve nature at its joints”, allow more effective modularity, and thus fewer design change cascades, wherein a change in one place forces many changes elsewhere.

It can take a lot of work to search for better design architectures that better facilitate modularity. Given the usual rate at which artificial code rots, only a limited amount of work here is justified. Sometimes systems that have partially rotted are “refactored”, by changing their high level architecture. Though expensive, such refactoring can cut the level of system rot, and is often cost-effective in terms of delaying a complete system rewrite. Better abstractions tend to promote better organization, which induces more effective more-slowly-rotting modularity.

A Deep Difference

Today, humans wanting code to do a task will first search for close-enough pre-existing code that might be lightly edited to do the job. Absent that, they will think for a bit, open a blank screen, and start typing. They will connect their new code to existing code as convenient, but will be wary of creating too many dependencies that reduce modularity. They will think and search for good ways to organize this new system, including good abstractions, but will put only limited effort into this. That is, to manage code complexity, humans tend to make new code that is highly modular, but not very well organized.

In contrast, as evolution designed and redesigned brains, it faced strong limits on the amount of brain hardware available. Brains take precious volume and are energetically expensive. And because evolution never managed to separate hardware and software, this hardware limit created strong limits on the amount of software possible. Limits far more restrictive than the memory limits imposed on humans who write code today. To add new software to brains, evolution could only a) add more hardware, b) delete old software, or c) seek more efficient representations and abstractions in order to save space.

Thus evolution just could not take the usual human approach of just opening a blank screen and writing new highly-modular but sloppily-organized code. Evolution instead had to keep searching hard for better ways to organize and integrate existing code, including better abstractions. This was a much more expensive process, but as it played out over eons it resulted in code that was much better organized and integrated, though less modular.

This perspective helps us to understand why brain code seems less modular than artificial code, why brain code doesn’t rot as fast and is more robust to damage, why it is harder to usefully break brain code into small units to do small tasks, why brain code is better at being more general, and why artificial code is more sequential. It also helps us understand why the usual focused processes for having brains make artificial code work reasonably well: brains know enough to predict how small chunks of code will behave, but only when that code is relatively modular.

This perspective also helps us to understand why abstraction is one of the brain’s key organizing principles. As I’ve said, human brains

collect things at similar levels of abstraction. The rear parts of our brains tend to focus more on small near concrete details while the front parts of our brain tend to focus on big far abstractions. In between, the degree of abstraction tends to change gradually.

Another important if less well understood organizing principle of brains is to separate a left and a right brain, perhaps to separate systems of credit assignment that don’t mix well. That is, to separate bottom-up processing that searches for fewer big things to explain many details, from top-down processing that searches for details to best achieve abstract goals.

In sum, a deeper perspective can help us to understand how brain and artificial code differ, and thus which kinds of code can win where: Brain code is better integrated and abstracted, but less modular, than artificial code.

Code-Cubed

Let me suggest a way to test this perspective, via data that should already be available, but which I haven’t yet found. In addition to brain code written by evolution, and artificial code written by brains, we can also consider “code-cubed”, i.e., code that is written by artificial code. This is “cubed’ because it is written by artificial code, which we can think of as “code-squared”, which is written by brain code, which we can think of as plain code, which is written by a non-code evolutionary process. Such code-cubed can obviously be written much more quickly than can ordinary code, at least at low levels of quality. But how else does it differ?

A large well-integrated brain that focuses its whole effort on thinking about particular chunks of code should produce in that artificial code a substantial degree of coherence and integration, at least on the scale of those chucks. However, when more modular and less well integrated artificial code writes code, that resulting code-cubed should be less well integrated. And as artificial code can write code much faster than can humans, and has plenty of empty memory available, artificial code will be tempted all the more to rely on new code and modularity to help manage complexity in the code it writes.

Thus this perspective suggests that code-cubed is even more modular, less well organized, and rots more quickly, than ordinary artificial code written by humans. For example, when we change the source code or compiler for a system, we then typically re-execute that complier on the source code, in effect re-writing from scratch. We do this instead of trying to edit the old compiled code in order to match the new source or new compiler.

Thus when we want code that can be usefully modified over longer periods of change, we should prefer ordinary artificial code to code-cubed. We should also prefer this when it is important that the code writer had a wider more general understanding of the task to be performed, and the other systems with which it needs to integrate.

What Wins Where

So in a future world where all types of code have access to the same cheap artificial hardware, and where competition pushes each application to use the type of code that is most locally cost-effective there, where should we expect to find brain code, where artificial code, and where code-cubed?

Brain code represents an enormous investment into a large tangled but well integrated and abstracted package that is hard to understand and modify, but has so far shown unparalleled power when applied to stable general broad tasks. This suggests that brain code may have a long future in applications that play to its strengths. (Long at least in terms of my usual favorite parameter: number of economic doubling times.)  Yes, eventually fully artificial code may become comparably well-integrated, but if ems are possible before then, the descendants of brain code may then have become even better organized and designed.

Most human organizations today are hierarchical, with low level activity focused more on relatively narrow contexts that matter less, need faster responses, and evolve more rapidly. In contrast, higher level activity allows slower responses, has more stable considerations, and must consider broader scopes and implications. Most artificial code today is also hierarchical, with low level code that tends to have a more narrow focus and contexts that change more rapidly, compared to high level code that must consider a wider range of more stable contexts, inputs, and other systems. In both types of systems, lower level tasks are more naturally modular, depending on fewer other tasks.

As smaller more focused more modular tasks that change more rapidly are better suited to artificial code, while less modular tasks that must consider wider context are better suited to brain code, we should expect artificial code to be more common at low organization levels, and brain code to be more common at high organization levels. That is, brains will manage big pictures, while artificial code manages details.

Among the tasks that humans do today, we can also distinguish more vs. less tangled tasks. Tangled tasks are closer to the more tangled center of a network of which tasks must coordinate with which other tasks. While tasks at higher organization levels do tend to be more tangled, some lower level tasks are also highly tangled. Brains also have advantages in these more tangled tasks, and once they are entrenched in many connected tasks are harder to displace from them.