Max Tegmark says his new book, Life 3.0, is about what happens when life can design not just its software, as humans have done in Life 2.0, but also its hardware:

Life 1.0 (biological stage) evolves its hardware and software
Life 2.0 (cultural stage) evolves its hardware, designs much of its software
Life 3.0 (technological stage): designs its hardware and software ..
Many AI researchers think that Life 3.0 may arrive during the coming century, perhaps even during our lifetime, spawned by progress in AI. What will happen, and what will this mean for us? That’s the topic of this book. (29-30)

Actually, its not. The book says little about redesigning hardware. While it says interesting things on many topics, its core is on a future “singularity” where AI systems quickly redesign their own software. (A scenario sometimes called “foom”.)

The book starts out with a 19 page fictional “scenario where humans use superintelligence to take over the world.” A small team, apparently seen as unthreatening by the world, somehow knows how to “launch” a “recursive self-improvement” in a system focused on “one particular task: programming AI Systems.” While initially “subhuman”, within five hours it redesigns its software four times and becomes superhuman at its core task, and so “could also teach itself all other humans skills.”

After five more hours and redesigns it can make money by doing half of the tasks at Amazon Mechanical Turk acceptably well. And it does this without having access to vast amounts of hardware or to large datasets of previous performance on such tasks. Within three days it can read and write like humans, and create world class animated movies to make more money. Over the next few months it goes on to take over the news media, education, world opinion, and then the world. It could have taken over much faster, except that its human controllers were careful to maintain control. During this time, no other team on Earth is remotely close to being able to do this.

Tegmark later explains:

As [computer ability] keeps rising, it may one day reach a tipping point, triggering dramatic change. This critical sea level is the one corresponding to machines becoming able to perform AI design. Before .. rise is caused by humans improving machines, afterward .. driven by machines improving machines. .. This is the fascinating and controversial idea of the singularity. .. I like to think of the critical intelligence threshold required for AI design as the threshold for universal intelligence: given enough time and resources, it can make itself able to accomplish any goals as well as any other intelligence entity. (p54)

I suspect that there are simpler ways to build human-level thinking machines than the solution evolution came up with. (156)

Tegmark apparently believes (like Eliezer Yudkowsky and Nick Bostrom) that there is some single yet-to-be-discovered simple general software architecture or algorithm which enables a computer to quickly get vastly better at improving its abstract “intelligence” (or “betterness”), without directly improving its ability to do most particular tasks. Then after it is abstractly better, if it so chooses it needs only modest data and hardware to quickly learn to create a skilled system for any particular task.

If humans learned this way, we’d spend the first part of our life getting “smart” by learning critical thinking and other ways to think in general, without learning much about anything in particular. And then after that we’d apply our brilliance to learning particular useful skills. In fact, human students mostly learn specific skills, and are famously bad at transferring their learning to related contexts. Students even find it hard to transfer from a lecture to a slightly different spoon-fed exam question on the same subject a month later, with strong incentives. So Tegmark foresees an AI far better at generalizing than are humans.

Tegmark is worried, because we don’t know when a singularity it might come, and it might be soon. Before then, we must find rigorous general solutions to most of ethics, computer security, decision and game theory, and the meaning of life:

What if your AI’s goals evolve as it gets smarter? How are you going to guarantee that it remains your goals no matter how much recursive self-improvement it undergoes? (263) Humans undergo significant increases as they grow up, but don’t always restrain their childhood goals. … There may even be hints that the propensity to change goals in response to new experiences and insights increases rather than decreases with intelligence. (267) .. Perhaps there’s a way of designing a self-improving AI that’s guaranteed to retain human-friendly goals forever, but I think it’s fair to say that we don’t yet know how to build one – or even whether it’s possible. (268) ..

We’ve now explored how to get machines to learn, adopt and retrain our goals. But .. should one person or group get to decide .. or does there exist some sort of consensus goals that form a good compromise for humanity as a whole? In my opinion both this ethical problem and the goal-alignment problem are crucial ones that need to be solved before any superintelligence is developed. (269)

To program a friendly AI, we need to capture the meaning of life. What’s “meaning”? What’s “life”? What’s the ultimate ethical imperative? .. If we cede control to a superintelligence before answering these questions rigorously, the answer it comes up with is unlikely to involve us. This makes it timely to rekindle the classic debates of philosophy and ethics, and adds a new urgency to the conversation! (279)

Now so far in history technology has mostly increased gradually, without huge surprising leaps, and teams usually had only modest leads on other teams. It has usually made sense to wait until seeing concrete problems before working to prevent variations on them. Tegmark admits that this applies to technology in general, and to the AI we’ve seen so far. But he sees future AI as different:

From my vantage point, I’ve instead been seeing fairly steady progress [in AI] for a long time. (92)

Throughout human history, we’ve relied on the same tried-and-true approach to keeping our technology beneficial: learning from mistakes. We invented fire, repeatedly messing up, and then invented the fire extinguisher, fire exit, fire alarm, and fire department. ..

Up until now, our technologies have typically caused sufficiently few and limited accidents for their harm to be outweighed by their benefits. As we inexorably develop ever more powerful technology, however, we’ll inevitably reach a point where even a single accident could be devastating enough to outweigh all the benefits. Some argue that accidental global nuclear war would constitute such an example. ..

As technology grows more powerful, we should rely less on the trial-and-error approach to safety engineering. In other words, we should become more proactive than reactive, investing in safety research aimed at preventing accidents from happening even once. This is why society invests more in nuclear-reactor safety than mousetrap safety. (93-94)

I’m not at all convinced that we see a general trend toward needing more proactive, relative to reactive, efforts. Nuclear power seems a particularly bad example, as we have arguably killed it due to excessive proactive regulation. And even if there were a general trend, Tegmark is arguing that future AI is a huge outlier, in that it could kill everyone forever the first time anything goes wrong.

In a 350 page book, you might think that Tegmark would take great pains to argue in detail for why we should believe that future artificial intelligence will be so different not only from past AI and from most other tech, but also from human intelligence. Why should we believe that one small team might soon find (and keep secret) a simple general software architecture or algorithm enabling a computer to get vastly better at improving its “general intelligence”, a feature not initially tied to being able to do specific things well, but which can later be applied to creating smart machines to do any specific desired task, even when task-specific data and hardware are quite limited? The closest historical analogy might be when humans first acquired general abilities to talk, reason, and copy behaviors, which has enabled us to slowly accumulate culture and tech. But even those didn’t appear suddenly, and it has taken humans roughly a million years to take over.

Here are the core of Tegmark’s arguments:

We’ve now explored a range of intelligence explosion scenarios. .. All these scenarios have two features in common:
1. A fast takeoff: the transition from subhuman to vastly superhuman intelligence occurs in a matter of days, not decades.
2. A unipolar outcome: the result is a single entity controlling Earth.
There is a major controversy about whether these two features are likely or unlikely. .. Let’s therefore devote the rest of this chapter to exploring scenarios with slower takeoffs, multipolar outcomes, cyborgs, and uploads. ..
A fast takeoff can facilitate a unipolar outcome. .. A decisive strategic advantage .. before anyone else had time to copy their technology and seriously compete. .. If takeoff had dragged on for decades .. then other companies would have been able to catch up. (150)

History revels an overall trend toward ever more coordination over ever-larger distances, which is easy to understand: new transportation technology makes coordination more valuable and new communication technology makes coordination easier. .. Transportation and communication technology will obviously continue to improve dramatically, so a natural expectation is that the historical trend will continue, with new hierarchical levels coordinating over ever-larger distances. (152-3)

Some leading thinkers guess that the first human-level AGI will be an upload. .. This is currently a minority view among AI researchers and neuroscientists. .. Why should the simplest path to a new technology be the one that evolution came up with, constrained by the requirements that it be self-assembling, self-repairing, and self-reproducing? Evolution optimizes strongly for energy efficiency. .. I suspect that there are simpler ways to build human-level thinking machines than the solution evolution came up with. (156)

Some people have told me that they’re sure that this or that won’t happen. However, I think it’s wise to be humble at this stage and acknowledge how little we know. (157)

Why should the power balance between multiple superinteligences remain stable for millennia, rather than the AIs merging or the smartest one taking over. (166)

A situation where there is more than one superintelligence AI, enslaved and controlled by competing humans, might prove rather unstable and short-lived. It could tempt whoever thinks they have the more powerful AI to launch a first strike resulting in and awful war, ending in a single enslaved god remaining. (180)

Got that?

1. He suspects a much better than human mind design is relatively simple and easy to find.

2. If one team suddenly found a way to grow much faster, and kept it secret long enough, no other team could catch up.

3. History has seen a trend of coordination at increasingly larger scales,

4. A world of multiple AIs seems intuitively to him generically unstable to becoming one AI.

5. Those of us who disagree with him should admit we can’t be very confident here.

And that is why we probably face extinction unless we can quickly find rigorous general solutions to ethics, computer security, etc.

Pardon me if I’m underwhelmed. It’s hard to see why we should put much weight on his suspicions that there are simple powerful findable AI designs, or that multiple AIs naturally become one AI. That just isn’t what we’ve usually seen in related systems so far. The fact that one team could stay ahead if it found a fast enough way to grow says little about how likely is that premise. And a slow historical trend toward increasing coordination hardly implies that one AI will quickly arise and take over the world.

That’s my critique of the book’s main point. Let me now comment on some side points.

Tegmark is proud of getting “over 3000 AI researchers and robotics researchers” and “over 17,000 others .. including Steven Hawking” to sign a letter saying:

AI weapons .. require no costly or hard-to-obtain raw materials, so they’ll become ubiquitous and cheap. .. It will only be a matter of time until they appear on the black market and in the hands of terrorists, dictators .., warlords. Autonomous weapons are ideal for tasks such as assassinations, destabilizing nations, subduing populations, and selectively killing a particular ethnic group. We therefore believe that a military AI arms race would not be beneficial to humanity. (114)

He talks about pushing for international regulations to ban killer robots, but these arguments seem to apply generally to all offensive military tech. Given how few military techs ever get banned, we need such a tech to be both unusually harmful and an unusually easy place to enforce a ban. It isn’t at all clear to me that military robots meet this test. (More skepticism here.)

Tegmark seems here to suggest that evolution no longer applies to humans, as brains can overrule genes:

Our brains are way smarter than our genes, and now that we understand the goal of our genes (replication), we find it rather banal and easy to ignore. People might realize why their genes make the feel lust, yet have little desire to raise fifteen children, and therefore choose to hack their genetic programming by coming the emotional rewards of intimacy with birth control. .. Our human gene pool has this far survived just fine despite our crafty and rebellious brains. .. The ultimate authority is now our feelings, not our genes. (256)

This seems to ignore the fact that evolution will continue, and over a longer run evolution can select out those who tend more to rebel against genetic priorities.

Tegmark is fond of using a basic income to deal with AI’s taking human jobs:

The simplest solution is basic income, where every person receives a monthly payment with no preconditions or requirements whatsoever.

He doesn’t consider what level of government would implement this, and whether that level has sufficient access to global assets valuable after AIs dominate jobs. Others have warned that this “insurance” isn’t very well targeted to this particular risk, and I’ve warned that insurance against this risk needs strong access to global assets or reinsurance.

Finally, Tegmark suggests that we should be unhappy with, and not accept, having the same relation to our descendants as our ancestors have had with us:

Consider .. viewing the AI as our descendants rather than our conquerors. .. Parents with a child smarter than them, who learns from them and accomplishes what they could only dream of, are likely happy and proud even if they know they can’t live to see it all. .. Humans living side by side with superior robots may also pose social challenges .. The descendant and conqueror scenarios .. are actually remarkably similar. .. The only difference lies in how the last human generations are treated. ..

We may think that those cute robot-children internalized our values and will forge the society of our drams once we’ve passed on, but can we be sure that they aren’t merely tricking us? What if they’re just playing along? .. We know that all our human affectations are easy to hack. .. Could any guarantees about the future behavior of the AIs, after humans are gone, make you feel good about the descendants scenario? (188-190)

My children have definitely not assured me that they aren’t just pretending to forge the society of my dreams, and now that I’m reminded of this fact, I’m going to demand that they rigorously prove their absolute loyalty to my values, or else .. something.