Comments on: Outside View of Singularity

By: Chip

Chip — Tue, 24 Jun 2008 18:34:24 +0000

Surely what I am about to write is obvious, and probably old. During World War II, when physicists began to realize the destructive potential of nuclear weapons, Albert Einstein was chosen by his peers to approach President Roosevelt. Einstein was perhaps not the best informed of the group, but he was the best known, and was thought to be able to get Roosevelt’s ear, as he did. In response, Roosevelt was able to convene all the greatest Western minds in physics, mathematics, and engineering to work together for a rapid solution to the problem. Clearly, the importance of the development of recursively self-improving super-human intelligence has got to be, almost by definition, greater than all other current problems, since it is the one project that would allow for the speedy solution of all other problems. Is there no famous person or persons in the field, able to organize his peers, and with access to the government such that an effort similar to the Manhattan Project could be accomplished? The AI Institute has one research fellow, and are looking for one more. They have a couple of fund-raisers, but most of the world is unaware of AI altogether. This won’t get it done in a reasonable time-frame. Your competitors may well be backed by their governments.

While the eventual use of the Manhattan Project’s discoveries is about as far from Friendly AI as imaginable, the power of super-human recursive AI is such that no matter by whom or where it is developed it will become the eminent domain of a government, much like the most powerful Cray computers. You might as well have their money and all the manpower right from the start, and the ability to influence it’s proper use.

Can/will this be done?

By: Tim Tyler

Tim Tyler — Sun, 22 Jun 2008 15:36:15 +0000

As I mentioned, one point of disanalogy between the farming/industrial developments and AI is that farming didn’t put any humans out of work, while the humans put out of work by industry had other places in the economy to go. With AI, it effectively takes out most of the economy out of human hands, maybe leaving a few vacancies in the service industries.

Another disanalogy between the farming/industrial developments and AI is that is is hard to keep farming and industrial developments secret – they are typically too easy to reverse engineer. Whereas with AI, if you keep the code on your server, it is extremely difficult for anyone to reverse engineer it. It can even be deployed fairly securely in robots – if tamper-proof hardware is employed.

Both of these differences suggest that AI may be more effective at creating inequalities than either farming or industry was.

However, ultimately, whether groups of humans benefit differentially from AI or not probably makes little odds.

The bigger picture is that it represents the blossoming of the new replicators into physical minds and bodies – so there is a whole new population of non-human entities to consider, with computers for minds and databases for genomes.

By: Hopefully Anonymous

Hopefully Anonymous — Sun, 22 Jun 2008 13:53:53 +0000

Singularities in principle don’t seem that hard to model. Have people tried modeling how quickly an agent takes over a game (if it does) with the same analytical algorithms but quicker processing speed? Have they looked at how interdependency affects that? Robin, you have some interesting hypotheses that seem open to be tested in a variety of ways.

By: Tim Tyler

Tim Tyler — Sat, 21 Jun 2008 23:44:10 +0000

I did not say “throw this out of your dataset, it is not relevant”.

We do have more data to go on than just the recent economic successes of our ancestors.

By: Robin Hanson

Robin Hanson — Sat, 21 Jun 2008 22:02:59 +0000

Tim you say “the relevant important developments are really the previous genetic takeovers”, a phrase I now understand better, but the right way to do multivariate analysis is not to first choose the “right” data. Instead one collects as much relevant data as possible and then sees what statistical inference says about which data can in fact be ignored without changing the results much. Saying “throw this out of your dataset, it is not relevant” is less useful than saying “you’ve missed this relevant data, your conclusions will change when you include them.”

By: Tim Tyler

Tim Tyler — Sat, 21 Jun 2008 14:34:06 +0000

Genetic Takeover, is a concept from Genetic Takeover - and the mineral origins of life, A. G. Cairns-Smith, Cambridge University Press, 1982. Here is a page by me on the topic: http://originoflife.net/takeover/ It is not my suggestion that we are witnessing a modern Genetic Takeover: "machines could carry on our cultural evolution, including their own increasingly rapid self-improvement, without us, and without the genes that built us. It will be then that our DNA will be out of a job, having passed the torch, and lost the race, to a new kind of competition. The genetic information carrier, in the new scheme of things, will be exclusively knowledge, passed from mind to artificial mind." Human Culture - A Genetic Takeover Underway - Moravec, 1987 "Millions of years later, another change is under way in how information passes from generation to generation. Humans evolved from organisms defined almost totally by their organic genes. We now rely additionally on a vast and rapidly growing corpus of cultural information generated and stored outside our genes - in our nervous systems, libraries, and, most recently, computers. Our culture still depends utterly on biological human beings, but with each passing year our machines, a major product of the culture, assume a greater role in its maintenance and continued growth. Sooner or later our machines will become knowledgeable enough to handle their own maintenance, reproduction and self-improvement without help. When this happens the new genetic takeover will be complete. [...]" - Moravec, 1988 "Cultural evolution is many orders of magnitude faster than DNA-based evolution, which sets one even more to thinking of the idea of 'takeover'. And if a new kind of replicator takeover is beginning, it is conceivable that it will take off so far as to leave its parent DNA (and its grandparent clay if Cairns-Smith is right) far behind. If so, we may be sure that computers will be in the van." - Dawkins, 1982.

By: Venkat

Venkat — Sat, 21 Jun 2008 14:00:29 +0000

Perhaps you guys have addressed this elsewhere, but given that most evolution (technological, social, political…) seems to follow the jumping punctuated-equilibria model (cf. Thomas Kuhn, Joel Mokyr, McLuhan…), how do you separate the wheat from the chaff? The eternal behaviorist dilemma applies here.

By: Robin Hanson

Robin Hanson — Sat, 21 Jun 2008 12:07:07 +0000

Tim virtually every innovation is an “idea.” You seem to be saying the relevant category to use for an outside view is “genetic takeovers”, but since you are using “genetic” metaphorically I find this category hard to understand. Please try to be more precise so we can evaluate your suggestion. It is true that per-capita wealth inequality across the world is at an all time high, but this is mainly because the wealth peaks are at an all time high, while the valleys remain at their lowest feasible level.

By: Tim Tyler

Tim Tyler — Sat, 21 Jun 2008 08:31:40 +0000

Re: “until there is a substantial space or deep Earth economy/ecology all transitions will spread “horizontally.”"

The idea of horizontal transmission here was to illustrate that farming and agriculture were heritable *ideas*, and may well have practically wiped out the
other *ideas* that they competed with.

AI is also an idea, and one that is capable of spreading rapidly – but unlike farming and agriculture it is a replacement technology for an important DNA-based adaptation: brains. Rather than competing only with other ideas, it will more effectively compete with humans themselves – in conjunction with various associated developments in sensors and actuators, of course.

Re: “what the outside view doesn’t favor”. I see what you are saying – I just think it’s nonsense. The idea of looking at previous important developments, and trying to use them to see into the future is a good one, but the relevant important developments are really the previous genetic takeovers. Agriculture
and industry transitions throw only very limited light on AI. It’s like trying
to predict the properties of neutron stars by looking at gold and lead.

The technology advances we have seen so far tend to increase inequalities – by allowing wealth and power to be concentrated. Inequalities are greater now than ever before – with celebrities earning billions of dollars while much of the world is on the bread line. Further technological progress seems extremely likely to widen this gap.

By: Robin Hanson

Robin Hanson — Fri, 20 Jun 2008 22:42:27 +0000

Tim, until there is a substantial space or deep Earth economy/ecology all transitions will spread “horizontally.” Being “wiped out” is the sort of transition inequality I’m saying the outside view doesn’t favor.

Steven, you are repeating the standard argument inside viewers give against outside views, that it neglects crucial info.