Category Archives: AI

Followup to:  Cascades, Cycles, Insight, Recursion, Magic

We seem to have a sticking point at the concept of "recursion", so I’ll zoom in.

You have a friend who, even though he makes plenty of money, just spends all that money every month.  You try to persuade your friend to invest a little – making valiant attempts to explain the wonders of compound interest by pointing to analogous processes in nature, like fission chain reactions.

"All right," says your friend, and buys a ten-year bond for $10,000, with an annual coupon of $500.  Then he sits back, satisfied.  "There!" he says.  "Now I’ll have an extra $500 to spend every year, without my needing to do any work!  And when the bond comes due, I’ll just roll it over, so this can go on indefinitely.  Surely, now I’m taking advantage of the power of recursion!"

"Um, no," you say.  "That’s not exactly what I had in mind when I talked about ‘recursion’."

"But I used some of my cumulative money earned, to increase my very earning rate," your friend points out, quite logically.  "If that’s not ‘recursion’, what is?  My earning power has been ‘folded in on itself’, just like you talked about!"

"Well," you say, "not exactly.  Before, you were earning $100,000 per year, so your cumulative earnings went as 100000 * t.  Now, your cumulative earnings are going as 100500 * t.  That’s not really much of a change.  What we want is for your cumulative earnings to go as B * e^At for some constants A and B – to grow exponentially."

"Exponentially!" says your friend, shocked.

"Yes," you say, "recursification has an amazing power to transform growth curves.  In this case, it can turn a linear process into an exponential one.  But to get that effect, you have to reinvest the coupon payments you get on your bonds – or at least reinvest some of them, instead of just spending them all.  And you must be able to do this over and over again.  Only then will you get the ‘folding in’ transformation, so that instead of your cumulative earnings going as y = F(t) = A*t, your earnings will go as the differential equation dy/dt = F(y) = A*y whose solution is y = e^(A*t)."

Continue reading "Sustained Strong Recursion" »

Continuation of:  Hard Takeoff

The analysis given in the last two days permits more than one possible AI trajectory:

1. Programmers, smarter than evolution at finding tricks that work, but operating without fundamental insight or with only partial insight, create a mind that is dumber than the researchers but performs lower-quality operations much faster.  This mind reaches k > 1, cascades up to the level of a very smart human, itself achieves insight into intelligence, and undergoes the really fast part of the FOOM, to superintelligence.  This would be the major nightmare scenario for the origin of an unFriendly AI.
2. Programmers operating with partial insight, create a mind that performs a number of tasks very well, but can’t really handle self-modification let alone AI theory.  A mind like this might progress with something like smoothness, pushed along by the researchers rather than itself, even all the way up to average-human capability – not having the insight into its own workings to push itself any further.  We also suppose that the mind is either already using huge amounts of available hardware, or scales very poorly, so it cannot go FOOM just as a result of adding a hundred times as much hardware.  This scenario seems less likely to my eyes, but it is not ruled out by any effect I can see.
3. Programmers operating with strong insight into intelligence, directly create along an efficient and planned pathway, a mind capable of modifying itself with deterministic precision – provably correct or provably noncatastrophic self-modifications.  This is the only way I can see to achieve narrow enough targeting to create a Friendly AI.  The "natural" trajectory of such an agent would be slowed by the requirements of precision, and sped up by the presence of insight; but because this is a Friendly AI, notions like "You can’t yet improve yourself this far, your goal system isn’t verified enough" would play a role.

So these are some things that I think are permitted to happen, albeit that case 2 would count as a hit against me to some degree because it does seem unlikely.

Here are some things that shouldn’t happen, on my analysis:

Continue reading "Permitted Possibilities, & Locality" »

Continuation of:  Recursive Self-Improvement

Constant natural selection pressure, operating on the genes of the hominid line, produced improvement in brains over time that seems to have been, roughly, linear or accelerating; the operation of constant human brains on a pool of knowledge seems to have produced returns that are, very roughly, exponential or superexponential.  (Robin proposes that human progress is well-characterized as a series of exponential modes with diminishing doubling times.)

Recursive self-improvement – an AI rewriting its own cognitive algorithms – identifies the object level of the AI with a force acting on the metacognitive level; it "closes the loop" or "folds the graph in on itself".  E.g. the difference between returns on a constant investment in a bond, and reinvesting the returns into purchasing further bonds, is the difference between the equations y = f(t) = m*t, and dy/dt = f(y) = m*y whose solution is the compound interest exponential, y = e^(m*t).

When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM.  An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely – far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology.  Our present life is no good indicator of things to come.

Or to try and compress it down to a slogan that fits on a T-Shirt – not that I’m saying this is a good idea – "Moore’s Law is exponential now; it would be really odd if it stayed exponential with the improving computers doing the research."  I’m not saying you literally get dy/dt = e^y that goes to infinity after finite time – and hardware improvement is in some ways the least interesting factor here – but should we really see the same curve we do now?

RSI is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it’s nowhere near being the only such factor.  The advent of human intelligence was a discontinuity with the past even without RSI…

Continue reading "Hard Takeoff" »