Category Archives: Overconfidence

The Magnitude of His Own Folly

Followup toMy Naturalistic Awakening, Above-Average AI Scientists

In the years before I met that would-be creator of Artificial General Intelligence (with a funded project) who happened to be a creationist, I would still try to argue with individual AGI wannabes.

In those days, I sort-of-succeeded in convincing one such fellow that, yes, you had to take Friendly AI into account, and no, you couldn’t just find the right fitness metric for an evolutionary algorithm.  (Previously he had been very impressed with evolutionary algorithms.)

And the one said:  Oh, woe!  Oh, alas!  What a fool I’ve been!  Through my carelessness, I almost destroyed the world!  What a villain I once was!

Now, there’s a trap I knew I better than to fall into –

– at the point where, in late 2002, I looked back to Eliezer1997‘s AI proposals and realized what they really would have done, insofar as they were coherent enough to talk about what they "really would have done".

When I finally saw the magnitude of my own folly, everything fell into place at once.  The dam against realization cracked; and the unspoken doubts that had been accumulating behind it, crashed through all together.  There wasn’t a prolonged period, or even a single moment that I remember, of wondering how I could have been so stupid.  I already knew how.

And I also knew, all at once, in the same moment of realization, that to say, I almost destroyed the world!, would have been too prideful.

It would have been too confirming of ego, too confirming of my own importance in the scheme of things, at a time when – I understood in the same moment of realization – my ego ought to be taking a major punch to the stomach.  I had been so much less than I needed to be; I had to take that punch in the stomach, not avert it.

Continue reading "The Magnitude of His Own Folly" »

GD Star Rating

Above-Average AI Scientists

Followup toThe Level Above Mine, Competent Elites

(Those who didn’t like the last two posts should definitely skip this one.)

I recall one fellow, who seemed like a nice person, and who was quite eager to get started on Friendly AI work, to whom I had trouble explaining that he didn’t have a hope.  He said to me:

"If someone with a Masters in chemistry isn’t intelligent enough, then you’re not going to have much luck finding someone to help you."

It’s hard to distinguish the grades above your own.  And even if you’re literally the best in the world, there are still electron orbitals above yours – they’re just unoccupied.  Someone had to be "the best physicist in the world" during the time of Ancient Greece.  Would they have been able to visualize Newton?

At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation.  I believe he had people under him on his project.  He was probably paid at least three times as much as I was paid (at that time).  His academic credentials were superior to mine (what a surprise) and he had many more years of experience.  He had access to lots and lots of computing power.

And like nearly everyone in the field of AGI, he was rushing forward to write code immediately – not holding off and searching for a sufficiently precise theory to permit stable self-improvement.

In short, he was just the sort of fellow that…  Well, many people, when they hear about Friendly AI, say:  "Oh, it doesn’t matter what you do, because [someone like this guy] will create AI first."  He’s the sort of person about whom journalists ask me, "You say that this isn’t the time to be talking about regulation, but don’t we need laws to stop people like this from creating AI?"

Continue reading "Above-Average AI Scientists" »

GD Star Rating

Competent Elites

Followup toThe Level Above Mine

(Anyone who didn’t like yesterday’s post should probably avoid this one.)

I remember what a shock it was to first meet Steve Jurvetson, of the venture capital firm Draper Fisher Jurvetson.

Steve Jurvetson talked fast and articulately, could follow long chains of reasoning, was familiar with a wide variety of technologies, and was happy to drag in analogies from outside sciences like biology – good ones, too.

I once saw Eric Drexler present an analogy between biological immune systems and the "active shield" concept in nanotechnology, arguing that just as biological systems managed to stave off invaders without the whole community collapsing, nanotechnological immune systems could do the same.

I thought this was a poor analogy, and was going to point out some flaws during the Q&A.  But Steve Jurvetson, who was in line before me, proceeded to demolish the argument even more thoroughly.  Jurvetson pointed out the evolutionary tradeoff between virulence and transmission that keeps natural viruses in check, talked about how greater interconnectedness led to larger pandemics – it was very nicely done, demolishing the surface analogy by correct reference to deeper biological details.

I was shocked, meeting Steve Jurvetson, because from everything I’d read about venture capitalists before then, VCs were supposed to be fools in business suits, who couldn’t understand technology or engineers or the needs of a fragile young startup, but who’d gotten ahold of large amounts of money by dint of seeming reliable to other business suits.

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

Continue reading "Competent Elites" »

GD Star Rating

The Level Above Mine

Followup toThe Proper Use of Humility, Tsuyoku Naritai

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science".  Mike Li read some of it, and then came back and said:

"Wow… it’s like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that -" and I said, "No, I know exactly what you mean."  It’s a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I’d enjoyed math proofs before I encountered Jaynes.  But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments.  Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower – power being used to overcome others.  Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection – something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion.  Of course, when you write a book, you get a chance to show only your best side.  But still.

Continue reading "The Level Above Mine" »

GD Star Rating

That Tiny Note of Discord

Followup toThe Sheer Folly of Callow Youth

When we last left Eliezer1997, he believed that any superintelligence would automatically do what was "right", and indeed would understand that better than we could; even though, he modestly confessed, he did not understand the ultimate nature of morality.  Or rather, after some debate had passed, Eliezer1997 had evolved an elaborate argument, which he fondly claimed to be "formal", that we could always condition upon the belief that life has meaning; and so cases where superintelligences did not feel compelled to do anything in particular, would fall out of consideration.  (The flaw being the unconsidered and unjustified equation of "universally compelling argument" with "right".)

So far, the young Eliezer is well on the way toward joining the "smart people who are stupid because they’re skilled at defending beliefs they arrived at for unskilled reasons".  All his dedication to "rationality" has not saved him from this mistake, and you might be tempted to conclude that it is useless to strive for rationality.

But while many people dig holes for themselves, not everyone succeeds in clawing their way back out.

And from this I learn my lesson:  That it all began –

– with a small, small question; a single discordant note; one tiny lonely thought…

Continue reading "That Tiny Note of Discord" »

GD Star Rating

Say It Loud

Reply toOverconfidence is Stylish

I respectfully defend my lord Will Strunk:

"If you don't know how to pronounce a word, say it loud! If you don't know how to pronounce a word, say it loud!"  This comical piece of advice struck me as sound at the time, and I still respect it. Why compound ignorance with inaudibility?  Why run and hide?

How does being vague, tame, colorless, irresolute, help someone to understand your current state of uncertainty?  Any more than mumbling helps them understand a word you aren't sure how to pronounce?

Goofus says:  "The sky, if such a thing exists at all, might or might not have a property of color, but, if it does have color, then I feel inclined to state that it might be green."

Gallant says:   "70% probability the sky is green."

Which of them sounds more confident, more definite?

But which of them has managed to quickly communicate their state of uncertainty?

(And which of them is more likely to actually, in real life, spend any time planning and preparing for the eventuality that the sky is blue?)

Continue reading "Say It Loud" »

GD Star Rating

A Genius for Destruction

This is a question from a workshop after the Global Catastrophic Risks conference.  The rule of the workshop was that people could be quoted, but not attributed, so I won’t say who observed:

"The problem is that it’s often our smartest people leading us into the disasters.  Look at Long-Term Capital Management."

To which someone else replied:

"Maybe smart people are just able to work themselves up into positions of power, so that if damage gets caused, the responsibility will often lie with someone smart."

Continue reading "A Genius for Destruction" »

GD Star Rating

Experience Increases Overconfidence

The latest Journal of Prediction Markets has a lit review on overconfidence, with a to-me surprising result: financial overconfidence increases with age, experience, and success.  Here are three investment experiments:

Kirchler and Maciejovsky (2002) …. demonstrated that subjects were overconfident in late trading periods … [but] underconfident during other trading periods. … Dittrich et al (2005) … found that age was positively correlated to overconfidence for complex tasks.  … Glaser et al (2005) … [found that financial market] professionals’ degrees of overconfidence was higher than that of the student subjects in most tasks, and it appeared that was because the "professionals are biased in job related tasks, such as forecasting real world financial time series."


A survey of German stock market forecasters conducted by Deaves et al (2005) … demonstrated that the market forecasters were overconfident in their predictions and that greater market experience and success, measured by correct predictions, increased their overconfidence.  … Markets are likely to become more overconfident when market returns are high. 

This is disturbing.  If overconfidence is positively correlated with ability, then observers can rationally take overconfidence as a signal of ability, and people can want to appear more overconfident to appear more able.  But to make this story work, somehow it should on average be easier to get away with being more overconfident when one is more experienced and successful.  How can this be?

This all seems to make it more reasonable than one might otherwise have thought to disagree about finance with older, more experienced, more successful folks.

GD Star Rating
Tagged as: ,

Joe Epstein on Youth

More on our overconfident kids from a thoughful essay by Joseph Epstein:

So often in my literature classes students told me what they "felt" about a novel, or a particular character in a novel. I tried, ever so gently, to tell them that no one cared what they felt; the trick was to discover not one’s feelings but what the author had put into the book, its moral weight and its resultant power. In essay courses, many of these same students turned in papers upon which I wished to–but did not–write: "D-, Too much love in the home." I knew where they came by their sense of their own deep significance and that this sense was utterly false to any conceivable reality. Despite what their parents had been telling them from the very outset of their lives, they were not significant. Significance has to be earned, and it is earned only through achievement. Besides, one of the first things that people who really are significant seem to know is that, in the grander scheme, they are themselves really quite insignificant.

GD Star Rating
Tagged as: ,

Against Devil’s Advocacy

From an article by Michael Ruse:

Richard Dawkins once called me a "creep." He did so very publicly but meant no personal offense, and I took none: We were, and still are, friends. The cause of his ire—his anguish, even—was that, in the course of a public discussion, I was defending a position I did not truly hold. We philosophers are always doing this; it’s a version of the reductio ad absurdum argument. We do so partly to stimulate debate (especially in the classroom), partly to see how far a position can be pushed before it collapses (and why the collapse), and partly (let us be frank) out of sheer bloody-mindedness, because we like to rile the opposition.

Dawkins, however, has the moral purity—some would say the moral rigidity—of the evangelical Christian or the committed feminist. Not even for the sake of argument can he endorse something that he thinks false. To do so is not just mistaken, he feels; in some deep sense, it is wrong. Life is serious, and there are evils to be fought. There must be no compromise or equivocation, even for pedagogical reasons. As the Quakers say, "Let your yea be yea, and your nay, nay."

Michael Ruse doesn’t get it.

Continue reading "Against Devil’s Advocacy" »

GD Star Rating