Category Archives: Overconfidence

Above-Average AI Scientists

By Eliezer Yudkowsky · September 28, 2008 7:04 am · Comments (70)

Followup to:  The Level Above Mine, Competent Elites

(Those who didn’t like the last two posts should definitely skip this one.)

I recall one fellow, who seemed like a nice person, and who was quite eager to get started on Friendly AI work, to whom I had trouble explaining that he didn’t have a hope.  He said to me:

"If someone with a Masters in chemistry isn’t intelligent enough, then you’re not going to have much luck finding someone to help you."

It’s hard to distinguish the grades above your own.  And even if you’re literally the best in the world, there are still electron orbitals above yours – they’re just unoccupied.  Someone had to be "the best physicist in the world" during the time of Ancient Greece.  Would they have been able to visualize Newton?

At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation.  I believe he had people under him on his project.  He was probably paid at least three times as much as I was paid (at that time).  His academic credentials were superior to mine (what a surprise) and he had many more years of experience.  He had access to lots and lots of computing power.

And like nearly everyone in the field of AGI, he was rushing forward to write code immediately – not holding off and searching for a sufficiently precise theory to permit stable self-improvement.

In short, he was just the sort of fellow that…  Well, many people, when they hear about Friendly AI, say:  "Oh, it doesn’t matter what you do, because [someone like this guy] will create AI first."  He’s the sort of person about whom journalists ask me, "You say that this isn’t the time to be talking about regulation, but don’t we need laws to stop people like this from creating AI?"

Continue reading "Above-Average AI Scientists" »

GD Star Rating
loading...

Competent Elites

By Eliezer Yudkowsky · September 26, 2008 8:07 pm · Comments (55)

Followup to:  The Level Above Mine

(Anyone who didn’t like yesterday’s post should probably avoid this one.)

I remember what a shock it was to first meet Steve Jurvetson, of the venture capital firm Draper Fisher Jurvetson.

Steve Jurvetson talked fast and articulately, could follow long chains of reasoning, was familiar with a wide variety of technologies, and was happy to drag in analogies from outside sciences like biology – good ones, too.

I once saw Eric Drexler present an analogy between biological immune systems and the "active shield" concept in nanotechnology, arguing that just as biological systems managed to stave off invaders without the whole community collapsing, nanotechnological immune systems could do the same.

I thought this was a poor analogy, and was going to point out some flaws during the Q&A.  But Steve Jurvetson, who was in line before me, proceeded to demolish the argument even more thoroughly.  Jurvetson pointed out the evolutionary tradeoff between virulence and transmission that keeps natural viruses in check, talked about how greater interconnectedness led to larger pandemics – it was very nicely done, demolishing the surface analogy by correct reference to deeper biological details.

I was shocked, meeting Steve Jurvetson, because from everything I’d read about venture capitalists before then, VCs were supposed to be fools in business suits, who couldn’t understand technology or engineers or the needs of a fragile young startup, but who’d gotten ahold of large amounts of money by dint of seeming reliable to other business suits.

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

Continue reading "Competent Elites" »

GD Star Rating
loading...

The Level Above Mine

By Eliezer Yudkowsky · September 26, 2008 5:18 am · Comments (79)

Followup to:  The Proper Use of Humility, Tsuyoku Naritai

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science".  Mike Li read some of it, and then came back and said:

"Wow… it’s like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that -" and I said, "No, I know exactly what you mean."  It’s a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I’d enjoyed math proofs before I encountered Jaynes.  But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments.  Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower – power being used to overcome others.  Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection – something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion.  Of course, when you write a book, you get a chance to show only your best side.  But still.

Continue reading "The Level Above Mine" »

GD Star Rating
loading...

That Tiny Note of Discord

By Eliezer Yudkowsky · September 23, 2008 2:02 am · Comments (29)

Followup to:  The Sheer Folly of Callow Youth

When we last left Eliezer1997, he believed that any superintelligence would automatically do what was "right", and indeed would understand that better than we could; even though, he modestly confessed, he did not understand the ultimate nature of morality.  Or rather, after some debate had passed, Eliezer1997 had evolved an elaborate argument, which he fondly claimed to be "formal", that we could always condition upon the belief that life has meaning; and so cases where superintelligences did not feel compelled to do anything in particular, would fall out of consideration.  (The flaw being the unconsidered and unjustified equation of "universally compelling argument" with "right".)

So far, the young Eliezer is well on the way toward joining the "smart people who are stupid because they’re skilled at defending beliefs they arrived at for unskilled reasons".  All his dedication to "rationality" has not saved him from this mistake, and you might be tempted to conclude that it is useless to strive for rationality.

But while many people dig holes for themselves, not everyone succeeds in clawing their way back out.

And from this I learn my lesson:  That it all began -

- with a small, small question; a single discordant note; one tiny lonely thought…

Continue reading "That Tiny Note of Discord" »

GD Star Rating
loading...

Say It Loud

By Eliezer Yudkowsky · September 19, 2008 1:34 pm · Comments (13)

Reply to:  Overconfidence is Stylish

I respectfully defend my lord Will Strunk:

"If you don't know how to pronounce a word, say it loud! If you don't know how to pronounce a word, say it loud!"  This comical piece of advice struck me as sound at the time, and I still respect it. Why compound ignorance with inaudibility?  Why run and hide?

How does being vague, tame, colorless, irresolute, help someone to understand your current state of uncertainty?  Any more than mumbling helps them understand a word you aren't sure how to pronounce?

Goofus says:  "The sky, if such a thing exists at all, might or might not have a property of color, but, if it does have color, then I feel inclined to state that it might be green."

Gallant says:   "70% probability the sky is green."

Which of them sounds more confident, more definite?

But which of them has managed to quickly communicate their state of uncertainty?

(And which of them is more likely to actually, in real life, spend any time planning and preparing for the eventuality that the sky is blue?)

Continue reading "Say It Loud" »

GD Star Rating
loading...

A Genius for Destruction

By Eliezer Yudkowsky · August 1, 2008 3:25 pm · Comments (14)

This is a question from a workshop after the Global Catastrophic Risks conference.  The rule of the workshop was that people could be quoted, but not attributed, so I won’t say who observed:

"The problem is that it’s often our smartest people leading us into the disasters.  Look at Long-Term Capital Management."

To which someone else replied:

"Maybe smart people are just able to work themselves up into positions of power, so that if damage gets caused, the responsibility will often lie with someone smart."

Continue reading "A Genius for Destruction" »

GD Star Rating
loading...

Experience Increases Overconfidence

By Robin Hanson · June 30, 2008 6:00 am · Comments (14)

The latest Journal of Prediction Markets has a lit review on overconfidence, with a to-me surprising result: financial overconfidence increases with age, experience, and success.  Here are three investment experiments:

Kirchler and Maciejovsky (2002) …. demonstrated that subjects were overconfident in late trading periods … [but] underconfident during other trading periods. … Dittrich et al (2005) … found that age was positively correlated to overconfidence for complex tasks.  … Glaser et al (2005) … [found that financial market] professionals’ degrees of overconfidence was higher than that of the student subjects in most tasks, and it appeared that was because the "professionals are biased in job related tasks, such as forecasting real world financial time series."

Also:

A survey of German stock market forecasters conducted by Deaves et al (2005) … demonstrated that the market forecasters were overconfident in their predictions and that greater market experience and success, measured by correct predictions, increased their overconfidence.  … Markets are likely to become more overconfident when market returns are high. 

This is disturbing.  If overconfidence is positively correlated with ability, then observers can rationally take overconfidence as a signal of ability, and people can want to appear more overconfident to appear more able.  But to make this story work, somehow it should on average be easier to get away with being more overconfident when one is more experienced and successful.  How can this be?

This all seems to make it more reasonable than one might otherwise have thought to disagree about finance with older, more experienced, more successful folks.

GD Star Rating
loading...
Tagged as: ,

Joe Epstein on Youth

By Robin Hanson · June 13, 2008 6:00 am · Comments (47)

More on our overconfident kids from a thoughful essay by Joseph Epstein:

So often in my literature classes students told me what they "felt" about a novel, or a particular character in a novel. I tried, ever so gently, to tell them that no one cared what they felt; the trick was to discover not one’s feelings but what the author had put into the book, its moral weight and its resultant power. In essay courses, many of these same students turned in papers upon which I wished to–but did not–write: "D-, Too much love in the home." I knew where they came by their sense of their own deep significance and that this sense was utterly false to any conceivable reality. Despite what their parents had been telling them from the very outset of their lives, they were not significant. Significance has to be earned, and it is earned only through achievement. Besides, one of the first things that people who really are significant seem to know is that, in the grander scheme, they are themselves really quite insignificant.

GD Star Rating
loading...
Tagged as: ,

Against Devil’s Advocacy

By Eliezer Yudkowsky · June 9, 2008 12:15 am · Comments (38)

From an article by Michael Ruse:

Richard Dawkins once called me a "creep." He did so very publicly but meant no personal offense, and I took none: We were, and still are, friends. The cause of his ire—his anguish, even—was that, in the course of a public discussion, I was defending a position I did not truly hold. We philosophers are always doing this; it’s a version of the reductio ad absurdum argument. We do so partly to stimulate debate (especially in the classroom), partly to see how far a position can be pushed before it collapses (and why the collapse), and partly (let us be frank) out of sheer bloody-mindedness, because we like to rile the opposition.

Dawkins, however, has the moral purity—some would say the moral rigidity—of the evangelical Christian or the committed feminist. Not even for the sake of argument can he endorse something that he thinks false. To do so is not just mistaken, he feels; in some deep sense, it is wrong. Life is serious, and there are evils to be fought. There must be no compromise or equivocation, even for pedagogical reasons. As the Quakers say, "Let your yea be yea, and your nay, nay."

Michael Ruse doesn’t get it.

Continue reading "Against Devil’s Advocacy" »

GD Star Rating
loading...

Overconfidence & Paternalism

By Robin Hanson · May 28, 2008 6:00 am · Comments (27)

Paul Graham tries to explain paternalism: 

Parents know they’ve concealed the facts about sex, and many at some point sit their kids down and explain more. But few tell their kids about the differences between the real world and the cocoon they grew up in. Combine this with the confidence parents try to instill in their kids, and every year you get a new crop of 18 year olds who think they know how to run the world.

Don’t all 18 year olds think they know how to run the world? Actually this seems to be a recent innovation, no more than about 100 years old. In preindustrial times teenage kids were junior members of the adult world and comparatively well aware of their shortcomings. They could see they weren’t as strong or skillful as the village smith. In past times people lied to kids about some things more than we do now, but the lies implicit in an artificial, protected environment are a recent invention. Like a lot of new inventions, the rich got this first. Children of kings and great magnates were the first to grow up out of touch with the world. Suburbia means half the population can live like kings in that respect.  …

Continue reading "Overconfidence & Paternalism" »

GD Star Rating
loading...
Tagged as: , ,