Overconfident Evaluation

I usually read the entire Washington Post, but apparently I didn’t notice last Sunday’s best article, about how top violinist Josh Bell played hauntingly beautiful music incognito in the subway and got $32 in donations (plus $20 from someone who recognized him), substantially less than the $150 predicted by the National Symphony Orchestra music director.  The lesson Tyler Cowen draws is "most people are ninnies, with little or no taste," but for me the lesson is how we vastly overestimate our evaluation abilities.   Tyler more relevantly asks:

If James Joyce were blogging today, without benefit of celebrity, and producing prose of the highest order, how many hits would he get a day?

Surely the answer is: hardly any.  If we passed a super-model on the street, we would just think her another pretty face.  We employers overemphasize job interviews in part from overconfidence in our evaluation abilities.  We academics review papers, not realizing how little our evaluations correlate with one another.  We lovers are way too sure that our mate is the best person around for us. 

Fellow contributor Nick Bostrom is a philosophy star, and I have the greatest respect for him.  But sadly, I was not particularly impressed for the first few years I knew him.  I may well not recognize a genius who comments here either.  I was once indignant that few recognized my potential when I was young, but I now see how unreasonable that was. 

We do not like to admit how terribly dependent we are on imperfect and noisy social institutions to evaluate one another.  We like to think our institutions just confirm the obvious.  For example, academic stars usually feel confident that all the good people have been recognized; but surely that is very wrong. 

The main lesson is that we underestimate the importance of our evaluation institutions, and so should devote more effort to improving them.  I’d love to create prediction markets to estimate the expected publication record of new academic hires.  This should improve our estimates, though estimates would probably vary a lot less across candidates than we expect.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • Rob Spear

    There is also the possibility that busking is a significantly different skill from concert performance.

  • http://pdf23ds.net pdf23ds

    There are *so* many different things you could take out of this experiment. My favored one is simply that it’s a bad idea to busk in the morning, when people are in a rush to get to work, and to do so in a subway entrance hall, where people are reluctant to stop. It’s (maybe intentionally) the worst time and place they could have picked. And I think that factor prevents one from drawing any other strong conclusions from the experiment, including yours, Robin.

  • B.S.

    I’ve often wondered about objective reality in so far as art and performance may be judged as objectively good. I have a friend who claims to like music that is objectively good. Is this just a bias? Or is there objectively good music and perhaps I just don’t have the ear for it?

  • TGGP

    B.S, there is no “objectively good” anything. There is no way to falsify the statement that something is “good” or “bad”, no test that can be done, no way to settle disagreements.

    On second thought, Complete are objectively superb and those who dissent do not merely hold different opinions, but are liars.

  • http://www.spaceandgames.com Peter de Blanc

    B.S., is your friend a Crab, by any chance?

  • http://riskmarkets.blogspot.com/ Jason Ruspini

    As pdf suggests, one may be biased in this situation. The initial reaction towards any form of solicitation in a public place should tend to be negative. Even if one recognizes the high quality of a certain performer despite their low-quality signal, one might not want to give them money since that would encourage public solicitation in general. This seems to be a better explanation than one of taste/cognition.

    Music can be objectively more difficult to perform and compose, and objectively more complex and subtle, but objectively “better” makes little sense.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    I gave eight examples of overconfident evaluation, yet you have all talked about only the first one. Even if that example is questionable, do you doubt the overall tendency?

  • http://cob.jmu.edu/rosserjb Barkley Rosser

    And the more that prediction markets get used for everything and its sibling, the more highly regarded will Robin Hanson be, :-).

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    My personal flaw is to overestimate people, not underestimate them. In general my opinion of people decreases over time much more than it increases – a sad trend to fold into initial estimates. The only person I can recall offhand who I previously seriously underestimated is, well, Nick Bostrom actually.

    I try to be conscious, when talking to people, of how poorly people seem to estimate me, and remember how poorly I may be estimating them – unfortunately since I have limited time, I still need to snap-categorize people and stop talking to the ones who are snap-categorized as insufficiently intelligent.

  • http://pdf23ds.net pdf23ds

    Robin, I can easily see how one’s tendency to over- or underestimate people could be determined by your emotional disposition, shaped by genetics and upbringing, and that the tendency might be correlated with pessimism and cynicism, or with baseline happiness, or some other measure. I don’t have any evidence for this, but none of your examples really ring true to me.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Eliezer and pdf, to be clear I was making no claim about any tendency to over or underestimate people. My claim was instead that we are overconfident in our estimates. We have lots of evidence of that regarding interviews, paper reviews, and lovers.

  • Doug S.

    If we passed a super-model on the street, we would just think her another pretty face.

    Even supermodels don’t look like supermodels when you see them in person. Remember that Dove ad?

  • Douglas Knight

    I don’t see much in the post that’s about overconfidence. Most of the examples seem to be cases where it isn’t worth the effort to make an accurate judgment.

    The case of employers does seem to be an example of overconfidence and error. Maybe academics are overconfident of their referee judgments, but should they act differently if they were less confident? They have to make a binary decision of publish or reject. (But many people do argue that academics don’t value originality enough, which would be a symptom of overconfidence.) Similarly, lovers may express false beliefs about their mates, but is this evidence that they’ve chosen the wrong time to settle down?

    The examples of the musician and the supermodel suggest merely that our institutions are chasing different goals than individuals. The marginal improvements aren’t worth it to the individual. Also, it may well be that they aren’t doing any better, that we trust the institutions too much in their judgments of beauty and talent.

  • http://www.pmcluster.com/ John T. Maloney

    R-

    Great post.

    Joyce blogging? That’s a scary thought.

    Finnegans Wake is ironic enough bound and in the stacks.

    Cheers,

    -j

  • jaim klein

    The cream will always float to the top. It will find out whom he has to impress and how to do it. Bad or stupid evaluating institutions can be cheated more easily buy the worthy.

  • jaim klein

    PS For example, you used a mechanism to check if I am a true human being. Did it work?

  • Jules

    I think if we see any of those professionals in a professional setting, we can make an assumption before knowing their quality that they have quality.

    If you listen to a professional orchestral performance, you assume everyone you hear has had years upon years of practice–and is very good–before they begin playing. Someone you see on the street, however, you assume they’re not that good.

    This is all based on the assumption that most people have no taste.

    Taste, analyzed by Paul Graham: http://paulgraham.com/goodart.html

  • http://www.saunalahti.fi/~tspro1/ Kaj Sotala

    The lovers example is different from the others, I think. In the others, one strives to be as accurate as possible. When thinking about one’s lover, most people probably recognize that there is an amount of purposeful self-deception in what they think. Overestimating a mate’s suitability for oneself – and being confident of that estimate – can help in maintaining fruitful relationships.