Arrogant Professionals

  • CEO – “We study a unique panel of over 11,600 probability distributions provided by top financial executives and spanning nearly a decade of stock market expectations. Our results show that financial executives are severely miscalibrated: realized market returns are within the executives’ 80% confidence intervals only 33% of the time. We show that miscalibration improves following poor market performance periods because forecasters extrapolate past returns when forming their lower forecast bound (“worst case scenario”), while they do not update the upper bound (“best case scenario”) as much. Finally, we link stock market miscalibration to miscalibration about own-firm project forecasts and increased corporate investment.” (more)
  • Doc - “A study led by the Harvard researcher Nicholas Christakis asked the doctors of almost five hundred terminally ill patients to estimate how long they thought their patient would survive, and then followed the patients. Sixty-three per cent of doctors overestimated survival time. Just seventeen per cent underestimated it. The average estimate was five hundred and thirty per cent too high. And, the better the doctors knew their patients, the more likely they were to err. … Studies find that although doctors usually tell patients when a cancer is not curable, most are reluctant to give a specific prognosis, even when pressed. More than forty per cent of oncologists report offering treatments that they believe are unlikely to work.” (more)
  • Lawyer - “[Consider] predictions by a sample of attorneys (n = 481) across the United States who specified a minimum goal to achieve in a case set for trial. … After the cases were resolved, case outcomes were compared with the predictions. Overall, lawyers were overconfident in their predictions, and calibration did not increase with years of legal experience. Female lawyers were slightly better calibrated … In an attempt to reduce overconfidence, some lawyers were asked to generate reasons why they might not achieve their stated goals. This manipulation did not improve calibration.” (more)

I strongly suspect these patterns are driven mostly by customers, i.e., that more accurate professionals would be less successful in inspiring confidence by others in them.  If you are a successful professional, that is probably in part because of your unjustified arrogance.

Added: Carl reminds us of an ’06 post on overconfident software managers.

GD Star Rating
a WordPress rating system
Tagged as: , , ,
Trackback URL:
  • http://www.franchise-info.ca Michael Webster

    Robin, there is a much better study of attorneys – which supports your basic thesis.

    Google “Randall Kiser” or “Let’s not make a deal”.

    In a nutshell, his studies going back almost 40 years now show that
    a) plaintiff attorneys have no better than chance in evaluating their case, and;
    b) while defendant attorneys are correct 75% of the time, when they are wrong the mistake is over $1 million dollars.

    This is appalling statistic for my profession. How on earth can we hold ourselves out as group of experts, if Kiser’s numbers are right?

    • http://religionsetspolitics.blogspot.com/ Joshua Zelinsky

      I’m not sure that that’s a great test. If there’s a very good reason to think a case will go one way or the other then I’d expect a settlement to be much more likely. The cases that go all the way should be the cases which are harder to predict the outcomes. That makes this sort of result still appalling but not quite as appalling.

  • the quiet one

    Do you consider yourself a successful professional, Robin?

  • http://steamboatdreaming.blogspot.com Dan Hill

    I don’t have the reference handy, but there’s a study that shows that IT project managers who estimate shorter schedules receive better evaluations from senior management than project managers who provide more accurate but longer estimates.

    Should we be surprised that without incentives that are strongly aligned with rewarding accuracy and sound mechanisms to objectively measure actual performance against estimates, it’s actually better to tell people what they want to hear than it is to tell the truth?

    • Carl Shulman

      It was posted on OB here.

  • Pingback: Tweets that mention Overcoming Bias : Arrogant Professionals -- Topsy.com

  • http://danieltarmac.blogspot.com/ Henry

    Whenever I see bias, I think: is there a way to make money from this? Unfortunately, most people seem averse to “gambling”, so betting against their mistaken beliefs doesn’t often work in practice. A risk calibration consultant could help bring their beliefs back into line with reality, but the problem is getting them to realise they have a problem in the first place.

  • http://thelateenlightenment.blogspot.com Michael Caton

    Robin – regarding the decisions of docs in hospice care, I think there are some very good reasons. First, if you or a loved one are terminally ill, and the doc discloses that treatment X gives you only a 5% chance of extending your life 6 months or more, do you still want it? Almost certainly, and the doc will still want to give it, on the chance that it can help you. Most patients will be happier if they’re receiving any treatment that has a remote chance of doing something, rather than accepting fate, and if there is no complicating question of a diminished quality of life as a result of that treatment, then it’s a no-brainer. (There often is that complication.)

    Second, my experience as a former drug developer and current medical student accords with the article’s claim that most oncologists are very reluctant to provide specific life expectancies – for exactly the reason that most of them are well aware that given our current state of knowledge, it’s a guessing game. So it seems strange that the article then goes into detail about how badly the oncologists guess. Yes! They know that! That’s why they’re so reluctant, you dummies! But the story seems to overlook this glaring inconsistency to continue playing into the old stereotype of physicians who think they’re God and are unable to gauge their own biases and shortcomings, and I think it’s about time we put that to bed. So far in my education I haven’t met any of those. Irritable and sleep-deprived docs, yes, a few. But suffering from wild hubris about their own knowledge? Not so far.

    • http://williambswift.blogspot.com/ billswift

      That doesn’t account for it being 5x too long. If it was just ignorance you would expect a fairly broad distribution around the correct average, or a little high do to normal optimism; there is something else going on here, too.

  • http://timtyler.org/ Tim Tyler

    Re: “If you are a successful professional, that is probably in part because of your unjustified arrogance.”

    If there is a causal association between arrogance and success, that would probably act as satisfactory justification for many.

  • Zvi Mowshowitz

    If we are used to this level of arrogance, and in my experience we are, then we calibrate the predictions of such people for them based on these high levels of arrogance. I would have assumed that in all the situations described above the professional is probably either overconfident, lying about the odds or both. In order to communicate accurate predictions they then need to keep making the same mistake as everyone else. In my experience these people usually know they are not well calibrated and when they need to be otherwise their answers change but much prefer the miscalibration most of the time.

  • http://daedalus2u.blogspot.com/ daedalus2u

    There is a thread just starting on LW that connects the irrational bias when economic choices are framed as a potential gain or as a potential loss in the context of end of life decisions.

    http://lesswrong.com/lw/2jd/open_thread_august_2010/2cnz?c=1

    I think these are both caused by the same type of bias.

  • http://daedalus2u.blogspot.com/ daedalus2u

    With people of limited ability modesty is merely honesty. But with those who possess great talent it is hypocrisy.

    Arthur Schopenhauer

  • JohnJ

    This reminds me of the “women see modesty in men as a sign of weakness” story. Perhaps there is a link between “successful” dating and arrogance?

  • http://www.slowCapital.com Karl Mattingly

    “unjustified arrogance” may be a universal trait.

    This was brought home to me when Burton Malkiel got us all to stand in a room and close our eyes. He then asked all those who knew they were an above average driver to raise their hands. Then he asked us to open our eyes and look around 95% of us had our hands up.

    I think he suggested [but I am rusty on my memory here] that if he had asked us the same question about our “investment savvy” the answer would be the same.

    And that in a nutshell is why Vanguard is an Index Fund manager not an Active Fund manager.

    What evolutionary advantage could this “unjustified arrogance” give us, when it is clearly not accuracy in predictions?

  • Pieter

    Most likely the disadvantage of making inaccurate predictions is outweighed by the advantage of higher social status.

    Leading your nomad tribe into bad weather or poor hunting grounds may les of a disadvantage as not being its leader.

  • http://kaggle.com Anthony Goldbloom

    Interesting post!

    Gruen Tenders turn out to be a simple way to induce an unbiased prognosis.

    It requires two pieces of information:
    1. profit projections (for example);
    2. the degree to which different financial controllers tend overestimate.

    Then simply adjust the profit projection by the degree of overconfidence.

  • Pingback: Tweets that mention Overcoming Bias : Arrogant Professionals -- Topsy.com

  • Robert Koslover

    Interesting. How accurate are the predictions of professors of economics?

  • mlb

    Federal Reserve chairmen come to mind.

  • http://enthusiasm.cozy.org/ Ben Hyde

    One theory for this pattern which I like, possibly since I gin’d it up myself, is that this easy to comprehended using a frame work from ecology. View it thru dialect of r/K selection theory. That theory is usually used to describe how many offspring a species has. Which strategy gives rise to more arrogant, overconfident offspring? Turning that around and arrogant overconfidence is a sign. Throw in a little survivor bias and your done. Says something about brightsiding doesn’t it?

  • Pingback: Notional Slurry » links for 2010-08-24

  • Pingback: Weekly Roundup 90: A Curated Linkfest For The Smartest People On The Web | SimoleonSense