The blogsphere adores Kruger and Dunning’s ‘99 paper "Unskilled and Unaware of It". Google blog search lists ten blog mentions just in the last month. For example:
Perhaps the single academic study most germane to the present election … In short, smart people tend to believe that everyone else "gets it." Incompetent people display both an increasing tendency to overestimate their cognitive abilities and a belief that they are smarter than the majority of those demonstrably sharper.
This paper describes everyone’s favorite theory of those they disagree with, that they are hopelessly confused idiots unable to see they are idiots; no point in listening to or reasoning with such fools. However, many psychologists have noted Kruger and Dunning’s main data is better explained by positing simply that we all have noisy estimates of our ability and of task difficulty. For example, Burson, Larrick, and Klayman’s ‘06 paper "Skilled or Unskilled, but Still Unaware of It":
We replicated, eliminated, or reversed the association between task performance and judgment accuracy reported by Kruger and Dunning (1999) depending on task difficulty. On easy tasks, where there is a positive bias, the best performers are also the most accurate in estimating their standing, but on difficult tasks, where there is a negative bias, the worst performers are the most accurate. This pattern is consistent with a combination of noisy estimates and overall bias, with no need to invoke differences in metacognitive abilities. In this regard, our findings support Krueger and Mueller’s (2002) reinterpretation of Kruger and Dunning’s (1999) findings. An association between task-related skills and metacognitive insight may indeed exist, and later we offer some additional tests using the current data. However, our analyses indicate that the primary drivers of miscalibration in judging percentile are general inaccuracy due to noise and overall biases that arise from task difficulty.
So why does Google blog search finds zero mentions of this refutation? My guess: because under this theory you should listen to those you disagree with instead of writing them off as idiots.
Now Kruger and Dunning do have a 2008 followup paper, and in their first paper they were able to construct one situation where more able people had lower errors in estimating their ability. Also, Burson et al. saw some weak tendencies like this:
We regressed perceived percentile on actual percentile among bottom-half performers and among top half performers. A Chow test confirmed at a marginal level of significance that bottom-half performers were less sensitive to their actual percentile … than were top-half performers.
Oddly, none of the dozen papers on this I’ve read pursue the obvious way to settle this question: look at the variance of ability estimates as a function of ability. But however that turns out it seems clear that mostly what is going on is that we all misjudge our ability and task difficulty.
One Trackback
[...] Fall, weil ich glauben möchte, dass das Festlegen und Aussprechen von Verboten gut gemeint ist und bloß das Wissen fehlt, anders und im besten Fall richtig zu handeln. Zwar ist bekanntlich der Weg zur Hölle gepflastert [...]
11 Comments
All are Skill Unaware
Think you’re smart? That probably means that you are not. From a piece with the above title at Overcoming Bias:…it seems clear that mostly what is going on is that we all misjudge our ability and task difficulty. You can say that again.
So why does Google blog search finds zero mentions of this refutation?
Um, because of confirmation bias? I don’t think one has to bring in any other theories to explain the zero references.
The ‘zero mentions’ link includes the following:
__Is That Juice On Your Face?__
**19 Nov 2007** by Russell Ball
You should also check out “Skilled or Unskilled, but Still Unaware of It” which suggests that ‘competent’ people can be victim to the same effect as the difficulty of the task increases. Left by Rob Crowther on Nov 30, 2007 1:11 PM …
Caffeinated Coder – http://geekswithblogs.net/coredump/Default.aspx – References
Perhaps google’s bot is smart enough to parse OB and realized it should look a bit harder?
Now that I’ve read the findings, how do I use the information to my advantage? Perhaps find specific quantifiable methods for evaluating task performance and use them to recalibrate metacognitive monitoring.
“My guess: because under this theory you should listen to those you disagree with instead of writing them off as idiots.” – yeah – I am sure people would use it this way. But actually it is a double edged sword – it does not give anyone any assurance that they are not on the other side (i.e. that it is them who is unskilled and unaware).
Looks like more support for opening a prediction market on all your projects – because you’re going to underestimate how long they’ll take, how hard they are, and how good your team is.
How does this result argue for “listening to those you disagree with instead of writing them off as idiots?” It sounds like it argues for doubting your positions, but doesn’t tell you how to learn more. And even if your own evaluations are more suspect than you’d like, they’re still all you have to go on. Not trusting your own judgment leaves you exposed to the din without any indication where to start.
I think one key thing the original paper mentioned was how people’s self-assessed percentile ratings changed when they saw other people’s performances: people with good test scores raised their self-assessments to a much more accurate level, while people with poor test scores didn’t change their assessments much.
“What happens when you get feedback” was not addressed by the rebuttal paper (which is probably correct about the “noise plus bias” hypothesis), and only marginally addressed by the original paper.
Trying to evaluate your own intellectual performance is a waste of time, because the only way you can do so is by using your intellect. That leads to massive problems with circularity.
As the saying goes, the easiest person to fool is yourself.
The only solution I am aware of is to find an objective measure of correctness, determine how that method rules on your position, and use the feedback to improve your thoughts. Repeat as necessary. (It’s always necessary.)
This solution is otherwise known as ’science’.
Cameron, that ref still doesn’t appear when I click the link.
Zbigniew, yup.
Doug, yes but that sort of test situation is uncommon.
Chris, a theory where idiots are more common has more support for the hypothesis that the specific person you are talking with is an idiot. And central to the concept of an idiot is that they aren’t worth talking to.
Robin: Fascinating. Still appearing as result 6 of 9 for me, now overshadowed by the several new additions that your own post encouraged. Perhaps personalised search history makes google searching even more subjective than it once was!
* And central to the concept of an idiot is that they aren’t worth talking to.
Are my human biasses leading me to conclude people are not worth talking to (idiots) when in fact I would benefit from such conversations? This particular bias suggest that there probably are more instances than I believe.
More generally, however, I find I give the label ‘idiot’ not so much to people who lack skills but to those who reason less abstractly than I and are unwilling or unable to cross that gap when necessary. In those cases attempting to ‘reason’ leads to conflict independently of any actual difference in stimulus -> response pairings of the respective models. In those situations people can be legitimately ‘idiots who aren’t worth reasoning with’ yet simultaniously worth observing, talking to in a non-reasoning format and learning from.
I’ll put it another way. It is easy to use ‘excuses’ like the earlier findings of “Unskilled and Unaware of It”. It is also easy to say “ha! I’m Biassed! I am just finding excuses to call people idiots and not talk to them! Reverse that conclusion!”. Yet it is far more difficult to arrive at an unbiassed evaluation of which people are in fact worth talking to, independently of social signalling and ego.
Just as I cannot be sure I am smarter than my ethics, I cannot be certain that I’m smarter than my biasses either. For all I know my biasses encourage me to make excuses not to talk to idiots because I’m just too dumb to comprehend the real reason that they’re a waste of my time. Uncovering the bias with the cause for the bias may be counter productive.