Comments on: No Overconfidence? http://www.overcomingbias.com/2008/12/no-overconfidence.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sun, 12 Feb 2012 01:09:57 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Juan Dubra http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389828 Juan Dubra Wed, 28 Jan 2009 02:58:42 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389828 I meant to say that in Svenson's study, 82.5% of the people placed themselves in the top 30% (not the top half as I said in my previous post). So there was no reason to expect that only 52% would place themselves in the top 30%. So adding this fact, the self selection recruitment, and the high motivation, we think we gave overconfidence a "fair shot" to reveal itself (as the person in the first post asks). I meant to say that in Svenson’s study, 82.5% of the people placed themselves in the top 30% (not the top half as I said in my previous post). So there was no reason to expect that only 52% would place themselves in the top 30%. So adding this fact, the self selection recruitment, and the high motivation, we think we gave overconfidence a “fair shot” to reveal itself (as the person in the first post asks).

]]> By: Juan Dubra http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389827 Juan Dubra Mon, 26 Jan 2009 17:38:30 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389827 Hi to all, and thanks for your interest. Regarding the first comment: first of all, we developed a theory of what would constitute a proper test of overconfidence. With this theory, we found that the two tests which were properly run found no overconfidence (tests based on "scales"; I know of no other tests that would be proper tests). Also, our theory allowed us to build the proper test that is referenced in the original post. We run the two treatments referenced in the firt comment because we sort of expected to find overconfidence: people took the original Svenson study quite seriously, and his data showed that 82.5% of the people placed themselves in the top half of the population; we selected people using a "self selection" advertisement (as in Camerer and Lovallo's AER paper) expecting to push people further into being more overconfident; finally, we had a treatment with "high motivation" which we expected to have the effect of making people more overconfident. Regarding Chris's comment: we take care of that issue in the paper. Regarding Zubon's comment: that is a good idea; you should look at Dunning's papers (the ones we reference in our work). best to all Juan Hi to all, and thanks for your interest.

Regarding the first comment: first of all, we developed a theory of what would constitute a proper test of overconfidence. With this theory, we found that the two tests which were properly run found no overconfidence (tests based on “scales”; I know of no other tests that would be proper tests). Also, our theory allowed us to build the proper test that is referenced in the original post. We run the two treatments referenced in the firt comment because we sort of expected to find overconfidence: people took the original Svenson study quite seriously, and his data showed that 82.5% of the people placed themselves in the top half of the population; we selected people using a “self selection” advertisement (as in Camerer and Lovallo’s AER paper) expecting to push people further into being more overconfident; finally, we had a treatment with “high motivation” which we expected to have the effect of making people more overconfident.

Regarding Chris’s comment: we take care of that issue in the paper.

Regarding Zubon’s comment: that is a good idea; you should look at Dunning’s papers (the ones we reference in our work).

best to all
Juan

]]> By: Don Geddis http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389826 Don Geddis Mon, 22 Dec 2008 21:20:08 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389826 Zubon echoes my interpretation. People are almost certainly not using consistent definitions for what "average <i>X</i>" even means. Zubon echoes my interpretation. People are almost certainly not using consistent definitions for what “average X” even means.

]]> By: Zubon http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389825 Zubon Mon, 22 Dec 2008 18:42:34 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389825 Alice thinks she is a better than average driver. By this, she means that she drives safely and has never had a crash. Bob thinks he is a better than average driver. By this, he means that he never speeds and obeys all the signs. Carla thinks she is a better than average driver. By this, she means that she drive very quickly and efficiently, passing idiots like Alice and Bob who take forever to get anywhere. David thinks he is a better than average driver. By this, he means he is a man, because you know how those woman drivers are. Alice thinks she is a better than average driver. By this, she means that she drives safely and has never had a crash.

Bob thinks he is a better than average driver. By this, he means that he never speeds and obeys all the signs.

Carla thinks she is a better than average driver. By this, she means that she drive very quickly and efficiently, passing idiots like Alice and Bob who take forever to get anywhere.

David thinks he is a better than average driver. By this, he means he is a man, because you know how those woman drivers are.

]]> By: Chris http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389824 Chris Mon, 22 Dec 2008 18:16:40 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389824 Yah, so what if a majority of people think they're better than average? If there are extreme outliers in the other direction, this could certainly happen. Shouldn't we be looking at if people think they're better than the median? Yah, so what if a majority of people think they’re better than average? If there are extreme outliers in the other direction, this could certainly happen. Shouldn’t we be looking at if people think they’re better than the median?

]]> By: Unnamed http://www.overcomingbias.com/2008/12/no-overconfidence.html#comment-389823 Unnamed Mon, 22 Dec 2008 15:10:47 +0000 http://prod.ob.trike.com.au/2008/12/no-overconfidence.html#comment-389823 Their <i>theory</i> questions the previous research on overconfidence, but their <i>data</i> don't say much. The basic idea behind their theory is that each person's expectations about their own performance (or beliefs about their abilities) have a probability distribution, and saying you're above average just means that you think there's more than a 50% chance that you're above average. So even if 98% of people say that they're going to place in the top half, that might not indicate overconfidence because some normatively permitted set of underlying probability distributions could produce those data (e.g. if 98% of people each think that they have a 51% chance of being in the top half and 2% think that they have a 1% chance). With the cutoff at 50/50 the data are completely meaningless - any set of data (even 98% vs. 2%) is consistent with their theory. In their study, they shift a little bit away from the 50/50 point, so that it would require extremely high levels of overconfidence to rule out their theory but it is <i>possible</i>. Unsurprisingly, they can't rule it out. Less than 60% of people thought they were more likely than not to place in the top 30% (only 52% thought so), and less than 83% of people thought they had more than a 60% chance of placing in the top half (only 64% did). Thus they cannot reject the hypothesis that people are behaving rationally. Next time, they should ask questions where there are narrower restrictions on the range of theory-consistent answers so that overconfidence has a fair shot to reveal itself. For instance, ask each person "Do you have at least an X% chance of placing in the top half?" for X = 20, 40, 60, and 80 (using incentive compatible gambles, as they do). I bet they'd find people putting too much probability mass on placing in the top half. Their theory questions the previous research on overconfidence, but their data don’t say much. The basic idea behind their theory is that each person’s expectations about their own performance (or beliefs about their abilities) have a probability distribution, and saying you’re above average just means that you think there’s more than a 50% chance that you’re above average. So even if 98% of people say that they’re going to place in the top half, that might not indicate overconfidence because some normatively permitted set of underlying probability distributions could produce those data (e.g. if 98% of people each think that they have a 51% chance of being in the top half and 2% think that they have a 1% chance).

With the cutoff at 50/50 the data are completely meaningless – any set of data (even 98% vs. 2%) is consistent with their theory. In their study, they shift a little bit away from the 50/50 point, so that it would require extremely high levels of overconfidence to rule out their theory but it is possible. Unsurprisingly, they can’t rule it out. Less than 60% of people thought they were more likely than not to place in the top 30% (only 52% thought so), and less than 83% of people thought they had more than a 60% chance of placing in the top half (only 64% did). Thus they cannot reject the hypothesis that people are behaving rationally.

Next time, they should ask questions where there are narrower restrictions on the range of theory-consistent answers so that overconfidence has a fair shot to reveal itself. For instance, ask each person “Do you have at least an X% chance of placing in the top half?” for X = 20, 40, 60, and 80 (using incentive compatible gambles, as they do). I bet they’d find people putting too much probability mass on placing in the top half.

]]>