I want to follow up on my earlier post on Philip E. Tetlock’s book, Expert Political Judgment, and in particular his discovery of differential predictive accuracy between individuals with the cognitive styles corresponding to "Foxes" vs "Hedgehogs". As several commenters guessed, his study found that Foxes (who have a flexible, adaptive, tentative cognitive style) significantly outperformed Hedgehogs (who are said to "know one thing and know it well" and to focus on a single, coherent theoretical framework in their analyses and predictions).
Can this modeling science be extrapolated to retail politics? For example, can one model a particular issue in politics and see what the model predicts and then use that prediction as a straw man or context from which to assess the competing mainstream arguments and policy choices. If nothing else, that ought to force more objectivity into what is now a mostly irrational and incoherent process. It also might make the always confident but usually wrong pundits and ideologues explain themselves at least some.
I understand from Tetlock's book that there are caveats and weaknesses, but the data from the better models suggested that maybe it is time to test this technology in the real world, even if the science is in its infancy.
I scored a +54 without having the bias of reading the book first. I imagine with the scores for each question not known to me ahead of time I might have hit down into the low 40s, it became obvious enough after a small number of questions that "my side" was the positive scoring one.
The biggest surprise to me is that Robin at -7 produces a blog I find so incredibly interesting.
Lamont, Owen A. "Macroeconomic Forecasts And Microeconomic Forecasters," Journal of Economic Behavior and Organization, 2002, v48(3,Jul), 265-280.
Abstract: "In the presence of principal-agent problems, published macroeconomic forecasts by professional economists may not measure expectations. Forecasters may use their forecasts in order to manipulate beliefs about their ability. I test a cross-sectional implication of models of reputation and information-revelation. I find that as forecasters become older and more established, they produce more radical forecasts. Since these more radical forecasts are in general less accurate, ex post forecast accuracy grows significantly worse as forecasters become older and more established. These findings indicate that reputational factors are at work in professional macroeconomic forecasts."
Tetlock's book is great and will be a classic. My main worry is that his results will not generalize beyond the area of foreign policy experts, who have obvious incentives to gain attention by pontificating some impressive theory to explain the muddled mess that is world events. These experts are rarely in charge of foreign policy, and so their expertize is clearly not based on how well they manage it; they gain fame instead by standing out from the crowd with an understandable position. Their audience may care less about their predictive accuracy than about their ability to spin a compelling and flattering narrative.
Can this modeling science be extrapolated to retail politics? For example, can one model a particular issue in politics and see what the model predicts and then use that prediction as a straw man or context from which to assess the competing mainstream arguments and policy choices. If nothing else, that ought to force more objectivity into what is now a mostly irrational and incoherent process. It also might make the always confident but usually wrong pundits and ideologues explain themselves at least some.
I understand from Tetlock's book that there are caveats and weaknesses, but the data from the better models suggested that maybe it is time to test this technology in the real world, even if the science is in its infancy.
New Yorker link now at http://www.newyorker.com/ar...
I scored a +54 without having the bias of reading the book first. I imagine with the scores for each question not known to me ahead of time I might have hit down into the low 40s, it became obvious enough after a small number of questions that "my side" was the positive scoring one.
The biggest surprise to me is that Robin at -7 produces a blog I find so incredibly interesting.
Reading a blog entry from last month by Greg Mankiw, a commentor mentioned this paper which discusses some similar failings among economists:
http://www.som.yale.edu/fac...
Lamont, Owen A. "Macroeconomic Forecasts And Microeconomic Forecasters," Journal of Economic Behavior and Organization, 2002, v48(3,Jul), 265-280.
Abstract: "In the presence of principal-agent problems, published macroeconomic forecasts by professional economists may not measure expectations. Forecasters may use their forecasts in order to manipulate beliefs about their ability. I test a cross-sectional implication of models of reputation and information-revelation. I find that as forecasters become older and more established, they produce more radical forecasts. Since these more radical forecasts are in general less accurate, ex post forecast accuracy grows significantly worse as forecasters become older and more established. These findings indicate that reputational factors are at work in professional macroeconomic forecasts."
"Hedgehogs actually do worse in their own fields where they are supposed experts than when they are forced to make predictions in other areas."
That doesn't sound bizarre to me given that hedgehog experts fare worse than chimps. Presumably hedgehog non-experts become more like chimps (random).
Tetlock's book is great and will be a classic. My main worry is that his results will not generalize beyond the area of foreign policy experts, who have obvious incentives to gain attention by pontificating some impressive theory to explain the muddled mess that is world events. These experts are rarely in charge of foreign policy, and so their expertize is clearly not based on how well they manage it; they gain fame instead by standing out from the crowd with an understandable position. Their audience may care less about their predictive accuracy than about their ability to spin a compelling and flattering narrative.