Tests For Hedgehogs?

Philip Tetlock famously showed that hedgehogs, who focus on one main analytical tool, are less accurate than foxes, who used a wide assortment of analytical tools, on simple long-term forecasts in political economy.

Over at Cato Unbound, two famous hedgehogs recently replied to Tetlock. John Cochrane argued that no one can do well at the unconditional forecasts that Tetlock studied, but that hedgehogs shine at conditional forecasts, such as GDP change given a big stimulus. Bruce Bueno De Mesquita noted that his hedgehoggy use of game theory is liked by the CIA and by peer review.

Today at Cato Unbound, I note that since Tetlock’s data is hardly universal, that leaves room for counter-claims that he missed important ways in which hedgehogs are more accurate. But I find it disappointing, and also a bit suspicious, that neither Cochrane nor De Mesquita express interest in helping to design better studies, much less in participating in such studies. I note that “it is certainly possible to collect and score accuracy on conditional forecasts”, and conclude:

Research patrons eager to fund hedgehoggy research by folks like Cochrane and De Mesquita show little interest in funding forecasting competitions at the scale required to get public participation by such prestigious folks. So hedgehogs like Cochrane and De Mesquita can continue to claim superior accuracy, with little fear of being proven wrong anytime soon. All of which brings us back to our puzzling disinterest in forecast accuracy, which was the subject of my response.

GD Star Rating
a WordPress rating system
Tagged as: , ,
Trackback URL:
  • dWj

    On a slightly different note, though, I wonder whether a futures market populated by a variety of hedgehogs (and no foxes) would do better than a futures market populated by foxes.

  • Douglas Knight

    The CIA doesn’t just “like” Bueno de Mesquita. They actually did a large test of their analysts against their people implementing his method:

    Stanley Feder, “Factions and Policon: New Ways to Analyze Politics,” in H. Bradford Westerfield, ed. Inside CIA’s Private World: Declassified Articles from the Agency’s Internal journal, 1955-1992 (New Haven: Yale University Press, 1995) (more citations of studies)

    So the few people who actually care about tests already buy his services. But they are so few that he thinks it not worthwhile to clearly advertise this test. He has nothing to gain from further tests.

  • Mitchell Silverman

    So why don’t we test this. There are plenty of pundits willing to predict next quarters GDP, employment data etc. Why not have a website that lists pundits and experts and their prediction accuracy? It would be a lot more fun following the advice of these people if I could look them up and see their forecast accuracy rate. This seems a worthwhile thing to do. I would use it. But I don’t want to maintain it.
    Any takers?
    Could this be an on-going class project for course in journalism or economics in general ?

    • Wonks Anonymous

      There used to be this site:
      http://www.overcomingbias.com/2009/04/wrong-tomorrow.html
      but it seems to be down at the moment.

    • Douglas Knight

      There still is Prediction Book.

      A basic problem is that no is willing to be graded. Tetlock found no one willing to participate under their own name. Another problem is that people do not make predictions that are easy to score.

      • Mitchell Silverman

        Yes – no one wants to be graded. I wasn’t going to ask them.
        As I think about it, this is probably a great project for a journalism or econ class. Every year the class will spend some of the course researching who’s predictions ought to be tracked. And then accumulate the predictions based on publicly available material. Whip it up into a nice spreadsheet and pop it into a wiki. Allow people to comment. Reserve the top comment spot for the predictor so he/she can explain why they are being misrepresented. update every term.
        Teaches research skills at a modest level, teaches how to find the “leading lights” and does some public good along the way.

  • http://entitledtoanopinion.wordpress.com TGGP

    For those who don’t keep their browser on Cato Unbound reloading for each update, Bruce Bueno de Mesquita responded to Robin’s follow-up and Robin in turn gave kudos that Mesquita subjected his models to tests against others, but wondered whether there was sufficient representation of “foxes” in such comparisons.

  • Sebastian

    “But I find it disappointing, and also a bit suspicious, that neither Cochrane nor De Mesquita express interest in helping to design better studies”

    It isn’t their one big area of analysis (hedgehog) so they don’t care about it as much as someone who likes lots of tools (fox).

  • Pingback: “I have a working knowledge of the entire universe and everything it contains” | Above the Market