Keeping score
A while back Robin Hanson mentioned that somebody should design a mechanism keep to track of pundits’ ‘scores’. Robin notes that there is no feedback mechanism to help the public figure out how accurate pundits’ predictions have been. Moreover, given the way the media works, it’s not clear they have an incentive to find accurate/reliable pundits, rather than entertaining and/or provocative ones. Currently, ‘pundits’ don’t suffer a reputational cost for their blunders. (Anybody recall weapons of mass destruction?) An academic way to go, of course, is to write a critical journal article (also nice at tenure time). For example, the economist/philosopher Erik Angner has written a very nice paper (2006), ‘Economists as Experts: Overconfidence in theory and practice,’ Journal of Economic Methodology 13(1): 1-24 [Fulltext (subscription required); Penultimate draft] in which he analyzes the (somewhat un-inspiring) track-record of Anders Aslund. (Aslund, you may recall, was the Swedish economists that suggested ‘shock therapy’ when he acted as an advisor to the Russian government between 1991 and 1994.) Of course, few pundits deserve such thorough treatment. So, we still await a nice mechanism to ‘score’ and aggregate pundits’ track-records (in the way, say, Ebay merchants are scored). Given the role pundits & talking-heads play in validating (and creating a narrative for) important public policy decisions, this could perform an important public service role. Of course, there is a catch-22 lurking here because it probably requires ‘experts’ to rate/score the ‘experts.’ But given that various foundations are willing to spend serious money on tracking media bias, why not fund a pundit score-keeping institute? In a future post, I’ll discuss the ‘serious’ business of science–and the need for score-keeping in it.
It might not be all that relevant, but the best piece on "shock therapy" I've read is Comparing Apples. Usually Russia is the only country discussed, but as that points out, it is something of an out-lier when it comes to its development after the fall of the USSR. The term shock therapy originally came from other former Soviet countries, but it didn't become well known until Russia tried to imitate them.
This post has pretty much been copied from other ones I made at gnxp and Sailer, which I was reminded of by the mention of Aslund, who tries to defend his reputation with regard to Russia here.
You make a good point. What if you were to weight individual judgments by the the judge's own "truthiness" rating for past claims (i.e. their NAV value)? I can see a bootstrap issue, but here's a proposed process: For phase one we only allow Futures claims (not Currents). During this phase, a base of expert predictors emerges. Phase two, we introduce Currents (which are more subjective), but at this point the experts will have a voice commensurate with their track record on prediction. Judges that are new to the system start with a baseline but can build a reputation over time.