Another lesson is that it’s crucial to have multiple points of view. All the participants in that particular market were software developers.

Do you recall what the price trend was? Was the probability falling as time went on?

Here’s a concrete example. I tried to engage in a little noise trading on tradesports during the period of uncertainty before Bush announced the Alito nomination. I ended up losing the (fortunately small) amount of money I put into it when I bet heavily against Alito about 4 hours before the nomination was announced, when he was running very, very high. My reasoning ran as follows: “There is no public information pointing more to Alito than anyone else, and the price seems too high for people with inside information to be responsible for the whole value. Therefore there must be some kind of irrationality at work, perhaps one of Cass Sunstein’s cascades. The price should go down.” Obviously, this was wrong. But to this day, I don’t know why. Maybe people with inside information put a lot more money in the market than I thought. Maybe a lot of people were a lot better at analyzing the political situation than I. And if that’s true, I don’t know what information or reasoning they used that I missed. Maybe a reliable expert made an announcement, supported by a very convincing argument, that I missed. There’s no way for me to learn how everyone other than me knew it was to be Alito four hours before.

By contrast, if some accurate pundit had made the prediction on the basis of arguments and it had turned out true, I’d have new information about the truth of those arguments, and so would the rest of the society. By giving that hypothetical accurate pundit a financial incentive to secretly bet on the result instead of argue for the result, the public was deprived of potentially a lot of information.

For instance, Joe could have a bias in favour of his local sports team. If Joe lives in the Bronx, he could always believe that the Yankees will win the World Series. The fact that he’d be right more often than a fan of any other baseball team doesn’t prove that his process is unbiased.

We need to consider, in other words, both validity and reliability, not just reliability.

1) Look at two stocks with nearly-identical properties, but with one subject to a test condition (say a “famous” CEO) and one free of that condition. It’s not a perfect test (lots of confounding factors), but if you do it enough times you do get a sense for the tendency of the market to be subject to the specific bias you’re testing. Many of the papers in the site linked above do just that – compare stocks with media attention to those without media attention, for instance, and find the degree to which media attention biases price.

2) Look at populations with specific biases. For instance, you can compare women’s investment behaviour to men’s investment behaviour. To the extent that they behave differently you can imply that the market will be biased towards the way that men view value (since far more men are involved in the market than women). Professor Odean found that women are better investors by a statistically significant margin, suggesting to me that they may be able to exploit the inefficiencies in the overall market caused by male biases swamping female biases in overall market pricing.

Nicholas asks a great question about absolute level of error in markets. Unfortunately, Nicholas, the trick is not to compare actual returns versus price, it’s to compare risk-adjusted actual returns versus price.

What you end up finding is that the ‘rational’ risk-adjustment is quite different from the ‘observed’ risk-adjustment. The observed is also possibly nonlinear and difficult to model theoretically (and tax-distorted and all kinds of other garbage), although interesting inroads have been made using Bayesian models.

So you’d get a very different value between market price and PV of returns, but it would be very diffuclt to say how much of that is due to errors in the market’s judgement of risk and how much is due to errors in the market’s judgement of the value of the underlying. And how much is due to non-market shifts such as tax distortion or demographic changes (baby boomers buying growth stocks in the 70s and income stocks in the 00s).

Of course, all those errors have should have distortionary effects on the predictive value of markets – it’s just hard to do the exact analysis you want to do and draw the conclusion you want to draw, I think. Could be wrong, of course.