In December I argued: I’d guess you can get 80% of the improvement that predict markets offer by using a much simpler solution: collect track records. … When people make forecast-like-statements, write them down in a clear standardized form, and then check back later to see who was more accurate. Along the way, create a consensus forecast by averaging recent forecasts, … If you collect enough forecasts to evaluate accuracy, and reward accuracy well enough, people will try hard to be right, and you’ll learn what kinds of people to listen to.
For the purpose of predicting, if not selling forecasting methods to managers, transparency is desirable.. meaning that the inputs to each trader's profit function, not to mention the function itself, should be known and verifiable by each trader. Whenever a trader's p&l function, real or play, takes the predictions of others as an input, this transparency is lost.
Auctions and algorithms can also be biased via large orders by "the house".. but one must calibrate one's cynicism and doubt.
Robin, what excuses do they give for hiding predictions from employees involved with the project in question? I'd expect that dealing with these excuses is no harder than dealing with the excuses for avoiding prediction markets.
I'm puzzled as to why competing analysts don't have enough incentive to complain about an alteration which causes a competitor to appear to have a better track record. Possibly because all the analysts who are aware that the history can be edited find it better to alter their own record than to blow the whistle on the others? It would seem unusual for a significant number of people to all act that way.For simple internal corporate predictions such as when will a project be completed, it still seems like prediction markets are more complex than needed. If companies adopt a simple rule of rewarding good forecasts with small bonuses, and allocate a fixed amount of total prediction-related bonuses (so that employees have a modest incentive to detect cheating), and each prediction is known to multiple employees, it seems like employees should be able to keep the system honest. Unless the rewards to inaccurate predictions are a good deal higher than my intuition leads me to expect.The black box proposal might reduce transparency enough to enable cheating. I'm unclear about how people try to justify a black box as opposed to keeping a record of predictions that is available to everyone on the project the predictions are about. Am I missing something in your black box reference?
Rafe, if it was easy for observers to tell if these problems were happening, they wouldn't be such problems. I have no special access to StarMine to be able to tell.
There is a company (StarMine.com) which rates stock analysts on the accuracy of their predictions. I'd be curious to know if you think they have any or all of the four biases which you worry about above.
For the purpose of predicting, if not selling forecasting methods to managers, transparency is desirable.. meaning that the inputs to each trader's profit function, not to mention the function itself, should be known and verifiable by each trader. Whenever a trader's p&l function, real or play, takes the predictions of others as an input, this transparency is lost.
Auctions and algorithms can also be biased via large orders by "the house".. but one must calibrate one's cynicism and doubt.
Peter, managers usually hide lots of things from employees; they don't need excuses to hide more, as the subject doesn't come up.
Robin, what excuses do they give for hiding predictions from employees involved with the project in question? I'd expect that dealing with these excuses is no harder than dealing with the excuses for avoiding prediction markets.
Peter, managers are reluctant to lose control over information, and hence are reluctant to allow a public record of predictions.
I'm puzzled as to why competing analysts don't have enough incentive to complain about an alteration which causes a competitor to appear to have a better track record. Possibly because all the analysts who are aware that the history can be edited find it better to alter their own record than to blow the whistle on the others? It would seem unusual for a significant number of people to all act that way.For simple internal corporate predictions such as when will a project be completed, it still seems like prediction markets are more complex than needed. If companies adopt a simple rule of rewarding good forecasts with small bonuses, and allocate a fixed amount of total prediction-related bonuses (so that employees have a modest incentive to detect cheating), and each prediction is known to multiple employees, it seems like employees should be able to keep the system honest. Unless the rewards to inaccurate predictions are a good deal higher than my intuition leads me to expect.The black box proposal might reduce transparency enough to enable cheating. I'm unclear about how people try to justify a black box as opposed to keeping a record of predictions that is available to everyone on the project the predictions are about. Am I missing something in your black box reference?
Rafe, if it was easy for observers to tell if these problems were happening, they wouldn't be such problems. I have no special access to StarMine to be able to tell.
There is a company (StarMine.com) which rates stock analysts on the accuracy of their predictions. I'd be curious to know if you think they have any or all of the four biases which you worry about above.