Comments on: The 80% Forecasting Solution http://www.overcomingbias.com/2006/12/the_80_forecast.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Thu, 11 Mar 2010 23:02:39 -0500 http://wordpress.org/?v=2.8.5 hourly 1 By: Robin Hanson http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423824 Robin Hanson Sat, 16 Dec 2006 13:24:50 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423824 Hal, it is hard to sympathize with people whose job it is to make forecasts, such as cancer doctors, who are reluctant to be "put on the spot." Hal, it is hard to sympathize with people whose job it is to make forecasts, such as cancer doctors, who are reluctant to be “put on the spot.”

]]> By: Hal Finney http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423823 Hal Finney Sat, 16 Dec 2006 07:35:18 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423823 One thing to watch out for is that many people are reluctant to be put on the spot by having their forecasts recorded and judged. Tetlock ran into this with his study, he had to offer his subjects anonymity. (And it's a good thing for them he did, since they did so badly.) The problem is that it might be that some of the best forecasters are most uncertain about their forecasts (Tetlock's foxes) and therefore might be reluctant to be scored like this, while some of the worst forecasters are overconfident (hedgehogs) and would be overrepresented in the sample. While one might hope that this would be self-correcting to some extent (people who do badly get dropped) it could still reduce the average accuracy of forecasts. One thing to watch out for is that many people are reluctant to be put on the spot by having their forecasts recorded and judged. Tetlock ran into this with his study, he had to offer his subjects anonymity. (And it’s a good thing for them he did, since they did so badly.)

The problem is that it might be that some of the best forecasters are most uncertain about their forecasts (Tetlock’s foxes) and therefore might be reluctant to be scored like this, while some of the worst forecasters are overconfident (hedgehogs) and would be overrepresented in the sample. While one might hope that this would be self-correcting to some extent (people who do badly get dropped) it could still reduce the average accuracy of forecasts.

]]> By: Anders Sandberg http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423822 Anders Sandberg Fri, 15 Dec 2006 09:47:48 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423822 One interesting idea that came up in a discussion yesterday with Stuart Armstrong was to force people to state their predictions as negatives: not what they think will happen, but what will *not* happen or be the case at a certain point in time. Rather than saying "In 2003 workers who are not computer fluent will have a hard time finding a job" we might look at whether "In 2003 workers not fluent in computers can still easily find a job" is untrue. In this case it seems that the first prediction is still hard to judge, while the second is much sharper and probably untrue. I guess this is an application of falsifiability. Human tend to state their predictions in an unfalsifiable positive form, but negating them often produces a sharper falsifiable form. Of course one can do a bad job of it and still fudge, but it seems a list of negative predictions might be more useful than a list of positive predictions for establishing a track record. One interesting idea that came up in a discussion yesterday with Stuart Armstrong was to force people to state their predictions as negatives: not what they think will happen, but what will *not* happen or be the case at a certain point in time. Rather than saying “In 2003 workers who are not computer fluent will have a hard time finding a job” we might look at whether “In 2003 workers not fluent in computers can still easily find a job” is untrue. In this case it seems that the first prediction is still hard to judge, while the second is much sharper and probably untrue. I guess this is an application of falsifiability. Human tend to state their predictions in an unfalsifiable positive form, but negating them often produces a sharper falsifiable form. Of course one can do a bad job of it and still fudge, but it seems a list of negative predictions might be more useful than a list of positive predictions for establishing a track record.

]]> By: Peter McCluskey http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423821 Peter McCluskey Thu, 14 Dec 2006 21:53:50 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423821 Acad: Tetlock showed that there is an important domain for which weighting by expert credentials would work poorly. But he hints that the cause is the way news media reward these so-called experts for making dramatic forecasts. In different domains, experts face different incentives and get reputations in different ways, so I don't expect Tetlock's results to generalize as well as you imply. Perry, here's a blog post explaining why the problem of incorrect papers is worse than you indicate, and a few ways to improve on it: http://www.marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html Acad: Tetlock showed that there is an important domain for which weighting by expert credentials would work poorly. But he hints that the cause is the way news media reward these so-called experts for making dramatic forecasts.
In different domains, experts face different incentives and get reputations in different ways, so I don’t expect Tetlock’s results to generalize as well as you imply.

Perry, here’s a blog post explaining why the problem of incorrect papers is worse than you indicate, and a few ways to improve on it: http://www.marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html

]]> By: jck http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423820 jck Wed, 13 Dec 2006 21:25:43 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423820 jason:"Is there yet a well-maintained registry of investment bank research analyst recommendations?" yes,but it's private,there is a fund in the uk that tracks around 200000 analysts trade recommendations a year,and if they have access to all that research it's because there is a good incentive for the banks: a piece of a commissions budget well north of $200 millions a year.[they have been doing that for close to 10 years]. jason:”Is there yet a well-maintained registry of investment bank research analyst recommendations?”
yes,but it’s private,there is a fund in the uk that tracks around 200000 analysts trade recommendations a year,and if they have access to all that research it’s because there is a good incentive for the banks: a piece of a commissions budget well north of $200 millions a year.[they have been doing that for close to 10 years].

]]> By: Alcibiades http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423819 Alcibiades Wed, 13 Dec 2006 19:21:46 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423819 With abundant blogs and auto-caching, perhaps we're getting close to such a day... Though to be sure, 1) people tend not to make concrete predictions, for fear of later scrutiny, as you've pointed out, and 2) CEOs and presidents don't blog, for the most part. (Mark Cuban is an exception, and makes quite a few concrete predictions too.) Have fun with the Yahooligans! With abundant blogs and auto-caching, perhaps we’re getting close to such a day…
Though to be sure, 1) people tend not to make concrete predictions, for fear of later scrutiny, as you’ve pointed out, and 2) CEOs and presidents don’t blog, for the most part.
(Mark Cuban is an exception, and makes quite a few concrete predictions too.)
Have fun with the Yahooligans!

]]> By: Anders Sandberg http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423818 Anders Sandberg Wed, 13 Dec 2006 18:57:59 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423818 In the case of doctors there is a growing system of track records. In Great Britain there is for example an online system showing survival rates of heart surgery at different hospitals and doctors: http://heartsurgery.healthcarecommission.org.uk/Survival.aspx Of course, presenting this kind of information right is tricky. Some doctors specialize in difficult cases and have a lower survival rate. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=164230 But this is largely a reporting and information design problem. Overall I think the information design aspect is worth pursuing (maybe I should make a post on this since it is a pet subject of mine). A good information design helps people judge tricky information like risk better, like the dartboards and roulette wheels for medical risk communication http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0030137 while bad design of course biases or obscures data. It is an interesting challenge to visualise track records well. In the case of doctors there is a growing system of track records. In Great Britain there is for example an online system showing survival rates of heart surgery at different hospitals and doctors:
http://heartsurgery.healthcarecommission.org.uk/Survival.aspx
Of course, presenting this kind of information right is tricky. Some doctors specialize in difficult cases and have a lower survival rate.
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=164230
But this is largely a reporting and information design problem.

Overall I think the information design aspect is worth pursuing (maybe I should make a post on this since it is a pet subject of mine). A good information design helps people judge tricky information like risk better, like the dartboards and roulette wheels for medical risk communication
http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0030137
while bad design of course biases or obscures data. It is an interesting challenge to visualise track records well.

]]> By: Jason Ruspini http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423817 Jason Ruspini Wed, 13 Dec 2006 18:40:02 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423817 "But if customers cared enough, records would happen." Is there yet a well-maintained registry of investment bank research analyst recommendations? If not, this is at least somewhat related to the reluctance of management to run internal prediction markets.. i.e, outsiders might overrate the demand for certain kinds of information in the power structure of the firm (or in a close-knit community of firms). “But if customers cared enough, records would happen.”

Is there yet a well-maintained registry of investment bank research analyst recommendations?

If not, this is at least somewhat related to the reluctance of management to run internal prediction markets.. i.e, outsiders might overrate the demand for certain kinds of information in the power structure of the firm (or in a close-knit community of firms).

]]> By: pdf23ds http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423816 pdf23ds Wed, 13 Dec 2006 18:20:48 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423816 The collecting of such track records could be a profitable enterprise for people looking to get into prediction market arbitrage. They could capitalize on people who're unwilling to participate in a betting market (because of other interests, or an aversion to gambling), but who have relevant and otherwise unrepresented knowledge. But I share your pessimism that this kind of product would be valued by consumers in general. The collecting of such track records could be a profitable enterprise for people looking to get into prediction market arbitrage. They could capitalize on people who’re unwilling to participate in a betting market (because of other interests, or an aversion to gambling), but who have relevant and otherwise unrepresented knowledge. But I share your pessimism that this kind of product would be valued by consumers in general.

]]> By: Perry E. Metzger http://www.overcomingbias.com/2006/12/the_80_forecast.html#comment-423815 Perry E. Metzger Wed, 13 Dec 2006 17:47:46 +0000 http://prod.ob.trike.com.au/2006/12/the-80-forecasting-solution.html#comment-423815 One must be very careful in the way that one implements such a proposal. Consider that if you have 1024 people making random predictions about 50% probability events, one of them may get a remarkable ten of those predictions right in a row, even though their method of prediction is random. Even if someone has a track record for prediction, one should not necessarily believe them! In case some think this is a theoretical problem, consider how many people participate in certain prediction activities (like investing), and then consider how many of them might, by pure accident, seem to be brilliant analysts when they are in fact just lucky. Once you have millions of points in your sample space, the usual "95% confidence" people prefer in scientific studies is not sufficient. (This reminds me -- I'd love to see a blog posting about the fact that, of every 100 papers out there claiming 95% confidence, by definition 5 are wrong. With the amount of science going on today, is 95% confidence actually a sufficient metric? One can expect rafts of incorrect papers even in quite prestigious journals purely on the basis of statistics even ignoring the possibility of fraud...) One must be very careful in the way that one implements such a proposal. Consider that if you have 1024 people making random predictions about 50% probability events, one of them may get a remarkable ten of those predictions right in a row, even though their method of prediction is random. Even if someone has a track record for prediction, one should not necessarily believe them!

In case some think this is a theoretical problem, consider how many people participate in certain prediction activities (like investing), and then consider how many of them might, by pure accident, seem to be brilliant analysts when they are in fact just lucky. Once you have millions of points in your sample space, the usual “95% confidence” people prefer in scientific studies is not sufficient.

(This reminds me — I’d love to see a blog posting about the fact that, of every 100 papers out there claiming 95% confidence, by definition 5 are wrong. With the amount of science going on today, is 95% confidence actually a sufficient metric? One can expect rafts of incorrect papers even in quite prestigious journals purely on the basis of statistics even ignoring the possibility of fraud…)

]]>