Comments on: Alas Amateur Futurism http://www.overcomingbias.com/2006/12/alas_amateur_fu.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sat, 11 Feb 2012 23:23:58 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Carl Shulman http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423965 Carl Shulman Sun, 30 Dec 2007 23:18:02 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423965 Alas, indeed. http://www.nytimes.com/2007/12/30/nyregion/thecity/30year.html?_r=1&ref=thecity&oref=slogin Alas, indeed.

http://www.nytimes.com/2007/12/30/nyregion/thecity/30year.html?_r=1&ref=thecity&oref=slogin

]]> By: Robin Hanson http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423964 Robin Hanson Sun, 10 Dec 2006 12:26:15 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423964 Chris A, one can check that probability forecasts are not random guesses by looking at their calibration; for the group of forecasts at with 90% confidence, do 90% turn out to be true? One can certainly have useful information without have an exactly "scientifically" valid model. Chris A, one can check that probability forecasts are not random guesses by looking at their calibration; for the group of forecasts at with 90% confidence, do 90% turn out to be true? One can certainly have useful information without have an exactly “scientifically” valid model.

]]> By: ChrisA http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423963 ChrisA Sun, 10 Dec 2006 08:39:14 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423963 Robin How can you be sure that an accurate prediction is not a lucky guess? Surely the point of being able to make predictions is to make use of them. The only way to be sure that even a string of successes are not lucky guesses is to understand the model that is producing the predictions, and if the model makes sense then the predictions are likely to be valid. But the complexity of reality makes such a model impossible. When you propose prediction markets for general predictions are you not just trying to "black box" the problem? We know that the people making the predictions don't have a real model of reality (since it is impossible) so they are simply guessing, are an average of guesses really better than a simple guess? Now I don't say that specific (as opposed to general) predictions of the future are impossible, for instance the oil and gas industry has got pretty good at predicting how long a project will take to build up to first oil five or six years in advance. There are literally millions of interactions involved in building an oil platform so it is not a trivial problem. However, through experience, oil companies have been able to build very good models of what it takes to do this. When a prediction is presented with probabilistic ranges on the start up date, we can say that these are likely to be representative of the real start up date. Note that even here there are frequent surprises when unexpected things occur. So I can understand that prediction market could be used to reveal information where the participants do have good models, such as the oil industry one. If someone does have a good model they have confidence in then they should be willing to bet more on the result so bringing the result closer to a real one. But how can we distinguish between the results where there is a real model driving the result and the ones where people are simply guessing? You could ask the experts if the problem was one that could be modelled - but in that case why not just ask for the model? This is a genuine problem for me on my projects. Since the budget is necessarily limited we have to decide what risks we should spend money to mitigate and which we should ignore. How do I get reasonable prediction of the probable risks if they have never happened before. I have thought of trying to use prediction markets, but can't get around the issue above. (There is also the problem that if the problem does not occur, then was it because it was successfuly mitigated or it was a bad prediction). Robin
How can you be sure that an accurate prediction is not a lucky guess? Surely the point of being able to make predictions is to make use of them. The only way to be sure that even a string of successes are not lucky guesses is to understand the model that is producing the predictions, and if the model makes sense then the predictions are likely to be valid. But the complexity of reality makes such a model impossible. When you propose prediction markets for general predictions are you not just trying to “black box” the problem? We know that the people making the predictions don’t have a real model of reality (since it is impossible) so they are simply guessing, are an average of guesses really better than a simple guess?

Now I don’t say that specific (as opposed to general) predictions of the future are impossible, for instance the oil and gas industry has got pretty good at predicting how long a project will take to build up to first oil five or six years in advance. There are literally millions of interactions involved in building an oil platform so it is not a trivial problem. However, through experience, oil companies have been able to build very good models of what it takes to do this. When a prediction is presented with probabilistic ranges on the start up date, we can say that these are likely to be representative of the real start up date. Note that even here there are frequent surprises when unexpected things occur.

So I can understand that prediction market could be used to reveal information where the participants do have good models, such as the oil industry one. If someone does have a good model they have confidence in then they should be willing to bet more on the result so bringing the result closer to a real one. But how can we distinguish between the results where there is a real model driving the result and the ones where people are simply guessing? You could ask the experts if the problem was one that could be modelled – but in that case why not just ask for the model?

This is a genuine problem for me on my projects. Since the budget is necessarily limited we have to decide what risks we should spend money to mitigate and which we should ignore. How do I get reasonable prediction of the probable risks if they have never happened before. I have thought of trying to use prediction markets, but can’t get around the issue above. (There is also the problem that if the problem does not occur, then was it because it was successfuly mitigated or it was a bad prediction).

]]> By: Robin Hanson http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423962 Robin Hanson Sun, 10 Dec 2006 01:35:29 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423962 Chris A, I'm much more interested in how accurate predictions are than in how "scientific" they are. Predictions *can* be expressed as probabilities, and they *can* be more or less accurate, even if they do not meet your standards for "scientific." Chris A, I’m much more interested in how accurate predictions are than in how “scientific” they are. Predictions *can* be expressed as probabilities, and they *can* be more or less accurate, even if they do not meet your standards for “scientific.”

]]> By: ChrisA http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423961 ChrisA Sat, 09 Dec 2006 18:10:33 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423961 Robin Surely trying to predict future events with probabilities is literally impossible, or at least highly unscientific. There is no way anyone can model the present in sufficient detail to develop testable predictions, the problem is far too complex, even ignoring the quantum effect. As an example of the difficulty could we feasibly predict the moves in a chess match that will take place a week from now? Predicting the general future is many times harder than that. When people present predictions, they are really presenting scenarios, possible events rather than probable or likely ones. That is why your "prediction" of the net was not profitable for you, for you are unable to calculate a probability around the event occuring. Chris Robin
Surely trying to predict future events with probabilities is literally impossible, or at least highly unscientific. There is no way anyone can model the present in sufficient detail to develop testable predictions, the problem is far too complex, even ignoring the quantum effect. As an example of the difficulty could we feasibly predict the moves in a chess match that will take place a week from now? Predicting the general future is many times harder than that. When people present predictions, they are really presenting scenarios, possible events rather than probable or likely ones. That is why your “prediction” of the net was not profitable for you, for you are unable to calculate a probability around the event occuring.

Chris

]]> By: Robin Hanson http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423960 Robin Hanson Fri, 08 Dec 2006 12:09:09 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423960 Hal, predicting the future seems impossible. Not exactly of course, but better than the usual public consensus. Many people claim that no one predicted the world wide web, but I was part of a group that did predict it well before the public consensus was aware of it. That group did not much benefit from their prediction, and were probably hurt on average, which helps us to understand why people don't do more such predicting. Hal, predicting the future seems impossible. Not exactly of course, but better than the usual public consensus. Many people claim that no one predicted the world wide web, but I was part of a group that did predict it well before the public consensus was aware of it. That group did not much benefit from their prediction, and were probably hurt on average, which helps us to understand why people don’t do more such predicting.

]]> By: Hal Finney http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423959 Hal Finney Fri, 08 Dec 2006 02:47:24 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423959 It may be that predicting the future is essentially impossible. This is suggested by the striking rarity of accurate future predictions combined with the great rewards available to anyone who can succeed at the task. It may be that predicting the future is essentially impossible. This is suggested by the striking rarity of accurate future predictions combined with the great rewards available to anyone who can succeed at the task.

]]> By: Robin Hanson http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423958 Robin Hanson Fri, 08 Dec 2006 01:36:42 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423958 Eliezer, yes, it is useful to think of futurism as show, but what other areas of apparently "serious" idea institutions/groups are also substantially entertainment industries in disguise? Newspapers? Libraries? Universities? Think tanks? Eliezer, yes, it is useful to think of futurism as show, but what other areas of apparently “serious” idea institutions/groups are also substantially entertainment industries in disguise? Newspapers? Libraries? Universities? Think tanks?

]]> By: Eliezer Yudkowsky http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423957 Eliezer Yudkowsky Fri, 08 Dec 2006 01:05:50 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423957 Futurism is a branch of the entertainment industry. Deal with it. Futurism is a branch of the entertainment industry. Deal with it.

]]> By: Robin Hanson http://www.overcomingbias.com/2006/12/alas_amateur_fu.html#comment-423956 Robin Hanson Fri, 08 Dec 2006 00:38:41 +0000 http://prod.ob.trike.com.au/2006/12/alas-amateur-futurism.html#comment-423956 Nick, good point, yes an important fifth group is our-new-direction speakers and facilitators. Nick, good point, yes an important fifth group is our-new-direction speakers and facilitators.

]]>