Comments on: Test Near, Apply Far http://www.overcomingbias.com/2008/12/test-near-apply.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sat, 11 Feb 2012 23:23:58 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Ian C. http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391323 Ian C. Thu, 04 Dec 2008 14:40:52 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391323 Some things change daily (the weather) and some things don't change for years (gravity). I think that if you want to predict far in to the future you need to have evidence of constancy of time. A local test might mislead you, because it might tell you your prediction is correct when it's only a short term phenomena/fad. But alternatively, If you can show a cause and effect chain from now to what you claim, then you don't need any historical evidence. But to do that you need the design complete basically, so it's not really a prediction any more. Some things change daily (the weather) and some things don’t change for years (gravity). I think that if you want to predict far in to the future you need to have evidence of constancy of time. A local test might mislead you, because it might tell you your prediction is correct when it’s only a short term phenomena/fad.

But alternatively, If you can show a cause and effect chain from now to what you claim, then you don’t need any historical evidence. But to do that you need the design complete basically, so it’s not really a prediction any more.

]]> By: Hal Finney http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391322 Hal Finney Wed, 03 Dec 2008 18:03:42 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391322 As far as modestly long term predictions, commodities futures markets (which are basically betting markets on the future prices of commodities) go out as far as nine years. Current betting is that crude oil will be $85 a barrel in December 2017. You can effectively lock in that price today, with a modest down payment, if you think the betting line is wrong. As far as modestly long term predictions, commodities futures markets (which are basically betting markets on the future prices of commodities) go out as far as nine years. Current betting is that crude oil will be $85 a barrel in December 2017. You can effectively lock in that price today, with a modest down payment, if you think the betting line is wrong.

]]> By: William Pietri http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391321 William Pietri Wed, 03 Dec 2008 17:18:40 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391321 Very interesting! It seems like this insight is being vigorously applied in the world of software development, where over the last 10 or so years a set of short-cycle, iterative methods -- collectively known as "agile software development" -- have come to prominence. Compared with more traditional approaches, with 18 to 36 month release cycles, agile methods release much more frequently. The slowest agile teams might release quarterly, but many release new software weekly or more often still. That relates here because it forces people, even ones with grand visions about the future, to look frequently at how their ideas work in practice. A habit of testing near drives continuous improvement in ideas, gradually winnowing the bazillions of possible ideas to those that really apply. Of course, seeing whether new software really is valuable is a lot easier than testing a theory of new kinds of matter. But I definitely agree that the problem isn't idea generation, it's idea testing. Very interesting!

It seems like this insight is being vigorously applied in the world of software development, where over the last 10 or so years a set of short-cycle, iterative methods — collectively known as “agile software development” — have come to prominence.

Compared with more traditional approaches, with 18 to 36 month release cycles, agile methods release much more frequently. The slowest agile teams might release quarterly, but many release new software weekly or more often still.

That relates here because it forces people, even ones with grand visions about the future, to look frequently at how their ideas work in practice. A habit of testing near drives continuous improvement in ideas, gradually winnowing the bazillions of possible ideas to those that really apply.

Of course, seeing whether new software really is valuable is a lot easier than testing a theory of new kinds of matter. But I definitely agree that the problem isn’t idea generation, it’s idea testing.

]]> By: frelkins http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391320 frelkins Wed, 03 Dec 2008 15:37:13 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391320 As someone who actually runs a prediction market, let me say that the farthest out I've gone so far is 24 months. The hardest part about far-future markets is 1 - ensuring you have chosen the correct and/or valid benchmark for judgement by that time; 2 - keeping traders interested enough in such a distant market. As someone who actually runs a prediction market, let me say that the farthest out I’ve gone so far is 24 months. The hardest part about far-future markets is 1 – ensuring you have chosen the correct and/or valid benchmark for judgement by that time; 2 – keeping traders interested enough in such a distant market.

]]> By: Eliezer Yudkowsky http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391319 Eliezer Yudkowsky Wed, 03 Dec 2008 13:19:16 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391319 Also, can futures markets really forecast distant events at all, if they tie up funds that aren't indexed to the S&P 500 or T-bonds? Also, can futures markets really forecast distant events at all, if they tie up funds that aren’t indexed to the S&P 500 or T-bonds?

]]> By: michael vassar http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391318 michael vassar Wed, 03 Dec 2008 13:06:57 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391318 I would say that this advice is almost perfectly wrong on almost every point. Popperianism is, strictly speaking, wrong, but it is a useful abstraction form many scientists that came basically from thinking about what Einstein in particular did. Historically, it was easier for Newton and Einstein to test their theories far and apply near once measurments improved. If we had a math that specifically predicted the consequences that cause us to invoke Dark Matter on all observed distant scales and that math was fairly elegant we would be done and wouldn't be that upset if it implied nearby experiments that we couldn't realistically actually do. We are bothered by string theory because it doesn't specifically predict our old observations, not just because it doesn't make new nearby predictions. One reason that I have serious concerns about the FAI endeavor is that generating genuinely deep new abstractions seems to be not just hard, but damn near impossible. I don't know anyone who has an obvious track record of doing it even once in the sense that Leibnitz, Aristotle or Hume did it many times. OTOH WRT testing prediction markets, so much the better for the theory if it works for the near term. Buffet's models explicitly don't work for outperforming markets in the short term, but I'm sure he'd be glad if they did. Outperformign futurists is a lower standard and maybe your techniques do. I would say that this advice is almost perfectly wrong on almost every point. Popperianism is, strictly speaking, wrong, but it is a useful abstraction form many scientists that came basically from thinking about what Einstein in particular did. Historically, it was easier for Newton and Einstein to test their theories far and apply near once measurments improved. If we had a math that specifically predicted the consequences that cause us to invoke Dark Matter on all observed distant scales and that math was fairly elegant we would be done and wouldn’t be that upset if it implied nearby experiments that we couldn’t realistically actually do. We are bothered by string theory because it doesn’t specifically predict our old observations, not just because it doesn’t make new nearby predictions. One reason that I have serious concerns about the FAI endeavor is that generating genuinely deep new abstractions seems to be not just hard, but damn near impossible. I don’t know anyone who has an obvious track record of doing it even once in the sense that Leibnitz, Aristotle or Hume did it many times.
OTOH WRT testing prediction markets, so much the better for the theory if it works for the near term. Buffet’s models explicitly don’t work for outperforming markets in the short term, but I’m sure he’d be glad if they did. Outperformign futurists is a lower standard and maybe your techniques do.

]]> By: Robin Hanson http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391317 Robin Hanson Wed, 03 Dec 2008 12:54:58 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391317 Eliezer, it is very hard to say what sort of other experience and evidence there would have been "near" hypothetical creatures who know of Earth history before humans, to guess if that evidence would have been enough to guide them to good abstractions to help them anticipate and describe the arrival of humans. For some possible creatures, they may well not have had enough to do a decent job. Eliezer, it is very hard to say what sort of other experience and evidence there would have been “near” hypothetical creatures who know of Earth history before humans, to guess if that evidence would have been enough to guide them to good abstractions to help them anticipate and describe the arrival of humans. For some possible creatures, they may well not have had enough to do a decent job.

]]> By: Eliezer Yudkowsky http://www.overcomingbias.com/2008/12/test-near-apply.html#comment-391316 Eliezer Yudkowsky Wed, 03 Dec 2008 12:11:22 +0000 http://prod.ob.trike.com.au/2008/12/test-near-apply-far.html#comment-391316 Considering the historical case of the advent of human intelligence, how would you have wanted to handle it using only abstractions that could have been tested before human intelligence showed up? (This being one way of testing your abstraction about abstractions...) We recently had a cute little "black swan" in our financial markets. It wasn't really very black. But some people predicted it well enough to make money off it, and some people didn't. Do you think that someone could have triumphed using your advice here, with regards to that particular event which is now near to us? If so, how? Considering the historical case of the advent of human intelligence, how would you have wanted to handle it using only abstractions that could have been tested before human intelligence showed up?

(This being one way of testing your abstraction about abstractions…)

We recently had a cute little “black swan” in our financial markets. It wasn’t really very black. But some people predicted it well enough to make money off it, and some people didn’t. Do you think that someone could have triumphed using your advice here, with regards to that particular event which is now near to us? If so, how?

]]>