A fascinating tale, from the book Dance With Chance: During the 1970s … It bugged the professor greatly that [business] practitioners were making these predictions without recourse to the latest, most theoretically sophisticated methods developed by statisticians like himself. Instead, they preferred simpler techniques which – they said – allowed them to explain their forecasts more easily to senior management. The outraged author … embarked on a research project that would demonstrate the superiority of the latest statistical techniques. …
I can confirm that the models at work in the NetflixPrize are ridiculously simple. Rather than complex bayesian statistical formulations and multilevel models, you have early stopping with a little ridge regression. I was humored to see the earlier reference to single exponential smoothing. I had just used something similar to that to great effect.
Though it's not clear to me how competitive the contest was. On multiple occasions, what were essentially amateurs scaled to the top 10 within a few months of beginning their efforts. I have a model that beats the best published ones, and I don't really have any clue what I'm doing.
Same comment as Phil Goetz. Empirically on the high-stakes ultra-competitive Netflix Prize, the best performance was not put forth by simple models but by combining many models ranging from simple to complex. But conversely, most statisticians who tried their hand at the Netflix Prize did much more poorly than the best performers. We may be looking at inadequate incentives, inadequate controls for overfitting, prestigious folk who are not the best performers, prestigious folk who overuse complex and impressive models with inadequate checking, or it may just be an empirical fact (though it would surprise me and I would have expected the opposite) that the machine learning community has its act together and the statistical learning community doesn't.
This is why I don't use things like neural networks to forecast prices. A neural network involves many parameters (weights).. the essence of over-fitting.
I wonder if machine learning experts would have done better than statisticians. It sounds to me like the statisticians are overfitting their models. ML practitioners optimize their model capacity and parameters using cross-validation.
simple, boss-pleasing techniques turned out to be more accurate than the statisticians’ clever, statistically sophisticated methodsin a work of Business Nonfiction make us wonder to what extent the author is seeking after truth and to what extent is he trying to flatter his readers?
The accuracy when various methods areand Hibon study was that simple methods, such being combined outperforms, on average, theas exponential smoothing, outperformed individual methods being combined and doessophisticated ones. Such a conclusion was in very well in comparison to other methods.
Which sounds a lot like maginalising over the parameters - which as I understand it is how a Bayesian would approach prediction
Ahh, thanks. I wasn't expecting him to refer to himself in the third person - and especially as "the professor". Anyway, here is the relevant paper in case anyone is interested:http://www.forecasters.org/...
I can't find a copy of the datasets, but from my very brief reading of the above, it seems that no covariates were used.
In any reasonably open market, anyone who believed they had accurate predictions could act on those predictions. In that scenario it seems like a truism that established market actors would, on the whole, perform better than non-actors.
Of course if the most accurate models increase in complexity, would-be actors might have trouble convincing investors of the worth of their models. But the replacement of old models with new ones should be an ongoing process, making it unlikely that academia, on the whole, could outperform established market actors.
George, looks like the professor was probably Spyros Makridakis, one of the authors of the book. One of his publications, e.g., is called "The Accuracy of Extrapolative (Time Series) Methods: Results of a Forecasting Competition".
Nice tale, but it may just be made up. Not a single name was mentioned. Who was the professor? I want to look up the papers he published. I'm a statistician, so it may affect how I do my work. But maybe that's not the purpose of this tale.
It's not clear whether the business practitioners' predictions were made solely on the abstract data that the statisticians used, or made in a business context at the relevant time, in which case there would have been much other secondary data (including other practitioners' predictions) that might have influenced their predictions. Fair contest?
I believe that a careful review of Tyler's blog will reveal that Tyler believed Gigerenzer's work irrelevant since his latest book was published after Gladwell's "Blink". Oops.
I can confirm that the models at work in the NetflixPrize are ridiculously simple. Rather than complex bayesian statistical formulations and multilevel models, you have early stopping with a little ridge regression. I was humored to see the earlier reference to single exponential smoothing. I had just used something similar to that to great effect.
Though it's not clear to me how competitive the contest was. On multiple occasions, what were essentially amateurs scaled to the top 10 within a few months of beginning their efforts. I have a model that beats the best published ones, and I don't really have any clue what I'm doing.
Same comment as Phil Goetz. Empirically on the high-stakes ultra-competitive Netflix Prize, the best performance was not put forth by simple models but by combining many models ranging from simple to complex. But conversely, most statisticians who tried their hand at the Netflix Prize did much more poorly than the best performers. We may be looking at inadequate incentives, inadequate controls for overfitting, prestigious folk who are not the best performers, prestigious folk who overuse complex and impressive models with inadequate checking, or it may just be an empirical fact (though it would surprise me and I would have expected the opposite) that the machine learning community has its act together and the statistical learning community doesn't.
This is why I don't use things like neural networks to forecast prices. A neural network involves many parameters (weights).. the essence of over-fitting.
Plenty of statisticians, and even social scientists, know that trick... you may be correct that they use it less often though.
I wonder if machine learning experts would have done better than statisticians. It sounds to me like the statisticians are overfitting their models. ML practitioners optimize their model capacity and parameters using cross-validation.
Shouldn't seeing a passage like
simple, boss-pleasing techniques turned out to be more accurate than the statisticians’ clever, statistically sophisticated methodsin a work of Business Nonfiction make us wonder to what extent the author is seeking after truth and to what extent is he trying to flatter his readers?
From the paper : this caught my I
The accuracy when various methods areand Hibon study was that simple methods, such being combined outperforms, on average, theas exponential smoothing, outperformed individual methods being combined and doessophisticated ones. Such a conclusion was in very well in comparison to other methods.
Which sounds a lot like maginalising over the parameters - which as I understand it is how a Bayesian would approach prediction
Robin, did this inspire your previous post about whether smart people help as much as it seems that they do?
Ahh, thanks. I wasn't expecting him to refer to himself in the third person - and especially as "the professor". Anyway, here is the relevant paper in case anyone is interested:http://www.forecasters.org/...
I can't find a copy of the datasets, but from my very brief reading of the above, it seems that no covariates were used.
In any reasonably open market, anyone who believed they had accurate predictions could act on those predictions. In that scenario it seems like a truism that established market actors would, on the whole, perform better than non-actors.
Of course if the most accurate models increase in complexity, would-be actors might have trouble convincing investors of the worth of their models. But the replacement of old models with new ones should be an ongoing process, making it unlikely that academia, on the whole, could outperform established market actors.
George, looks like the professor was probably Spyros Makridakis, one of the authors of the book. One of his publications, e.g., is called "The Accuracy of Extrapolative (Time Series) Methods: Results of a Forecasting Competition".
Nice tale, but it may just be made up. Not a single name was mentioned. Who was the professor? I want to look up the papers he published. I'm a statistician, so it may affect how I do my work. But maybe that's not the purpose of this tale.
It's not clear whether the business practitioners' predictions were made solely on the abstract data that the statisticians used, or made in a business context at the relevant time, in which case there would have been much other secondary data (including other practitioners' predictions) that might have influenced their predictions. Fair contest?
Heck, I was just about to email you the video with Robin Hogarth!
Here: http://knowledge.insead.edu...
This should bring a lot of attention to Gerd Gigerenzer's work:
http://www.edge.org/3rd_cul...
I believe that a careful review of Tyler's blog will reveal that Tyler believed Gigerenzer's work irrelevant since his latest book was published after Gladwell's "Blink". Oops.