Comments on: Simple Forecasts Best

By: Questions and Answers « Greenhorn Capitalists

Questions and Answers « Greenhorn Capitalists — Fri, 17 Jul 2009 03:17:16 +0000

[...] July 16, 2009 · Leave a Comment 1. Do the latest statistical methods beat a simple moving average? (Robin Hanson). [...]

By: Aron

Aron — Fri, 17 Jul 2009 01:10:03 +0000

I can confirm that the models at work in the NetflixPrize are ridiculously simple. Rather than complex bayesian statistical formulations and multilevel models, you have early stopping with a little ridge regression. I was humored to see the earlier reference to single exponential smoothing. I had just used something similar to that to great effect.

Though it’s not clear to me how competitive the contest was. On multiple occasions, what were essentially amateurs scaled to the top 10 within a few months of beginning their efforts. I have a model that beats the best published ones, and I don’t really have any clue what I’m doing.

By: Eliezer Yudkowsky

Eliezer Yudkowsky — Thu, 16 Jul 2009 18:08:09 +0000

Same comment as Phil Goetz. Empirically on the high-stakes ultra-competitive Netflix Prize, the best performance was not put forth by simple models but by combining many models ranging from simple to complex. But conversely, most statisticians who tried their hand at the Netflix Prize did much more poorly than the best performers. We may be looking at inadequate incentives, inadequate controls for overfitting, prestigious folk who are not the best performers, prestigious folk who overuse complex and impressive models with inadequate checking, or it may just be an empirical fact (though it would surprise me and I would have expected the opposite) that the machine learning community has its act together and the statistical learning community doesn’t.

By: Overcoming Bias : How Exceptional Is Gelman?

Overcoming Bias : How Exceptional Is Gelman? — Thu, 16 Jul 2009 16:07:31 +0000

[...] week I mentioned that fancy stat forecasts are consistently beat by simple moving averages; have you done field [...]

By: Jason Ruspini

Jason Ruspini — Fri, 10 Jul 2009 14:29:15 +0000

This is why I don’t use things like neural networks to forecast prices. A neural network involves many parameters (weights).. the essence of over-fitting.

By: Simple Forecasts Best | Midas Oracle .ORG

Simple Forecasts Best | Midas Oracle .ORG — Fri, 10 Jul 2009 09:35:20 +0000

[...] Simple Forecasts Best Written by Chris F. Masse on July 10, 2009 — Leave a Comment Simple Forecasts Best [...]

By: Michael Bishop

Michael Bishop — Thu, 09 Jul 2009 16:46:24 +0000

Plenty of statisticians, and even social scientists, know that trick… you may be correct that they use it less often though.

By: The Ambrosini Critique » Blog Archive » Macroeconomic forecasting

The Ambrosini Critique » Blog Archive » Macroeconomic forecasting — Wed, 08 Jul 2009 20:50:28 +0000

[...] (1) almost always beats (2). If you wanna do (1), it is pretty straight forward to do in Excel’s analysis [...]

By: Phil Goetz

Phil Goetz — Wed, 08 Jul 2009 18:24:28 +0000

I wonder if machine learning experts would have done better than statisticians. It sounds to me like the statisticians are overfitting their models. ML practitioners optimize their model capacity and parameters using cross-validation.

By: Annoyed

Annoyed — Tue, 07 Jul 2009 22:34:21 +0000

Shouldn’t seeing a passage like

simple, boss-pleasing techniques turned out to be more accurate than the statisticians’ clever, statistically sophisticated methods

in a work of Business Nonfiction make us wonder to what extent the author is seeking after truth and to what extent is he trying to flatter his readers?