A Note on Simple Models

What implicitly come out from the example he uses is that models are often simple from the point of view of those making them, but only them (Bayseian analysis, for instance, is only simple once you’ve properly understood it). Might be another bias; not sure of its implications here.

Ah, I see. You were suggesting that complicated models are prone to bias because of what their creators put into them; I was suggesting that simple models are prone to bias because they are less likely to be rejected when contrary evidence arrives.

Is this the case? Politically, it does seem that people do cling to simpler models rather than complicated ones – the enthusiasm generated by models like “invade iraq to bring peace” and “get out of iraq to bring peace” is all out of proportion to that of “let us negociate a rational compromise, involving this, this and this and even this, to bring peace.” On a personal note, my bosses would continually try and get the simpler model they understood adopted, rather than the more complicated ones that were really called for.

But it would be good to get this studied properly (could not easily find any studies online, don’t have access to most journals). Then we could put up procedures to avoid both types of bias, if warranted (restricting complex models to retrodictions, not predictions, and publicly announcing the failure of a simple model when it does fail – rather than the usual “the model needs to be refined”; maybe, but the model, as is, has failed).

But I think the most important is a splitting of the predictions of a model from its assumptions, as the second are prone to bias and the first are testable. I feel that this is more easy to do in complicated models.

A substantial amount of effort in statistics and machine learning goes into just figuring out how much freedom a model has, so that it can be penalized accordingly – for example, the “effective degrees of freedom” of regularized linear regressions.

Once you know the number of the effective degrees of freedom, there’s the question of how to penalize them – two popular approaches are the Akaike Information Criterion (which underestimates the penalty) and the Bayesian Information Criterion (which overestimates it). Some major approaches don’t count degrees of freedom at all. Vapnik-Chervonenkis dimension views the affair from the standpoint of the data, not the model, asking what is the most complex data that the model class can fit exactly – if a model involves a hundred variables, but can only fit a very limited class of potential outcomes, we should pay more attention to it than a hundred-variable model which can precisely fit a much broader class of outcomes. The more advanced versions of Minimum Description Length take account of the precisions of parameters and predictions, not just their number (but are often extremely hard to compute).

However, I think everyone pretty much agrees on that:

(1) All else being equal, having more effective freedom (not the same as the raw number of variables in the model) is bad for you, just as, all else being equal, having more data is good for you;
(2) There is a deep connection between the relative simplicity of a model, compared to the amount of training data it is fitted to, and that model’s ability to generalize to new data.

I’m sorry, I don’t quite get what you mean. I don’t think I was advocating explicit biases (just the fact that lay people shouldn’t feel they need to understand every assumption of a model). Can you clarify?

I’m entirely for it (as long as precautions are taken to test the models on enough results, vis Anders information point). It would take the ideology and biases out of the assumptions, and let the model live or die only on its results.

However, once you’re got a model, then humans will pick it apart, try and understand it, and build new models. You need that for science to progress, but new biases will then be introduced – to hopefully be eliminated later.

As for feasibility… no idea!

Very good point there. In the absence of easy testing, then simple models do have an advantage.

It is the complex models for simple predictions that really should be avoided.

That feels generally true. But what about, say, old-fashioned military ballistics? A complicated model, fed with many variables (temperature, pressure, wind speed, etc…) whose sole prediction is where a shell will land on a two dimensional plane.

(And in principle (as long as it’s tested against predictions, not retrodictions), there’s nothing wrong with a model that would take the information of the entire world stock exchanges, plus the murder rate in every Brazilian town, to accurately predict the price of fish tomorrow.)

A slight change of your information model would put the input information as “the number of free parameters in the system” and the output as “the amount of tested predictions”.

Then ballistics models are fine, as the amount of information output is basically infinite – just keep firing the gun, again and again.

Complicated economics models would be a problem though, as you might have to wait a long time to test enough predictions to have more information that your free parameters input. So complex models should have higher standards of proof than simple ones – requiring either complicated results, or lots and lots of simple results.

Economics would be a good field to try this with because of the difficulty of acquiring new empirical data (leading economists to rely heavily on retrodictions, e.g., of stock market data going back to the start of stock markets) and the strong tendency of humans to have prejudices about economics and to form factions of economic belief. Then again, economics would be a bad field to try this in because the economic realm is affected by a vast tangle of complex causal chains; e.g., the very complex realm of human psychology strongly affects the economic realm. Thus the cost (in researcher’s time) of creating a formal model of economics that includes most of the relevant knowledge that economists have about economics would be sky high.

Climate science is another area that necessarily relies heavily on retrodictions and is prone to factionalization, but is not nearly as complex as economics (but might still be too complex to codify as a single formal model).

I would appreciate people sharing their impressions on the desirability and feasibility of taking humans out of the model-picking and model-refinement loop in selected fields, e.g., climate science.

If a model is making truly accurate predictions, and they are informative and useful, then you shouldn’t care how many parameters it has. If someone can consistently predict tomorrow’s stock prices today, you don’t care how many parameters he is using. His model is good, by virtue of its predictive success in a difficult domain.