Abstract: Complexity in the Paradox of Simplicity

Philip Fellman
University of Southern New Hampshire

I don’t work in settings where I choose between theory A and theory B, or between model A and model B. If I have 2 models I’m considering, I’d rather embed them continuously in a larger model that includes both the originals as special cases. (That’s called “continuous model expansion” as opposed to “discrete model averaging.”) There’s more discussion of this in a couple of the sections in chapter 6 of Bayesian Data Analysis.

Similarly, to Utilitarian: I’m not really ever estimating the probability of hypotheses. Rather, I use hypotheses to estimate parameters and make predictions. Then I use these predictions to evaluate the model.

In any event, Occam’s razor may not be essential for the original premise of the post. Suppose we think an alien planet is just as likely to be named Alpha as to be named Beta, and that those aliens are just as likely to have blue skin as blue skin. Suppose these events are probabilistically independent of playing ping pong, riding unicycles, and liking the number 7. Define the event A’ as follows: “Person A was abducted by aliens from the planet Alpha; they had green skin, they liked to play ping-pong, they rode around on unicycles, and their favorite number was 7.” By the assumptions stated above, P(A’) = P(B). But P(A’) <= P(A), because A’ is a subset of A. So P(B) <= P(A). (Most likely, the inequality will be strict.)

I agree that it can be a useful statement to say, for example, that 97% of the variance in the data is explained by only two factors, or to say that certain factors are lost in the noise, or to compare different datasets with regard to how many factors are needed to explain most of the interesting patterns. However, I don’t see this in terms of a prior distribution favoring simpler models; I see it as a choice of how to summarize a complicated model (for example, summarizing by the largest and most important factors). I agree completely with you that if a simple model does pretty well, this can be interesting, and a social scientist can look at how these factors change over time. I just completely reject the statement in the original blog entry above that the simpler model is “more probable.” I’m much happier working with the possibility of a really complicated model (which would be structured; I’m not talking about estimating 435 parameters with a flat prior) and then using “Occam’s razor,” if you’d like to call it that, to prepare useful summaries.

(BTW, I actually did this kind of analysis of the Swedish parliament,
http://www.eudoxa.se/politics/omrostningar.html
and could show that it was mostly 1-dimensional like British politics, unlike the 4D Norwegian politics
http://www.essex.ac.uk/ecpr/events/jointsessions/paperarchive/mannheim/w5/narud.pdf
and the 2D US politics
http://www.citebase.org/abstract?identifier=oai:arXiv.org:nlin/0505043
)

Kolmogorov complexity applies in social science too: an explanation with many ad hoc details is generally valued less than an explanation based on a few simple, general principles. One reason to value the simpler explanation is that it is less prone to overfitting, and can be expected to work well in the future too (unless it is too vacuous to actually tell us anything, of course).

A lot of people get seduced by complex “realistic” models in climate or neuroscience. The problem is that you seldom learn anything from them. They do things, but we cannot follow what happens or why, and the only way of predicting what will happen when parameters are changed is to run the model anew. That might be close to how reality works, but it does not help us understand what is going on or make robust predictions. Often reducing the model to equivalent but simpler models gives valuable insights.

In the case of the political votes, I think the simple models would do a pretty good job of predicting how representatives vote. That would be useful for a political analyst, and for the sociologist it would be interesting to examine what the components are (and how they change over time). That sounds like a good start for research to me. Instead starting with the most complex assumption (everybody votes for their own special reasons) doesn’t get us very far.

If it’s an esthetic principle, that’s ok. We just have different esthetic senses. You like Japanese rock gardens, I like overstuffed sofas. (Refer to Radford’s quote in my first comment above.) Based on the popularity of Occam’s Razor, I suspect that Radford and I have a minority preference. Nonetheless I think my preferences have worked well in the many applied problems that I’ve worked on.

I would argue that the appropriate esthetic principles can depend on what problems you’re working on. Many aspects of physics and genetics are inherently discrete (for example, just one or two genes determining something), whereas social and environmental phenomena are typically explained by many factors.