As H.L. Mencken wisely observed, every complex problem has an answer that is simple, plausible, and wrong. We can pretty much count on ordinary minds not only to think of these solutions, but to offer them. So what is the genius to do but “specialize” in solutions that others do not think of? That’s what makes genius worth having. Yes, the need to specialize in things other don’t think of is a bias – no matter how smart you are, if you have a hammer, things tend to look like nails – but I wouldn’t attribute it to something as small-minded as showing off, as it is serves a wholly rational purpose of using rare resources for rare purposes.

I liken it to writing and to quote Pascal … “My Reverend Fathers, my letters have not usually followed so closely, nor been so long. The small amount of time that I have is the cause of both. I would not have made this so long except that I do not have the leisure to make it shorter.”

Perhaps the reason for complexity is lack of time to make it simpler?

What you say is true if indexes were a collection of stocks picked by really smart people who evaluate what businesses have wonderful prospects etc.

In reality all they look to do is follow a simple algorithm that needs no smarts.

You just start off with the CAPM style reasoning which leads you to conclude that the optimal portfolio to hold (from the point of view of minimum variance ‘risk’ for a particular expected return), is simply buying the ‘market’. This in other words means that you should put some fraction of your money in all stocks and other risky assets in the world, with the weightage being proportional to the market capitalization of the stock or financial instrument and the remaining money in a “risk-free” instrument like US treasury bills.

All index funds do is to approximate at least a country’s market, by picking a sufficiently large number of stocks with high market cap. And thereby avoiding the hassle for the individual investor to actually go and buy 500-10000 odd stocks in a market cap weighted fashion.

So for all practical purposes, the index is fairly “zero knowledge”, since the algorithm is quite precise and public, which is “create a large basket of stocks that approximates the whole market to the extent possible”.

In the case of the S&P 500, the stocks in the index are chosen by a committee of humans, using their own judgment. (They attempt to select stocks that form a representative sample of the economy of the United States as a whole.) If index funds that follow the S&P 500 beat actively managed funds, then wouldn’t that make the members of that committee abnormally good at stock picking?

Interestingly, because index funds exist, whether or not a stock is listed in an index affects the price of the stock. If a stock is taken off the S&P 500, all the index funds that follow the S&P 500 have to sell it, depressing the price of the stock. Similarly, a stock that gets added to an index experiences a surge of demand, causing its price to rise.

Important questions don’t have answers. They have increasingly accurate and complex approximations. Academics are more willing to consider more complex approximations. That doesn’t mean they have a lower opinion of the simple answers.

(Though it wouldn’t surprise me if they did.)

Finding subtle differences is fun because of the pleasure of grokking. I always get a kick out of understanding something on a deeper level than it’s being explained to me.

As for stocks: People also have come up with complicated strategies which are supposed to help them make money while gambling. If there is the potential for easy money with very little work, people will overanalyze.

The show-off bias is what comes in at the end when you share your discoveries, even though they are useless.

See this Ted Talk by Murray Gell-Man about how he looks for “simplicity” as an indication of truth:

http://www.youtube.com/watch?v=UuRxRGR3VpM

Therefore, smart people should only be given complex problems. Let those with less intelligence (or at least more humility) handle the simple ones, because they will perform more accurately (i.e. with less bias).

Therefore, smart people should have assistants. Or, to quote Don Lapre, “never do anything yourself when someone else is ready, willing, and able to do it for eight bucks an hour”. (Actually, in my experience, you can’t expect competence for less than twelve.)