I was watching the 1966 classic film, A Man For All Seasons, about the Thomas Moore’s principled stance in opposition to Henry VIII’s grab for power. Ultimately, Moore is found guilty of treason, and in the final scene, after giving his executioner the customary tip for a clean blow, tells him, ‘don’t worry, you are sending me to God’. Considering the fear of death is one of the greatest anxieties for a conscious being, what a fabulous delusion! Clearly, biases can be helpful if looked at in a broader context, in this case, the belief in God is not evaluated based on evidence of God’s existence, but rather, the effect it has on the believer.
One particular area where the cognitive bias of overconfidence can be helpful is in entrepreneurship or investing. Keynes hypothesized in The General Theory that if "spontaneous optimism falters, leaving us to depend on nothing but a mathematical expectation, enterprise will fade and die". Indeed, if you look at the data on stock picking, or investing with professionals, it is a puzzle why people don’t merely all use passive indices because their average alpha is negative, and further, the average returns to highly undiversified C-corps or partnerships, seems woefully insufficient to justify their significant volatility But over-confidence of entrepreneurs is helpful because it offsets the underestimation in other benefits of investing or entrepreneurship. Basically, the totally rational investor won’t count what he can’t quantify, but much of the value in investing, or entrepreneurship, comes from an option value that is impossible to quantify. I know someone who built a product based on an optimization routine for superior investment strategy—the flagship idea failed, it never had a chance, but his system is now a popular risk management tool. I know someone else who created a system to consolidate earnings expectations in order to beat the market—it too failed, but his database of earning expectations was put together thoughtfully, and it became successful platform for disseminating this data. Actions affect one’s knowledge, reputation, and contacts in ways extremely difficult to quantify, because you can’t really define the probability state space. But the idea is simple, that people often get into some field to find gold, but then make their fortune selling shovels.
Investing for most people is irrational, in that regular retail investors and professionals don’t outperform the indices, trade too much and are too undiversified (transaction costs merely lower one’s returns, on average). In sum, their Sharpe ratios would be much higher if they bought index funds as John Bogle, Burton Malkiel, or Eugene Fama recommend, and presumably we all want higher Sharpe ratios. But think of the payoff to investing not merely the return from the investment, but rather, conditional upon success, access to a career in finance. One can (must?) parlay a successful call into a career as a broker, investment adviser, or some middle management position in finance (see this Bloomberg article touting the expertise of recent ‘winners’). Some initial filter, however imperfect, is needed by those who use financial professionals. An individual taking risk in this context suggests they had private cues indicating they were better than average, because risk-taking is costly, and though highly noisy, successful investments are better indicators than losers. Further, the act of investing can illuminate some parochial services where one does have an edge, and our specialized economy is based on a myriad of activities that are generally unknown until you get your hands dirty as a practitioner. Sharpe ratios ignore this bigger picture, and the fact that millions of people work in finance, and many are very well paid, suggest this option value is nontrivial.
Successful strategies are based a kluge of assumptions, and the key is you are evaluating an assumption not in its effect on one act, but rather as part of a meta-strategy, as a general disposition that affects many acts, and every act has ripple effects on one’s life. Instead of evaluating assumption X on Y by looking at X and Y, you instead look at the effect of holding belief X on the person believing X. I suspect every novice investor is somewhat overconfident, and too many people invest recklessly (‘gamblers’, Buffet derisively calls them). But there is a method to the madness, because the illogic of overconfident investing offsets the general underappreciation of the option-value of investing on one’s career, where you can provide a variety of auxiliary services in finance, and risk-taking is part of the application process.