Overcoming Bias

Heroic Job Bias

Friday Eliezer described a Superhero bias:

The police officer who puts their life on the line with no superpowers, no X-Ray vision, no super-strength, no ability to fly, and above all no invulnerability to bullets, reveals far greater virtue than Superman.

Superman risks less, yet we celebrate him more.  Eliezer followed with:

John Perry was a New York City police officer ... the fact that an atheist and a transhumanist can still be a police officer, can still run into the lobby of a burning building, says more about the human spirit than all the martyrs who ever hoped of heaven.

Actually the hero bias is far worse than Eliezer imagines, because in fact police and firefighters do not risk their lives more for you!  Yes, these workers do die more often on the job, but they die much less often because of the job.  You see, these are relatively high status jobs, and having low status is quite deadly for humans.  So in fact people risk their life more by becoming waiters, bus drivers, truck drivers, plumbers, painters and lots of other jobs!  A 1999 Demography article shows how male mortality risk varies with job category:

Publication Bias and the Death Penalty

The front page of Sunday's New York Times contained an interesting article reviewing research linking  the death penalty to homicide trends.  Adam Liptak attempts to provide a balanced account of the debate, noting first one set of findings:

According to roughly a dozen recent studies, executions save lives. For each inmate put to death, the studies say, 3 to 18 murders are prevented.

And then my own research:

The death penalty “is applied so rarely that the number of homicides it can plausibly have caused or deterred cannot reliably be disentangled from the large year-to-year changes in the homicide rate caused by other factors,” John J. Donohue III, a law professor at Yale with a doctorate in economics, and Justin Wolfers, an economist at the University of Pennsylvania, wrote in the Stanford Law Review in 2005. “The existing evidence for deterrence,” they concluded, “is surprisingly fragile.”

Surely a dozen studies is itself evidence of robustness.  Why then is then is it that we find these results are fragile?  Two words: Publication bias (also known as the file drawer problem).  Our research revealed that alternative approaches to testing the execution-homicide link can yield a huge array of possible results (positive and negative).  But if only strong pro-deterrent results are reported (and the others remain in the file drawer), this could look misleadingly like there is a pro-deterrent consensus.

It turns out that there are some rather simple tests for publication bias.  Our friends in medicine provide a useful intuition.  Imagine that there are many separate drug trials being considered - some with large samples, some with small samples.  If all results are being reported, then smaller samples should, on average, yield similar estimates to larger samples, albeit with a bit more noise (in both directions).  So the standard error of an estimate should be uncorrelated with the coefficient.  But if researchers only report statistically significant estimates, then they will only report results with t-statistics>2, yielding a strong correlation between standard errors and coefficient estimates.

You can probably guess what we find.

Seat Belts Work

Yesterday, James Annan pointed us to a literature (see here and here) suggesting seat belt (or air bag) use does not actually saves lives.   Yes, all else equal those wearing seat belts are more likely to survive accidents, but those who know they are better protected also compensate by driving less safely.  The net effect is theoretically ambiguous, and the empirical data was long not clear.   

Looking at Wikipedia seems to support skeptics, as skeptic arguments cited there are much stronger than the pro arguments, which are mainly government sources that don't seem to address skeptical points.  And since I'm pretty skeptical about the effect of medicine on health, it seemed natural for me to embrace this skeptical argument. 

But I dug deeper and found this 2003 Review of Economic Studies article (ungated here):

Using a unique panel data set on seat belt usage in all U.S. jurisdictions, we analyze how such laws, by influencing seat belt use, affect the incidence of traffic fatalities. ... We find that such usage decreases overall traffic fatalities. The magnitude of this effect, however, is significantly smaller than the estimate used by the National Highway Traffic Safety Administration. In addition, we do not find significant support for the compensating-behavior theory, which suggests that seat belt use also has an indirect adverse effect on fatalities by encouraging careless driving.  ...

Our estimate of the potential savings in lives from increased seat belt usage are less than half of the estimate used by the National Highway Traffic Safety Administration (NHTSA). ... Our estimates indicate that the national usage rate would increase from 68% to 77%, and 500-1200 lives would be saved annually, if all states now having secondary enforcement moved to primary enforcement.

Of course I'm always worried about data mining and publication biases toward getting the results authors expect to see.  But I don't see any obvious indications here, and the results seem robust to several variations.  So I'll tentatively conclude seat belts do save lives.  Of course this is not to say this gain is worth the cost in terms of whatever it is that makes drivers reluctant to wear seat belts.   

What Wisdom Tradition?

Russ Roberts wrote: 

I happen to believe that concealed handguns do deter crime and allowing concealed handguns is a good thing. ... But what's clear to me is that my belief in the virtues of allowing concealed hand guns has little to do with the empirical evidence. And I would argue that the opponents are really in the same boat.

I then asked:

If Russ relies little on data to draw his conclusions, then on what does he rely?  ... Can't we say the same thing about theory, that we mainly just search for theory arguments to support preconceived conclusions?  ... Either you should believe that truth-correlated data or theory has substantially influenced your belief, or you should retain only a very weak belief. 

Russ has responded:

When scholars can run hundreds of multivariate regressions at very low cost, it easy to convince yourself that the results that confirm your prior beliefs are the "right" results.  The Pragmatists ... believed that the rationalism of Descartes had a dangerous element of hubris. ... Your grandmother is right. She believes in certain things. When you ask her to justify her beliefs she shrugs and says she can't. ... Norms of behavior that survive, survive because they're effective even when no one understands why.

If Not Data, What?

My colleague Russ Roberts writes: 

Over the years, I have become increasingly skeptical of the power of statistical techniques to measure causation in complex systems.  ... I happen to believe that concealed handguns do deter crime and allowing concealed handguns is a good thing. And you can claim that the evidence that shows I'm right is "good" statistical analysis. The other side disagrees. They claim it's "bad" statistical analysis. Who's right? I have no idea. But what's clear to me is that my belief in the virtues of allowing concealed hand guns has little to do with the empirical evidence. And I would argue that the opponents are really in the same boat. They just don't like guns and they've dressed up their prejudices in fancy statistical analysis. ...

If Russ relies little on data to draw his conclusions, then on what does he rely?  Perhaps he relies on theoretical arguments.  But can't we say the same thing about theory, that we mainly just search for theory arguments to support preconceived conclusions?  If so, what is left, if we rely on neither data nor theory? 

Try saying this out loud: "Neither the data nor theory I've come across much explain why I believe this conclusion, relative to my random whim, inherited personality, and early culture and indoctrination, and I have no good reasons to think these are much correlated with truth."  That does not seem a conclusion worth retaining.  If this is really your situation, you should move to a nearly intermediate position of uncertainty.   Either you should believe that truth-correlated data or theory has substantially influenced your belief, or you should retain only a very weak belief. 

HT to Arnold Kling.

What Evidence Reluctant Authors?

Steven Landsburg is a great economics writer, with a monthly Slate column and many popular economics books.  But he is also a sharp theorist, and his October 2007 Journal of Public Economic Theory paper is in fact the best theory paper I've read in several years, as it helps to resolve fundamental issues in moral philosophy. 

Now unfortunately Landsburg's paper is too philosophical to get the attention it deserves from economists, and too mathematical to get the attention it deserves from philosophers.  But one might hope that at least he would be calling as much attention as possible to his paper. Alas, it seems not.  I doubt he will mention it in his column or popular books.  And though he is speaking at our department in a few weeks, he has so far rejected my suggestion to talk on this paper; he'd rather talk about his third paper so far on quantum game theory. 

So I am left to wonder: does he know something I do not about the value of this paper?  In any case, here is the idea:

Confident Proposer Bias

My recent exchange with Tyler highlighted a common bias:  people presume that policy proposal authors claim high success probabilities. 

Because I had written sympathetically about futarchy (public policy decision markets) and private law, Tyler presumed that I assigned high probabilities to the success of such alternatives.  I countered: 

I think Tyler and I both agree that private law and decision markets are among the twenty most interesting big potential policy ideas in the last few decades, that such big ideas have little chance of being adopted wholesale and even then would work out very differently than anticipated, but that such ideas are nevertheless well worth exploring and trying out first on small scales. 

A novel approach to policy deserves more attention, including sympathetic discussions, when there is a positive expected payoff from further explorations of it.   Such explorations can include math models, lab experiments, and field experiments.  A positive payoff comes if such explorations can refine the approach into useful fielded implementations, while a negative payoff comes from wasted effort and harmful implementations.  If the cost of experimenting is low and the final positive payoff could be very high, a novel policy approach can be worth exploring even with a very low probability of success.

Colorful Characters

Novelists often use characters to voice controversial views they are reluctant to say publicly themselves.  In Tyler Cowen's Discover Your Inner Economist, out this week, I seem to be similarly featured as a colorful character who can voice views Tyler is reluctant to embrace directly in a popular book:

Nowhere is signaling more important that in the family.  Whereas direct cash incentives work only so well in families, signals are crucial in building family trust and cooperation.  ...  If I do not want to buy a warranty, my wife considers it irresponsible.  ...

My colleague, economist Robin Hanson, argues that many protective activities are really about "showing that you care."  What would we think of the parent who did not do "everything possible" to protect his or her children?  What kind of man would tell his wife that he weighted costs and benefits before spending on an additional biopsy for her cancer?  ...

The more our families feel we are violating our commitments to care, the harder the reception that economic advice will receive.  ... Think of "the economics of the family" as The Truth That Dare Not Speak Its Name.  If you are the economically informed member of your family, or perhaps even an economist, don't flaunt it.  ...

And anyone who studies signaling behavior for long enough will be repulsed by social hypocrisy and will be tempted to become some kind of intellectual radical, maybe a revolutionary, maybe a more peaceful eccentric, and this has happened to Robin.  When it comes to showing that he cares, Robin wonders why it isn't enough that he cares.

Actually, I'm pretty sure Tyler wonders all these things as well.  Now I'm a big Tyler fan, and I've been painted as a colorful character before, for example at Fortune and The Register, and it does beat obscurity. 

But I will grump about a pattern I've noticed: people don't seem very interested in getting the details right about colorful characters; they'd rather exaggerate.  People don't mind saying Fred wore a purple suit to the meeting, even if it was really closer to a light pink violet, if the point was just that Fred's suit was weird. 

I don't just have strange opinions just to be strange, however; the details matter to me.  So I feel the need to make more (long) corrections.  I doubt Tyler will consider these will be news; for his purposes these details weren't worth walking down the hall to clarify.

Added: See Tyler on Robin on Tyler on Robin.

Functional Is Not Optimal

If you look at what a random animal is doing at a random time, you will usually find a complex behavior, with many correlated parts.  Similarly if you look at a random physical part of an animal, you will find a complex structure, with many correlated parts.  The best way to make sense of such structures is usually to look for the functions they might perform, i.e., the way that they might contribute to survival and reproduction, now or in the past, of the animal or its parasites.  Sure, some structures are "spandrel" side effects of other functional structures, but (at the aggregate level) most are not. 

This, however, does not at all imply that most such structures are anything close to optimal, giving the best possible outcomes.  Consider that when we transplant plants or animals to a new continent, they often handily out-compete existing species.  This could not happen if the existing species were close to optimal.  Often their structures worked in the past, but have not adapted enough to changing context. 

It is similar for most human organizations.  A good way to understand what a random organization is doing at a random time, or to understand a random part of its physical or communication structure, is to ask how such structures once functioned to help the organization or its parasites to survive or grow.  But this hardly means that every organization is optimal; many are clearly dinosaurs losing to new rivals. 

A lot of human behavior does not immediately make sense in terms of survival and reproduction.  We sing, dance, joke, worship, collect, chatter, decorate, party, dribble, travel, argue, blog, and much more.  With a little thought, we can come up with functional explanations, such as signaling theories, for most of this behavior.  Many people, however, reject such explanations, saying functional explanations would imply human behavior is optimal, which it is clearly not.  Thus, they argue, odd human behavior must instead be due to random mistakes, spandrels, or our changing environment. 

This "head in sand" attitude is so contrary to what we know about animals and organizations that I have to conclude that not only is it seriously wrong, but that humans seem built to self-deceive about the functions of their behavior.  While our behavior may be far from optimal, there is surely a detailed correspondence between our behavior and the functions they perform.  But we prefer to be innocently unaware of the signals our behavior functions to send and receive.

From a lunch conversation with Mike Makowsky, Bryan Caplan, and Alex Tabarrok.

When Differences Make A Difference

In the May 18 Science, Philip Tetlock reviews Scott Page's book The Difference, and basically nails it:

Casual readers could easily conclude that Page .. has clinched the argument for the egalitarians.  Indeed, Page arguably invites the interpretation that there may be no awkward efficiency-equality tradoffs when he repeatedly declares that "diversity trumps ability."  ... In brief, diversity appears to trump ability - at least when we equate high ability with drawing lucky starting points in sharply constrained searches for solutions.  But elitists will argue that the game was rigged. 

Scott's book touts the virtues of having a big toolbox.  The more tools you can try on your problem, the better your chances of solving it.  And if the individuals who are consistently best at solving some class of problems tend to have a similar tool sets, then it can be wise to also include some different "less able" people on your team, to gain access to their different tools.  This can justify promoting organizational "diversity," at least when the dimensions of diversity being considered actually correlate with differing tool sets.  Of course it is noteworthy that many people promoting ethnic, racial, or gender diversity oppose including ideological diversity. 

Scott Page was a new assistant professor at Caltech in '93 when I was a new grad student.  He has done very well for himself since then, and recently he asked me to clarify the relation between my perspective on prediction markets and the views he describes in his book.   That is easy, as there is almost no relation. 

The point of prediction markets is to give participants incentives to shut up when they do not know, and to speak up when they have something useful to add.   Such incentives can help regardless of whether any particular kind of diversity helps or hurts in any one context - the participants will have an incentive to find and include whatever works.  Prediction markets are not about the wisdom of crowds any more than they are about the wisdom of credentialed experts - they are about the wisdom of whatever works.