Overcoming Bias

Randomly Review Criminal Cases

This Slate article suggests that prosecutor Mike Nifong's unethical actions in the Duke Lacrosse case might not be that unusual among prosecutors.  The article argues that the only reason Nifong's unethical behavior was uncovered was because the Duke case generated an extraordinary amount of scrutiny.

I propose that we randomly select a few criminal cases for intensive review.  The review would give us an indication of how honest prosecutors are and would provide some deterrence against unethical prosecutorial conduct.

Over 90% of criminal cases settle through plea bargaining.  These cases never reach trial.  So, as the Slate article points out, any prosecutorial misconduct that might have occurred in these cases would almost certainly go undetected.  For my proposal to succeed plea bargained cases would have to be eligible for random intensive review.

Ideally, at least one person participating in the review would have subpoena power.  But if the government was unwilling to go along with my random review idea, private organizations could conduct it by themselves.  Each year, for example, law schools could randomly select, say, 30 criminal cases that were concluded in the past year.  These schools could then have their students investigate every aspect of the case to determine if justice was done.

White-collar crime and moral freeloading

Steven Levitt's book Freakonomics has a section where he tries to gain insights into white-collar crime via the experiences of a bagel-seller called Paul F:

early in the morning, [Paul] would deliver some bagels and a cash basket to a company's snack room; he would return before lunch to pick up the money and the leftovers. It was an honor-system commerce scheme, and it worked.

There was a lot of fascinating data coming out of his scheme, especially as to when and why people would cheat and not pay for their bagels; however, for those with an eye to bias, the most relevant observation was that:

the same people who routinely steal more than 10 percent of his bagels almost never stoop to stealing his money box.

More Lying

The subject of lying has come up here recently, with some tips on how to detect liars. But you'd expect that those whose job involves ferreting out liars, such as police officers or immigration judges, would be better at it than the rest of us. But this study claims that Swedish judges on the Migration Board (MB) are about as good at recognising the signs of lying as students:

Overall, the beliefs held by MB personnel were not more in tune with research findings on objective cues to deception than were the beliefs of the students. In addition, MB personnel often refrained from taking a stand concerning the relation between specific behaviours and deception; they exhibited substantial within-group disagreement [...]

In fact, we all seem to make the similar stereotypical mistakes when judging lies, implying that we believe our lie detecting abilities are much better than they actually are:

[...] there is a lack of overlap between the cues research has shown to be associated with deception (objective cues) and the cues people believe to be associated with deception (subjective cues) [...] Generally, these subjective cues to deception are indicators of nervousness. It seems as if people believe that a liar will feel nervous and act accordingly; however, far from all liars do [...]

But is there any group that is actually good at detecting lies? Indeed there is: of convicted criminals. Why? The most likely hypothesis seems to be that criminals have much more experience in deception, and, crucially, have feedback: when they're lied to, they generally discover it (to their cost) later.

So, unless you have a criminal record or a great experience in being lied to, the best is most definitely not to trust your gut.

Manipulating Jury Biases

The top google link for the phrase "Overcoming Bias", other than links associated with this forum, is to a book length pdf called "Overcoming Jury Bias."  This is a well written guide to lawyers on how to get a jury to agree with them.   It does show how to overcome jury bias when that would help you, but it also shows how to enhance jury bias when that would help you:

A woman who claims to be the victim of a sexual assault must appear in court as a "victim". If the woman appears before the jury wearing tight clothing, ostentatious jewelry, overdone cosmetics and a wild hair style, this will create an incongruence in the minds of the juror of this woman as a "victim". ...

The Butler Did It, of Course!

Here is a paper showing the potential practical utility of detecting and reducing biases: Confirmation bias in criminal investigations by O'Brien and Ellsworth. In an experiment subjects read a police file and were asked halfway through about their hypotheses of who the murderer was; practically everybody named the obvious suspect. On completing the entire file, where a second and stronger suspect emerges in the later half, they still tended to suspect the first guy. In a second experiment the subjects were asked to generate counter-hypotheses about why their suspect might be innocent, and this reduced the confirmation bias.

Another troubling source of bias is false confessions triggered by this confirmation bias and then strongly supporting the erroneous conclusion. The Psychology of Confessions by Kassin and Gudjonsson reviews this. During the preinterrogation review police, believing themselves to be better at detecting deception than they are, tend to confidently make false positive detections of deception in innocent people. Once they have convinced themselves they have caught a suspect a police interrogation then becomes guilt-presumptive and rather effective in generating false confessions, in particular in cognitively challenged people. And finally, juries and judges are easily convinced by the confessions.

Nothing of this may be total news to anybody on this blog, but it is still rather worrying how strong biases are accepted in police investigations and the legal system. Maybe the counter-hypothesis trick at least could be made part of police procedure: at certain points during investigations the investigators have to state possible disconfirming hypotheses for the record.

Statistical Discrimination is (Probably) Bad

A while back I wrote a post suggesting that it would be better if people didn't statistically discriminate (i.e., refused to use information on the average characterists of a group when making judgments about individuals from that group). The idea (not original to me) was that an individual from a group with bad average characteristics will lack an incentive to invest in improving since they won't be judged on their individual merits anyway. Various comments and discussions and trackbacks have generated a few further thoughts:

1. There is no guarantee that a refusal to discriminate will increase economic efficiency; for that to be true, it would have to be the case that the benefit of improved investment incentives outweighs the cost of discarding useful information.

2. The benefit of a refusal to discriminate increases if you place any weight on the normative proposition that everyone deserves to be judged on their own merits.

3. The benefit also increases if you believe that discrimination leads to alienation and various forms of costly anti-social behavior in the discriminated-against group.

4. Bryan Caplan suggests that statistical discrimination is at least mitigated, and possibly eliminated, by the fact that high-attribute individuals in groups with low average attributes have an incentive to "counter-signal" by taking some action to show that they are in fact high attribute. It is true that the possibility of counter-signalling will mitigate the harm from statistical discrimination, but I don't see how it can ever make it go away. Someone who bears both the direct cost of investment and the additional cost of counter-signalling will have less incentive to invest than someone who bears only the direct cost. Furthermore, counter-signalling may not be cheap; it's pretty darn costly to write a dissertation under an advisor known for high-tech mathematics just to show you don't suck at math if you didn't want to write with that guy anyway, you may just decide to punt and go to law school instead. Finally, the problem may accumulate over an individual's life as each investment not made makes the next one costlier until the point where an investment that would otherwise have been possible no longer is.

Two Cheers for Ignoring Plain Facts

Robin argued in an earlier post that we should use stereotypes to the extent that they contain useful information. The impulse to use all relevant information is generally a good one, but there are solid grounds for certain rule-based refusals to do so. For example:

1. The fact that jurors are supposed to assume that the guy in the dock committed the crime with probability 1/N, when everyone knows that it is really much higher than that, restrains cops who might otherwise exploit their power to get someone convicted on little evidence other than the fact that they were arrested.

2. A ban on statistical discrimination in employment increases the incentive for members of groups with high average levels of some bad attribute to invest in themselves, secure in the knowledge that discrimination will not prevent them from realizing the gains from those investments.

3. A ban on racial profiling, even though everybody knows that there are large cross-group differences in crime and terrorism, makes it harder for the authorities to harass people from unpopular groups just for kicks (and despite much progress on this front, there are no small number of cops who are inclined to do exactly that, and a policy in which it was possible to do so would attract many more of them). Perhaps more importantly, a color/race/religion blind policy, by its very nature, alters the experience of receiving scrutiny by the authorities. If 22 year-old Muslim men and 75 year-old Swedish women are equally likely to get searched at the airport, then the search is experienced as a nuisance. Everyone rolls their eyes, looks at their watches, and commiserates together about what a hassle it all is. When it's only the 22 year-old Muslim males getting searched, the experience is totally different and much less benign, both for the "bad" groups getting searched and for the "good" groups walking by them.

Law as No-Bias Theatre

Why is there law?   Some say for social justice, others for economic efficiency.  I suspect that "law is theater"; i.e., law is there to make disputants shut up.  When one person is mad as hell at another, law wants an outcome where neither they nor their friends yell and complain, and make the law look bad.  To achieve this, hard to understand legal processes make it hard to know what exactly to complain about, a long expensive process saps the energy needed to complain, and the option of endless appeals makes it unclear when to complain.

A complaining loser's best argument is often "it wasn't fair"; there was bias.  So law-as-theater predicts law-as-no-bias-theater; law will bend over backwards to avoid any possible appearance of bias.   How far does our law go in this direction?   Consider what law would be like with unbiased jurors.

If we expected jurors (or judges) to try their best to achieve social justice or economic efficiency, without substantial bias, we would have law, but not laws.  That is, there would lots of law, i.e., activity in a system for settling disputes, but few laws, i.e., rules about how to settle disputes.   The legal system would be simple:  you bring your complaint to a jury, you and your opponent each tell your side, and then the jury makes any decision they think appropriate.  Think King Solomon. 

Juries (or judges) would thus have complete control over the legal process.   They would talk to anyone about anything they liked, hardly limited by any rules of procedure or evidence.  Their final judgment would be any outcome they chose, based on any consideration they liked, hardly limited by any laws or precedent.  In fact, of course, law is nothing like this. 

Most of the legal biases that concern people are not due to juror interests.   After all, we can pretty much eliminate strong interest biases by just preventing jurors (or judges) from extorting disputants, or from sharing any substantial interests with disputants   Thus the structure of our legal system is driven in large part by fears of non-interest-based biases in juror beliefs.   

While we have built elaborate legal structures to deal with these supposed biases, legal scholars have spent almost no effort to document that such biases are real.  Legal rules constraining jurors are thus seen biases, justified as responding to unseen biases.   If the law were about social justice or economic efficiency, you'd think the legal system would study biases a lot more.   But if law is more about no-bias-theatre, needing only to make it hard for disputants to complain of bias, what would be the point?