Noise in the Courtroom!

The acquitted often walk out court with a huge smile on their face, convinced that any stain on their character has been erased.

But more guilty people get tried and acquitted than the average of the population. So barring a dramatic Perry-Mason-like revelation, the trial is evidence of guilt – noisy evidence, but evidence none the less. It isn’t legal or scientific evidence, but it is evidence that a Bayesian should use.

But while employers can often access criminal records of convictions, they are generally barred from finding out about acquittals (especially if the accused take steps to have their arrest expunged); and the potential employee is often allowed to lie if asked directly. This noisy measure is deemed officially unavailable.

Should it be available? And in what way is this noisy measure different from those used in education and in medicine?

GD Star Rating
loading...
Tagged as: , ,
Trackback URL:
  • sa

    That’s a clever observation but only if you assume that guilty are likely to be brought to trial than the average population.

  • Joshua Fox

    This noisy measure is deemed officially unavailable.

    A democratic society puts limits on the tremendous power of the state against the individual, even at the cost of abandoning this apparent Bayesian evidence. In fact, the power of the state all too easily results in bias (systematic deviation from the truth) which may become so great, if unchecked, as to dominate the noisy evidence mentioned.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    I think of it like this: A reputation is a social construct distinct from the purely Bayesian expectation. As game theory shows, the power of reputation is not just that we track other people’s reputations, but that other people act differently depending on the effect they think it will have on their reputations.

    Let’s say a study shows that people with blue eyes are more likely to commit crimes, as are people who wear blue earrings. I may take the blue earrings into account, but deliberately refuse to take the blue eyes into account, in choosing to interact with someone – because I want to maintain the incentive structure. Blue earrings you can wear or remove, according to your choice, so that making your reputation dependent on blue earrings does not interfere with the basic mechanism of using reputational incentives to reward or punish choices. But people do not choose their eye color, so to treat that as part of a person’s reputation is unjust. Note that this is a decision, not an estimate.

    I can’t control what people accuse me of; I can control whether I’m guilty or innocent. Thus an acquitted charge should not show up on my reputation. And since people are very bad at discounting illegal evidence, it is wiser to have process controls for keeping out illegal evidence. (A perfect Bayesian would look, but a perfect Bayesian could keep a separate mental book for reputation.)

    Imagine a Tit-for-Tat world where blue agents are somewhat more likely to be of Always D type than Tit-for-Tat type, compared to green agents; if you start defecting against agents just because they’re blue, then even the TfT blue agents may have no motive to cooperate with you. Reputation is a decision to create social incentives, not just an estimate.

  • Stuart Armstrong

    A very good analysis Eliezer (close to my own feelings, but much better expressed). But I don’t feel that what goes into reputation is quite so clear cut. What of a (relatively) stupid doctor? He can’t change his intelligence (yet), so his stupidity should not show up in his reputation. So would you object to, for instance, an IQ ranking of doctors?

  • Stuart Armstrong

    That’s a clever observation but only if you assume that guilty are likely to be brought to trial than the average population.

    I think that’s a safe assumption. A small amount of guilty are likely to be brought to trial; add to that other people chosen at random, and you still have the guilty more likely to be brought to trial than the average (most of Stalin’s victims were innocent, but there were still more guilty among his victims that among the rest of the population).

    In fact, the power of the state all too easily results in bias (systematic deviation from the truth) which may become so great, if unchecked, as to dominate the noisy evidence mentioned.
    A good point. But others can ruin a reputation just a easily as the state, and cause even greater bias (journalists, for instance). Should they be similarly checked?

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    What of a (relatively) stupid doctor? He can’t change his intelligence (yet), so his stupidity should not show up in his reputation. So would you object to, for instance, an IQ ranking of doctors?

    An interestingly borderline question. I think that while we do not choose our IQs, to pretend that it doesn’t matter in real life is simply insane. We don’t choose the moral character that makes our choices, either, though it can modify itself over time. Also, an IQ test is behavior.

    But I, personally, would rather know the result of a test of medical knowledge and skill, than a doctor’s IQ test. Fluid intelligence doesn’t become skill until it has crystallized. That would probably be my answer. Of course, if you come back with evidence showing that IQ is a much better predictor of medical outcomes than any test of medical knowledge, I’d have to go with the IQ test, then.

  • michael vassar

    I would want to know exactly how the test was given, but without any specific knowledge of the test I’d go with the IQ test rather than one claiming to be “of medical knowledge and skill”. I bet that a test of medical knowledge that did a better job than an IQ test at predicting outcomes *could* be designed by anyone who could produce a good new IQ test (taking an IQ test and adding some science questions, practice diagnostics, and tests of rationality like those in the cognitive reflection test or in Judgment Under Uncertainty would very likely be enough to do substantially better), but incentives for having it adopted in the medical system would be weak at best. I don’t see strong optimization pressures pushing for, say, MCATs that predict medical outcomes.

    As a practical matter, it seems to me that pretending IQ doesn’t matter in real life is considered morally obligatory by a substantial fraction of the population, with fairly disastrous effects. I suspect that the problem is that people tend to attach their estimates of a person’s intelligence with their estimates of the person’s basic moral worth. As a result, acknowledging that some people are less intelligent than others is seen as dehumanizing. Of course, not acknowledging that a person with an IQ of 90 may have difficulty with certain academic tasks and punishing such people for their poor performance isn’t exactly the nicest solution either. It would probably be better to try to recognize more types of achievement and not to couple essentially all non-parental adult approval between ages 6 and 10 to academics. I have certainly noticed that academic abilities seem to correlate strongly with pro-social tendencies in the US and not nearly so strongly, if at all, in developing countries that I have lived in, which suggests to me that the observed correlation is due to some problem with US culture, possibly the problem I have suggested.

  • http://entitledtoanopinion.wordpress.com/ TGGP

    What a coincidence, I just had a post about IQ this morning.

  • http://entitledtoanopinion.wordpress.com/ TGGP

    I figure that these days there is so much information out there and the costs of obtaining it continually decrease by so much, it might be difficult to keep that sort of thing from people that want it.

  • Ak Mike

    Actually, Stuart’s factual predicate is mistaken. Court records of criminal cases involving adults, irrespective of whether they result in a conviction or an acquittal, are nearly always easily-available public records. They are indexed by the name of the accused. They are frequently available on line.

  • Paul Gowder

    I’m not sure why you are so confident about your underlying base rates (i.e. guilty people being tried and acquitted). It’s a piece of conventional wisdom, but is there actual data on this? (And where would one get actual data on this, i.e. a reliable way of telling guilt apart from being convicted?)

    Moreover, perhaps there are legitimate social reasons for being biased in favor of the acquitted. For example, if even a good proportion of the acquitted are innocent, we might decide that it’s better to protect them from the consequences of noisy signals even at the cost to employers of inadvertently hiring some criminals. We do this all the time in related situations. Consider racial profiling. Even though there are statistics suggesting that, e.g., people of X race are more likely to be selling drugs than the people of Y race, we judge that the cost to the innocent people of being profiled is higher than the cost to society as a whole from having to go without.

  • Stuart Armstrong

    Actually, Stuart’s factual predicate is mistaken. Court records of criminal cases involving adults, irrespective of whether they result in a conviction or an acquittal, are nearly always easily-available public records.

    My understanding from what I read online was that these could be expunged (made unavailable) if the acquited made the relvant steps. But even so, it doesn’t undermine the central idea: employers cannot ask about acquitals, cannot expect a true answer if they do ask, and generally cannot use an acquital as grounds not to hire or to fire someone.

    I’m not sure why you are so confident about your underlying base rates (i.e. guilty people being tried and acquitted). It’s a piece of conventional wisdom, but is there actual data on this?

    No data that I know of, but some math may help to analyse the problem. Let G be the fact of being guilty. Let A be the fact of being tried and aquitted for a crime. Then Bayes theorem gives:

    P(G|A) = P(G and A)/P(A) = P(A|G)/P(A) * P(G).

    The ratio P(A|G)/P(A) is the critical one – it says whether you are more likely to be guilty, given a trial and an acquittal, than the general population. So is P(A|G) greater than P(A) or not?

    Masses of gratuitous prosecutions will increase P(A). Masses of speculative prosecutions (where the police feel that someone is guilty, without a reliable chance of conviction, and prosecute him anyway) will increase P(A|G). However, since P(A) starts off very low (few people get prosecuted, let alone acquitted, in a criminal case), even ten times more gratituous prosecutions than speculative ones will probably still leave P(A|G) higher than P(A), as long as the police are at least somewhat more reliable than random guesswork. I also feel that there are few gratitous prosecutions leading to acquittals in western countries (i.e. generally, if the prosecution is gratuitous, they will have faked the evidence enough to ensure a conviction; they don’t often prosecute people at random just for the fun of prosecuting).

    So in criminal cases, I feel that P(G|A) is higher than P(G). In civil cases, where P(A) is not so low and the number of gratuitous prosecutions is huge, I don’t feel I can conclude anything.

    Moreover, perhaps there are legitimate social reasons for being biased in favor of the acquitted.
    There are, very strong ones with deep historical roots (see for instance The Trial by Sadakat Kadri). I’m interested in teasing out what differentiates different types of noisy measures, and why some are acceptable to people and others are not.

  • Stuart Armstrong

    An interestingly borderline question. I think that while we do not choose our IQs, to pretend that it doesn’t matter in real life is simply insane. We don’t choose the moral character that makes our choices, either, though it can modify itself over time.

    Maybe the crux of the matter is that we want better doctors, and less criminals. The incentives are different: a dumb doctor who leaves the profession because of an unfair IQ test is not as much of a loss as someone with blue eyes (or the wrong colour of skin) who becomes a criminal because of an unfair reputation.

  • http://entitledtoanopinion.wordpress.com/ TGGP

    Thanks for the book recommendation, Stuart.