Lazy Lineup Study

Thursday’s Nature suggests standard police line-ups may not be so bad:

The traditional US procedure is familiar to any fan of television cop shows. Witnesses are presented with a line-up that includes both the suspect and a number of innocent people, or foils’, and are asked to identify the perpetrator. In the early 1990s … then attorney-general, Janet Reno, invited experts to form a working group to address how this method could be improved. … The working group’s most important recommendation was that line-ups should be conducted in a double-blind fashion, so that neither the witness nor the official overseeing the procedure would know who the suspect was. The group also recommended that the suspect and foils be presented sequentially rather than simultaneously, and that the witness be asked to make a decision after each one rather than waiting until the end. …

In 2003, the Illinois State Police commissioned its own study to test line-ups under real-world, field conditions …, with the cooperation of two psychologists and three of the state’s police departments … [they] spent a year conducting some 700 eyewitness identifications. Some of the procedures were non-blind and simultaneous; the rest were double-blind and sequential. Both conditions were a mix of live line-ups and photo arrays. The team found that the double-blind, sequential technique produced higher rates of foil picks – that is, clear errors – and lower rates of suspect picks than the traditional, nonblind line-up. …

Horrified, psychologists pointed to deep methodological flaws in the study, which was never submitted to peer review. According to the psychologists, the decision to change two variables at the same time blind/non-blind and simultaneous/sequential made it impossible … to draw any conclusions from their data. … Nobel laureate Daniel Kahneman [and others said] … the design flaw "has devastating consequences for assessing the real-world implications of this particular study". …

Criminal Defense Lawyers … sued the police departments that took part in the pilot programme to release their data so that they could have them reanalysed. So far the departments have refused, and Mecklenburg … says it has only alienated the police, making it less likely that they will cooperate with field studies in future. …

A study published last year … applied strict double-blind sequential rules to British-style video parades. This was a lab study, but with what psychologists call high ecological validity … most striking finding was that the number of correct identifications dropped from 65% under British rules, to 36% under the strict rules.  … The implication is that the proposed reforms might reduce the wrongful conviction rate … [but] more criminals might be let off the hook.

Psychologists’ complaints here seem like overblown turf-protection to me.  Sure it might be nice to distinguish two effects, but with limited data there is nothing wrong with testing a package of two recommendations together.  We should be open to whatever the data show, and standard procedures may well win out.  But why oh why has it taken so long to actually study all this?   

Added:  On reflection, I should consider it similarly likely that the reporter just messed up here, and that psychologists objected mostly to the outcome metric, and that changing two variables at a time was more a minor complaint. 

GD Star Rating
Tagged as:
Trackback URL:
  • There is an inherent bias, nay flaw, in the quoted text. Number of positive IDs is not meaningful. The double blind- aspect should be studied so that the police officer would have a suspect that is innocent and see what happens to the rate of these false positives. If positive IDs fell from 65 to 36, we have absolutely no (well, we may have some, but not here) way of knowing how many of those 31 percent were, in fact, innocent or at least not the ones that should have been IDd in the first place. There seems to be an implicit assumption here, that they were all guilty. What is worse the underlying idea seems to be that the police never apprehend innocent people.

  • I disagree – where there is one methodological flaw, there are others.

    It seems these studies wouldn’t be *that* expensive to carry out, so I don’t understand why we’re arguing about the value of a flawed study rather than discussing the results of some more recent, more careful experiment.

  • eddie

    Agreed with Tiedemies.

    The purpose of a line-up should not be to determine whether the witness can identify the suspect, and thus the measure of quality of a line-up procedure (blind or not, sequential or not) should not be how frequently the victim picks the suspect rather than the foils.

    The purpose of a line-up should be to determine whether or not the suspect is the person that the witness saw. Thus, the quality measure should be how frequently the victim picks the suspect using a given protocol relative to how frequently they pick any foil in the same protocol with no suspect included in the line-up. In other words, which protocol provides the strongest evidence that the suspect is the witnessed person given a positive identification?

    In truth, of course, the actual purpose of a line-up is to allow the police and prosecutors to obtain evidence that can persuade a jury that the chosen suspect is guilty. Police and prosecutors have strong incentives to successfully prosecute, but little incentive to correctly prosecute. By that metric, given the studies above the “standard” line-up is clearly to be preferred (by police and prosecutors) since it has a greater rate of producing evidence that appears convincing to a jury.

    The same problem occurs with the use of the “Reid Technique”. The three parts of the technique are 1) “factual analysis” in which the cop identifies suspects, 2) “behavioral analysis interview” in which the cop interviews suspects and uses techniques to purportedly identify whether or not they are guilty but which almost certainly merely serve to confirm the cop’s pre-existing determination that they probably are, and 3) “interrogation” in which the cop, having determined that the suspect is guilty, uses techniques to persuade them to confess. The fact that these techniques can easily lead to obtaining confessions from innocent parties is not merely irrelevant to police and prosecutors (since their goal is to successfully prosecute, not correctly prosecute), but is in fact outright denied by the originator of the technique and its supporters. “Why would an innocent person confess?” is the refrain, despite the fact that the interrogation technique is explicitly designed to persuade someone to confess no matter how strongly they desire not to and no matter what their reason for not wanting to confess is.

    The fact that blind, sequential line-ups produce a far lower hit rate than standard line-ups should send chills down everyone’s spine. It implies that standard line-ups are producing a lot of false positives (where “false positive” doesn’t mean “the witness picked a foil” but rather “we concluded the suspect was guilty because the witness picked the suspect”).

    The implication is that the proposed reforms might reduce the wrongful conviction rate … [but] more criminals might be let off the hook.

    The same would be true if our standard procedure was simply to convict 100% of suspects and our proposed reform was to instead put evidence before a jury.

  • Indeed, I’m with Tiedemies; if you use a non-blind method, and the person conducting the line-up knows who the guilty party is, then their effect in tending to lead the witness toward selecting that person will of course make it more likely that the guilty party will be selected. But then the line-up isn’t providing meaningful additional evidence.

    The basic problem is that eyewitnesses are very unreliable, so the only way to get them to produce a lot of the desired identifications is to fudge the procedure. But surely that isn’t a reason to fudge the procedure; rather, it’s a reason to look for better forms of evidence.

  • eddie

    I agree with Robin’s comments about the psychologists’ objections to the study. The study isn’t flawed. Furthermore, the study doesn’t show that blinded, sequential line-ups are inferior to standard line-ups (as the psychologists’ reactions seem to say). Instead, the study shows that standard line-ups are seriously flawed because they produce very different results from blinded, sequential line-ups – which should be the gold standard for producing evidence that the suspect is in fact the witnessed person.

  • Doug S.

    What about blinded, simultaneous line-ups?

  • Michael

    A good way to administer a multiple choice exam is to penalize for a wrong answer and give a small reward for no answer (and of course the largest reward for a correct answer)–we most of all wish to discourage guessing, and hence false positives (false negatives are of course a good thing in police line-ups as they show immediately that the witness is not reliable). Must these witness pick a “suspect” or can they choose not to answer? If not, I’d also be curious as to which method produces more non-answers.

  • Michael

    By “false positive” I meant picking the suspect even if that person is the not the person the witness saw, and by “false positive” I meant picking a foil.

  • Michael

    Sheesh! Replace that second “false positive” with “false negative.” I’m clearly in serious need of another coffee.

  • I, too, second the first commenter, and I also second Robin’s view that it’s a shame that this kind of thing is not better researched. It would seem to be rather important, no?

    Of course, you shouldn’t judge reality based on fictional crime TV programmes, but I always wondered whether it would not be better to have in a lineup, say, seven tall men with short dark hair and glasses rather than seven men that look wildly different.

    Does anyone know whether things actually work this way?

  • Dynamically Linked

    Instead of picking one person from the line-up, we should have the witness assign a probability to each person plus “none of them”. Then blind vs. non-blind and simultaneous vs. sequential won’t matter, since the jury can just factor out the influence of the procedure on the probability assignments.

  • DL,

    I think you’re strongly overestimating people’s ability to handle probabilities. In fact, I wouldn’t want to be on such a jury.

  • Recovering irrationalist

    most striking finding was that the number of correct identifications dropped from 65% under British rules, to 36% under the strict rules.

    So (if representative) 45% of identifications are down to either official’s influence or just those biases dependent on simultaneous vs sequential presentation? I dread to think the percentage with all biases included.

  • billswift

    You might want to read Elizabeth Loftus’s Witness for the Defense. It has some stories about how police actually run/ran lineups, and some of the stupid (or evil, it’s hard to tell the difference unless they brag) things they’ve done.
    P.S. Loftus is a psychologist specializing in the study of memory, she has a couple of other good books out that I’ve read.

  • Doing away with any type of suspect protection (presumption of innocence, fair trial, &c) always reduces the number of convicted criminals, while also reducing the number of convicted not-guilty persons.

  • billswift

    @luispedro – You got it backwards.

    Doing away with any type of suspect protection (presumption of innocence, fair trial, &c) always **INCREASES** the number of convicted criminals, while also **INCREASING** the number of convicted not-guilty persons.