Statistical Discrimination is (Probably) Bad

A while back I wrote a post suggesting that it would be better if people didn’t statistically discriminate (i.e., refused to use information on the average characterists of a group when making judgments about individuals from that group). The idea (not original to me) was that an individual from a group with bad average characteristics will lack an incentive to invest in improving since they won’t be judged on their individual merits anyway. Various comments and discussions and trackbacks have generated a few further thoughts:

1. There is no guarantee that a refusal to discriminate will increase economic efficiency; for that to be true, it would have to be the case that the benefit of improved investment incentives outweighs the cost of discarding useful information.

2. The benefit of a refusal to discriminate increases if you place any weight on the normative proposition that everyone deserves to be judged on their own merits.

3. The benefit also increases if you believe that discrimination leads to alienation and various forms of costly anti-social behavior in the discriminated-against group.

4. Bryan Caplan suggests that statistical discrimination is at least mitigated, and possibly eliminated, by the fact that high-attribute individuals in groups with low average attributes have an incentive to “counter-signal” by taking some action to show that they are in fact high attribute. It is true that the possibility of counter-signalling will mitigate the harm from statistical discrimination, but I don’t see how it can ever make it go away. Someone who bears both the direct cost of investment and the additional cost of counter-signalling will have less incentive to invest than someone who bears only the direct cost. Furthermore, counter-signalling may not be cheap; it’s pretty darn costly to write a dissertation under an advisor known for high-tech mathematics just to show you don’t suck at math if you didn’t want to write with that guy anyway, you may just decide to punt and go to law school instead. Finally, the problem may accumulate over an individual’s life as each investment not made makes the next one costlier until the point where an investment that would otherwise have been possible no longer is.

GD Star Rating
Tagged as: ,
Trackback URL:
  • Paul Gowder

    Wouldn’t that be signaling rather than counter-signaling?

    Anyway, all this is pretty obviously right — but I’d add that #3 includes not just emotional effects, but a possible feedback loop with economic effects. For example… If you can’t get a job, you’re more likely to have to resort to crime not just because you’re alienated, but because on the margin some people get tipped over into crime being the utility-maximizing choice. And that increased number of criminals in the out-group increases the negative perception of the group, reducing their chance to get jobs, increasing the number of marginal people who have to turn to crime, etc. etc. etc. I have an intuitive suspicion (for which I have absolutely no evidence) that this sort of dynamic is at work in a bunch of places — certain kinds of sub-group crime, maybe even terrorism? (Action taken against Arabs makes terrorists makes people take action against Arabs makes terrorists, and “an eye for an eye makes the whole world go blind…”)

    Also, Bryan seems wrong in part. He says “as my Dixit example illustrates, if people think your group is bad in some way, the marginal benefit of counter-stereotypical behavior is probably unusually BIG. The expected mathematical ability of a non-libertarian who writes under Dixit goes from very good to excellent. The expected mathematical ability of a libertarian who writes under Dixit goes from poor to excellent. These look like incentives for a self-reversing prophesy to me.” But it isn’t the relative change in expected mathematical ability (and thus expected utility) that counts, it’s the absolute level. If a libertarian is considering seeking a career as a mathematician, he’s going to value the expected utility of that career path as the probability of successfully writing under Dixit multiplied by the absolute utility benefits from his expected mathematical ability, plus the probability of not being able to write under Dixit (maybe Dixit doesn’t like him) multiplied by the utility from that. The fact that the non-Dixit case kills him still reduces the expected utility.

  • Robin Hanson

    I’m not sure what it means to for a person to be “judged on their own merits.” We never know the real quality of people; we are always trying to infer this quality from various cues. Cues include race, gender, age, height, performance on some test, how many widgets you sold today, and so on. Anti-discrimination in practice says to ignore some cues, which forces you rely more on other cues. But every cue is effectively shared by many people, and so every cue you rely on treats someone as a member of that group, and it hurts some groups as it helps other groups. For example, if you rely on the sales today cue, you reward the group with high sales and hurt the group with low sales.

  • Paul Gowder

    Robin: “own merits” is, I think, an underlying normative judgment about the sorts of cues that ought to count. You’re of course right that there’s no fundamental difference between the sort of impact that height has on the judgment (say) of a hiring manager of the probability that a candidate will be a good seller and the sort of impact that yesterday’s sales have. (Except, of course, that the correlation coefficient is probably much much higher on the latter.)

    But I think many of us believe that judging someone on their merit doesn’t just mean maximizing the subjective probability that they’ll have the not-directly-observable qualities sought, but also conferring some kind of desert for good character and talent. (Even in market transactions.) So the short person who doesn’t get the sales job because of height discrimination feels cheated because he has worked just hard, has lived just as virtuous a life, and is just as good a seller as the tall person, but, by the accidents of genetics, happens to have a quality that’s poorly correlated with sales success, and as a result, has an objectively worse life. (For some reason, we count things like talent as meritorious despite the genetic/cultural accident that probably confers a lot of that too. But, well, that’s a whole ‘nother issue.) It’s fundamentally unfair from short-person’s perspective to get fewer good things in life because of a personal quality that has no causal relationship between the actual (as opposed to his expected) benefit he can provide to the other party in the transaction.

  • pdf23ds

    While it might be that if some forms of discrimination did not take place, that overall economic efficiency might rise (as well as societal equitability), that doesn’t mean that a given employer in our actual society would be better off not discriminating. The negative self-fulfilling effects of that discrimination are due to the discrimination as a whole, but the immediate positive effects are limited to the employer. So there’s a tragedy of the commons here.

  • Robin Hanson

    Paul, there are many ways to distinguish types of cues. One useful distinction is between cues you as an individual can influence, such as sales today, and cues that you cannot influence (much), such as race or gender or height. But anti-discrimination laws are not obviously based on this distinction – some forbidden cues such a religion are things we can influence, and some allowed cues such as beauty allow little influence.

  • anon

    Robin: “judging someone on their own merits” means assessing the relevant cues. Should we judge the mathematical ability of an economist based on whether or not he is a libertarian, or by using metrics that actually measure mathematical ability. Note that libertarianism, at least according to you, is something we can change (though I’m sure many people would have a problem with this point).

    Bryan Caplan’s point is a good one, but it doesn’t always work. People can falsely signal their own attributes; an arab terrorist could dress like a western businessman to avoid being searched at an airport. On the flipside, to simply search all people who appear to be arab could easily produce the negative externalities that David Balan refers to above.

  • Paul Gowder

    Robin: true enough. Although in some senses, religion can be understood as involuntary — it’s not like we choose our fundamental beliefs the same way that we choose whether we’re going to work or play poker tonight. And I don’t think the law’s purpose is to establish perfect fairness. I’ll bet that generally we do think it’s unfair, and we do say that we’re not judging people on their real merit, when we hire someone because of their beauty (for a non-beauty based job) — regardless of what the law says.

  • Greg Marsh

    Your first caveat seems to me to be the key one. While the ideal of non-discrimination is admirable enough, in practice ‘benign’ discrimination is practically essential as a labour-saving device. Consider university choice, for instance. There’s evidence that a significant component of the expected discounted utility of attending, say, Yale, stems from the generally positive judgments that people will make about students who have attended it. In gt terms, attendance acts both as a credible signal of an invidual’s willingness to forego earning a salary for four years and/or having an easy ride a on a less demanding course, as well as of her ability to fulfil the competitive entrance requirements.

    This kind of device might theoretically become inefficient if the resulting arms race had no diagonal benefits (beyond a ‘hygiene’ threshold of affluence, for instance, the race to acquire credible signals of wealth is probably orthognal to sum net utility), but in the context of acquiring an education, net utility probably rises. Nevertheless for the individual — as opposed to the group — it is not what I happen to learn on my course (how much of what is learned is really relevant in later life?) but precisely my expectation of others’ propensity discriminating that is my main incentive.

  • Stuart Armstrong

    But anti-discrimination laws are not obviously based on this distinction – some forbidden cues such a religion are things we can influence, and some allowed cues such as beauty allow little influence.

    I feel that’s more a question of “bowing to the inevitable” – trying to enforce anti-beauty discrimination would be impossible, and in most circumstances, religious identity is imovable – for belief and social reasons. Plus, the various religious groups are very influencial.

    Let’s look at one specfic case of discrimination: I’m a racist boss wondering whether to hire a salesman from another ethnic group. I’ve decided not to hire him. Several points appear:
    1) Turning him down on racist grounds definetly spares me any effort, a net gain.
    2) However, I have restricted my field of possible candidates – a loss, dependent on the number of other candidates.
    3) Other people may share my prejudice, so this salesman may not have sold well to them – a gain, dependent on how many others share my views.
    4) However, hiring a salesman from the same background as myself may mean he shares the same prejudices as me – a loss if there are clients of the other ethnicity. conversely, the one I turned down may have known a lot about selling to his own group.
    5) Hiring this worker, and interacting with him, may have developed my empathy for this other group. May be a loss or a gain (there are some professions – accounting, boder guards – where pretty much any empathy is bad). May have lead to new business opportunities, have made my judgement more accurate, or taken up my time for no efficiency gain.

    These are just a few points – there are many more.

    That last point – empathy – may be the key to anti-discrimination laws. They seem to hit precisely those domains where accurate empathy is most difficult (man to woman, the disabled, other religions, other cultures – other races is a more subtle, but comes under other cultures).

    Whethe you can mandate empathy by law is another matter…

  • Robin Hanson

    Stuart, I don’t see why beauty discrimination is any more inevitable than race or gender discrimination. And most things that we can change, like our jobs, locations, friends, etc. we usually prefer not to change.

    It is striking to me how many people want to speak up and declare support for standard anti-discrimination rules, and how little consensus there seems to be on reasons for these rules.

  • Stuart Armstrong

    It is striking to me how many people want to speak up and declare support for standard anti-discrimination rules, and how little consensus there seems to be on reasons for these rules.

    Good point – the rules are a messy hodgepodge, generally created when some group became organised enough to petition for them, are fought against by employers (or whatever group is targeted) and then slanted by whatever party happens to be in power at the time.

    They also tend to apply to things that are normally easy to measure (which is why beauty doesn’t often make the cut).

    And most things that we can change, like our jobs, locations, friends, etc. we usually prefer not to change.

    It’s no only a question of preference, but a question of effort, cost, and efficiency. Changing religions, in most circumstances, involves large costs and efforts, and a (short-term) disruption or lack of efficiency. Changing sexes and races is even more expensive, and not totally successful. This is all out of proportion with whatever real gain can be had by such change (as opposed to other’s judgement of that gain).

    Cost alone can’t be used as sensible underpining for anti-discrimination laws, as getting an education, say, or repressing one’s violent urges, also carry costs.

    Sticking to job discrimination: as a suggestion, would a sensible basis for anti-discrimination laws be “discrimination should not be made on the basis of elements which carry a large cost to change, and which are unreliable indicators of the ability of the applicant to do the job”?

    In that case, one could shift the burden of effort (figuring out if the applicant is trully qualified) to the employer (the effort being much less than a sex/religious change).

    That seems a good moral basis for the laws (and would include beauty in most professions). Whether this moral basis should be accepted is a judgement call, and its eventual implementation would need to be balanced against other issues.

    The situation would be somewhat different from nowadays – discrimination on the basis of a handicap would be considered the worst, as being the hardest to change. Permitted discrimination levels would also vary depending on technological developments.

    Any feelings on this idea?

    (PS: I don’t personally entirely agree with this principle, but my reservations feel more like emotional biases rather than reasoned objections)

  • Steve Sailer

    Are there any readers with children who can swear they don’t practice statistical racial discrimination themselves when it comes to their own children’s safety and welfare? Did they buy a house in an inner city black neighborhood because it was cheap, yet conveniently close to city jobs, with lots of street parking? And did they send their kids to the local public school (not a special program)? And do they have their kids walk home from the conveniently close local school?

  • David J. Balan

    pdf23ds, you are right that there is often an incentive to engage in statistical discrimination. I am arguing that it is a virtue to refrain from doing so even if one does have an incentive.

    Steve Sailer, I am not suggesting that people should be willing to pay an infinite price to avoid even a small amount of statistical discrimination. I’m certainly not willing to do that. I am saying that: (i) avoiding statistical discrimination is a worthy liberal principal that is worth sacrificing for; and (ii) the social payoffs are likely to be large. BTW, the example is not a good one because in the example the relevant fact is that the neighborhood is crime-ridden, which presumably can be objectively known without recourse to statistical discrimination.

  • pdf23ds

    “Stuart, I don’t see why beauty discrimination is any more inevitable than race or gender discrimination.”

    A strong argument could be made that beauty discrimination is much more integrated into our cognitive architecture, and much less amenable to manipulation by changing social mores: more of a bias than a prejudice. For instance, there’s not any organized group or grassroots movement that I’m aware of standing up for the rights of ugly people, as there is for women and black people and gay people, and even fat people. If fat people have their own movement and ugly people don’t, I don’t think there’s a lot of hope for one until ugliness becomes treatable. (Say, when we start having robot bodies or something.)

  • Michael Sullivan

    There’s a purely personal advantage to avoiding most statistical discrimination. It avoids succumbing to hammer-nail bias (“when all you have is a hammer, everthing looks like a nail”).

    Statistical cues tend to be very weak relative to other, more relevant cues. When they really are *all* you have, or in the rare cases where they provide fairly strong evidence, it makes sense to use them. But the road is littered with examples where the use of a statistical cue leads one to ignore much stronger individual cues that were available with little extra effort.

    I believe we have a signficant bias toward overvaluing our initial information about a problem (first-impression bias). To me, this is equivalent to “hammer-nail bias” — the first tool we use to look at a problem becomes a lens that we tend to have on everytime we look at the problem, so to speak. A tool that we use often enough, becomes a lens that we leave on our eyes all the time, and we have trouble imagining what the world would look like without it.

    Therefore it behooves us as much as possible to avoid letting a weak cue be the first information we process about someone/thing, or to make a habit of using a tool that primarily generates very weak cues.

    No, I don’t have any science to justify my claim, I’m just throwing it out there as a possibility based on anecdotal evidence. If I am correct, then using statistical discrimination in a computer algorithm for a first order sort on large data sets is not so damaging, but using it habitually in everyday life is likely to lead one down a precarious path.

  • konshtok

    The next post is about how much other people opinion matters to us
    If it’s true and we only give it about 30% then it means that the determining factor in the behavior of discriminated people is what they think they can and should do and not what the discriminating people think
    So discrimination needs to be very strong and self esteem very low to deter high enders from trying to improve their condition
    this all assumes that discrimination is not enforced by law and force

  • Tim Tyler

    The top paragraph seems like a straight-forwards recommendation
    to sacrifice one’s own welfare, in the hope of encouraging a
    large-scale social aim. It seems rather like a recommendation
    to donate money to some worthy charity.

  • Bill

    Discrimination. We need it, to some extent. We have to be able to tell the difference between good and bad, and there are certain characteristics of a person that provide tell-tale signs that the person is good or bad. However, the bad thing about discrimination is that people become “profilers” in that they pick up initial vibrations or characteristics, and then pre-maturely judge that person as either good or bad without really giving the person a chance. Like, a person can be really good, but you observe this person smoking crack. The person can still be really good, but buys and smokes it privately without letting it on, dealing it, or sharing it. Not that crack is good, but the person’s dependancy on it doesn’t harm anyone else but himself. On that note, we can make the judgement that even though the crack habit is bad, the person is harmless. On the other hand, though, his dealer…
    Discrimination is also used against black people in the form of a stereotype. “All tall black men play basketball,” is a good example. There are many false conclusions that can come out of this, one of which is that a “tall black man uses basketball as a crutch to get through life.” Which could mean, in essence, that “tall black men are unable to effectively learn, therefore, as long as they are good at basketball, an education is unnecessary.”
    I am a salesman. So, I might be selling products in a neighborhood, and the entire neighborhood has discriminated against me because of my choice of profession. I am turned away at the door because, “all salesmen are crooks.”
    Here’s a good one that Criminal Minds had a habit of characterizing. “All skinny white men with glasses are serial killers or perverts.” Although, one of the profilers is a skinny, paranoid, master mind of forensic science, the same conclusion without any further knowledge can be made about him. His fictional history, is actually, that as a high school student, he was raised by his bi-polar, agrophobic mother who was also schizophrenic. Though he was not any of those things, he “acted out” the character in his childhood until he finally was able to overcome it through forensic science to become one of the greatest crime-solving profilers in history. If we, as run-of-the-mill, judgemental people, had actually encountered this man early in his life, we would have tossed him to the drug dealers, making the assumption that he was a worthless human being. Then, he might have ended up some kind of perverted, serial killer.
    Interesting. How many people have YOU turned into killers due to false judgements on their characters?