Preferred Unfair Evaluations

Imagine a company had a team of sales people, who were assigned to sales regions that varied in their promise and difficulty of sales. The people assigned to easy rich regions tended to have high sales, while those assigned to poor difficult regions tended to have low sales. And such sales figures were used to decide compensation, raises, promotions, etc.

You might imagine many would call this unfair, and that embarrassed leaders would change their evaluation system to control for the varying sales regions. They might rotate people between different regions, or change the size and location of regions to make them more similar.

But in a great many places, you’d just be wrong. They wouldn’t change the regions, and they’d feel fine using region sales to allocates praise and rewards. Don’t believe me?

Consider that we teachers are judged on student evaluations that do not control for the difficulty of our class or students. Consider that we judge students on GPAs that don’t control for difficulty of class or teacher or time of day. Consider that we academics are judged on our number and level of publications, but without controlling for what resources or obstacles we had re such things, like grants, teaching loads, student assistants, prestigious affiliations, etc.

It would be straightforward to start down the road of trying to control such things. Things might get harder somewhere down that road, but the first part of the road is pretty easy. Yet we don’t even start.

And consider that we all know that elections are distorted by the fact that many voters are not very well informed. It would be easy to correct for this, as Jason Brennan explains:

On Election Day, everyone gets to participate, and participate as an equal. However, when they participate, they do not merely vote for a candidate, party, or position on a referendum. Rather, they have to do three things:

1. Tell us who they are, by indicating their demographic information, such as sex, gender identity, income level, ethnicity, employment status, and so on. …

2. Citizens will take, say, a thirty-question quiz of basic political information.

3. Tell us … which candidate or party they support in an election, or which position they support in a referendum.

… Once we have all three sets of data, the data is anonymized and released to the public domain. The government electoral commission then uses the data to estimate, via predetermined methods, what the public would have wanted if it were demographically identical but had gotten a perfect score on the knowledge test. This result—the public’s enlightened preference—is then instantiated. For instance, if the enlightened public favors Remain but the actual public favors Leave, the country remains. Since the data is public, the government’s calculations can easily be verified or challenged. …

What goes on the test? Answer: Have the citizens decide using a deliberative poll. A month or so before preference voting day takes place, randomly select, say, five hundred citizens from around the country. Pay them to spend a few days deliberating to design the thirty-question battery of questions. Require their employers not to penalize them.

Yet people seem overwhelmingly opposed to such a policy. I’m not sure what to call it, but there’s something deep and important to understand here, on why we often don’t want to correct when we can for blatant unfairness.