This review article published 36 years ago shows that it was well known back then that teacher evaluations by college students are predictably influenced by time of day, class size, course level, course electively, and more. Thus one could get more reliable teacher evaluations by building a statistical model to predict student evaluations using these features plus who taught what, and then using each teacher coefficient as that teacher’s evaluation. Yet colleges almost never do this. Why?
But this is worse than no explanation at all. "When mob bosses who are best at violence rise to the top of a competition for boss-hood, why should they and their allies favor non-violent criteria for how to pick bosses?" Because they don't want to die. Bosses who got their jobs thru politics should be especially wary of underlings who use politics. Like Stalin, they should get rid of anybody near the top who poses a threat.
If teaching during a particular time slot consistently yields lower evaluations, and evaluations matter at all for tenure, then whoever sets the schedule can threaten teachers who want tenure with unfavorable time slots, which then factors into coalition politics.
Non-rhetorical question: how much are subordinates' coalitions the same as bosses', especially as you go up multiple levels? If the boss is neutral in most of the politics below, the incentive is to prevent costly politicking, but if he has reason to prefer one side over the other, he'll encourage it. I can't imagine low-level employees' politics being relevant to upper management; in what meaningful way could a base-level worker take sides against the CEO? Which means upper management could benefit by enforcing fairer evals at low levels.
If people really cared about fairness in employment, they'd make everyone's salaries public. I mean, take the case famous of Lilly Ledbetter: it took her years to discover that she was making far less money than her male co-workers.
"Perhaps they create a lower bound for employee performance."
This may be true for professors: the grades put on the evals are mostly worthless, but the optional written comments will highlight negative excesses. Really bad grades on the eval will also signal something (class hours and cupakes may mean the difference between a 6out of 10 and an 8 out of 10, but if multiple classes give a professor a 2 out of 10 there may be something to it). Evals may be able to catch negative excesses.
Perhaps they create a lower bound for employee performance.This is similar to the way that even though elections don't give us great people as leaders, we get much better leaders than we would get if they didn't have at least _some_ form of accountability.
"performance often having a statistically significant but substantively small effect on advancement"
The glass could be half full rather than half empty. How much variation among orgs is there? If there is much, do ones in which performance has a larger effect on advancement outperform others? Apart from obviously not enough to dominate and thus change the quote to "...and...large".
"Why do firms have evals at all, if they're known to be broken?"
HR isn't the stock market, we can't just assume (almost) all information is already known and widely disseminated. Even if everyone knew evals are broken it would take many cycles for alternatives to be tried, tested and the results tobe widely understood and disseminated. A single cycle can last for many years because we're talking about people's careers here. Quite possibly the system just hasn't been running long enough.
Also I think the assumption that ANYONE really cares about actual teacher performance, much less their evaluations, is misguided.
Most academics tend to care more about their research and for the most part, being academics themselves, so do most people in the administration. So why have evals at all or consider them in granting tenure? Well I think they are largely a mechanism to placate students and parents while also giving the school at least some control over the truly awful educators and making sure professors didn't simply completely ignore teaching to the point where it created actual bad PR and resentment from students.
I think those people in academia who honestly really care about teaching (and there are a substantial number) already realize that even controlling for all these factors there is no reason to think teaching evals truly reflect the pedagogical performance of the teacher and not their willingness to engage in eval boosting but student harming tricks. For instance, when a substantial fraction of your curriculum is out of your hands student evals do better when you feign excitement and lie about the usefulness of useless material (like integration tricks that should be relegated to computers) even though the students are better off realizing they should focus their time elsewhere. Furthermore, if time of day and the like effect evals so strongly that controlling for them makes a substantial difference than after controlling things like handing out cupcakes (yes I know people who do it) probably would make an even bigger difference and render the evals negatively correlated with the professor's honesty.
So I think that once people think hard about evals they either don't want them to be taken seriously or realize that they can be so easily gamed as to render their results worthless. Thus no one wants to further legitimize them by appearing to take what they show seriously by building a complicated model that people might blindly trust.
What makes this even more horrifying are some of the models that suggest that actual contribution of the better people may be highly nonlinear in their job performance.
Correcting teacher evaluations for time of day, etc., wouldn't diminish coalition politics. So, you have an example showing that measurements aren't improved even when there's no cost to coalition power. This actually weakens the argument.
And I'm no mob expert, but I bet the bosses don't take it well if a subordinate whacks a rival to get a promotion, no matter how many times the boss has done exactly that.
If the whacked rival was a member of an opposing coalition, they might love it.
I think we need to dig more into why people at the top don't (or can't) discourage politics below. Even if they're politicking themselves, they could benefit by preventing it at lower levels--it would just take a little hypocrisy. It doesn't seem that hard to say that senior management performance can't be evaluated the same way as at lower levels. And I'm no mob expert, but I bet the bosses don't take it well if a subordinate whacks a rival to get a promotion, no matter how many times the boss has done exactly that.
Why do firms have evals at all, if they're known to be broken? It could be that, while they don't measure performance accurately, they're doing *something* perfectly well, like reassuring stockholders or making employees think their performance matters. Maybe it's not feasible to achieve meritocratic promotions, and accurate evals would only put it on paper how un-meritocratic your processes are. Or maybe 'better' evals would just be gamed at higher organizational cost.
I figured that broken-eval systems were used mostly to assuage upper management anxiety and reduce liability issues when you fire someone. Nobody really cares whether they work or not, as long as the company seems to be hitting its numbers and working reasonably well.
But this is worse than no explanation at all. "When mob bosses who are best at violence rise to the top of a competition for boss-hood, why should they and their allies favor non-violent criteria for how to pick bosses?" Because they don't want to die. Bosses who got their jobs thru politics should be especially wary of underlings who use politics. Like Stalin, they should get rid of anybody near the top who poses a threat.
Definitely check out the book Moral Mazes if you haven't already.
If teaching during a particular time slot consistently yields lower evaluations, and evaluations matter at all for tenure, then whoever sets the schedule can threaten teachers who want tenure with unfavorable time slots, which then factors into coalition politics.
Non-rhetorical question: how much are subordinates' coalitions the same as bosses', especially as you go up multiple levels? If the boss is neutral in most of the politics below, the incentive is to prevent costly politicking, but if he has reason to prefer one side over the other, he'll encourage it. I can't imagine low-level employees' politics being relevant to upper management; in what meaningful way could a base-level worker take sides against the CEO? Which means upper management could benefit by enforcing fairer evals at low levels.
The purpose of performance evaluations is to justify decisions, not to make them.
If people really cared about fairness in employment, they'd make everyone's salaries public. I mean, take the case famous of Lilly Ledbetter: it took her years to discover that she was making far less money than her male co-workers.
"Perhaps they create a lower bound for employee performance."
This may be true for professors: the grades put on the evals are mostly worthless, but the optional written comments will highlight negative excesses. Really bad grades on the eval will also signal something (class hours and cupakes may mean the difference between a 6out of 10 and an 8 out of 10, but if multiple classes give a professor a 2 out of 10 there may be something to it). Evals may be able to catch negative excesses.
Perhaps they create a lower bound for employee performance.This is similar to the way that even though elections don't give us great people as leaders, we get much better leaders than we would get if they didn't have at least _some_ form of accountability.
"performance often having a statistically significant but substantively small effect on advancement"
The glass could be half full rather than half empty. How much variation among orgs is there? If there is much, do ones in which performance has a larger effect on advancement outperform others? Apart from obviously not enough to dominate and thus change the quote to "...and...large".
"Why do firms have evals at all, if they're known to be broken?"
HR isn't the stock market, we can't just assume (almost) all information is already known and widely disseminated. Even if everyone knew evals are broken it would take many cycles for alternatives to be tried, tested and the results tobe widely understood and disseminated. A single cycle can last for many years because we're talking about people's careers here. Quite possibly the system just hasn't been running long enough.
Also I think the assumption that ANYONE really cares about actual teacher performance, much less their evaluations, is misguided.
Most academics tend to care more about their research and for the most part, being academics themselves, so do most people in the administration. So why have evals at all or consider them in granting tenure? Well I think they are largely a mechanism to placate students and parents while also giving the school at least some control over the truly awful educators and making sure professors didn't simply completely ignore teaching to the point where it created actual bad PR and resentment from students.
I think those people in academia who honestly really care about teaching (and there are a substantial number) already realize that even controlling for all these factors there is no reason to think teaching evals truly reflect the pedagogical performance of the teacher and not their willingness to engage in eval boosting but student harming tricks. For instance, when a substantial fraction of your curriculum is out of your hands student evals do better when you feign excitement and lie about the usefulness of useless material (like integration tricks that should be relegated to computers) even though the students are better off realizing they should focus their time elsewhere. Furthermore, if time of day and the like effect evals so strongly that controlling for them makes a substantial difference than after controlling things like handing out cupcakes (yes I know people who do it) probably would make an even bigger difference and render the evals negatively correlated with the professor's honesty.
So I think that once people think hard about evals they either don't want them to be taken seriously or realize that they can be so easily gamed as to render their results worthless. Thus no one wants to further legitimize them by appearing to take what they show seriously by building a complicated model that people might blindly trust.
What makes this even more horrifying are some of the models that suggest that actual contribution of the better people may be highly nonlinear in their job performance.
Plausible explanation, tortured evidence.
Correcting teacher evaluations for time of day, etc., wouldn't diminish coalition politics. So, you have an example showing that measurements aren't improved even when there's no cost to coalition power. This actually weakens the argument.
And I'm no mob expert, but I bet the bosses don't take it well if a subordinate whacks a rival to get a promotion, no matter how many times the boss has done exactly that.
If the whacked rival was a member of an opposing coalition, they might love it.
I think we need to dig more into why people at the top don't (or can't) discourage politics below. Even if they're politicking themselves, they could benefit by preventing it at lower levels--it would just take a little hypocrisy. It doesn't seem that hard to say that senior management performance can't be evaluated the same way as at lower levels. And I'm no mob expert, but I bet the bosses don't take it well if a subordinate whacks a rival to get a promotion, no matter how many times the boss has done exactly that.
Why do firms have evals at all, if they're known to be broken? It could be that, while they don't measure performance accurately, they're doing *something* perfectly well, like reassuring stockholders or making employees think their performance matters. Maybe it's not feasible to achieve meritocratic promotions, and accurate evals would only put it on paper how un-meritocratic your processes are. Or maybe 'better' evals would just be gamed at higher organizational cost.
I figured that broken-eval systems were used mostly to assuage upper management anxiety and reduce liability issues when you fire someone. Nobody really cares whether they work or not, as long as the company seems to be hitting its numbers and working reasonably well.