Before the modern world, most jobs had a big physical component. And so physical ability (strength, speed, stamina, coordination, etc.) was one of the main things people tried to show off. Yes, people did try to show off physical abilities on the job. But when people got serious about showing off, they created special off-the-job contests, such as races and games.
Agreed. In some kinds of institutions the biggest obstacle is an unwillingness to use metrics. When is it a sign of unfortunate incentives and when it is a sign of innumeracy and failure to appreciate the possibilities?
To my knowledge, no one has attempted to replicate the parole study. But the theoretical basis for interpreting it is thrown into question by the failure of replication attempts of studies supporting the ego-depletion/decision-fatigue concept.
Thanks Stephen. Aware of the replicability issue generally (and how enticing a meme the study had produced) I did a cursory search for followups found only noncritical summaries. I *was* impressed that the article was edited by Daniel Kahneman (poke at myself there:-)
The Slate article is fascinating but doesn't seem to mention the parole study. Prompted by your comment I did find a critical response to the parole study by Weinshall-Margell and Shapard, and a response to that by the original study authors (Danzinger et al), both in 2011. But I gather from your comment that there's more than just that. Do you have any pointers?
This might be one of the reasons why so many software developers prefer to write "interesting" tools themselves, especially compilers, instead of reusing them. "I wrote a compiler" is such a great claim to make, even if the compiler was written (and a programming language invented) as part of something like developing an accounting app.
I strongly agree with this. The alternative to experts is often laymen who often have neither the incentive nor the ability to discriminate between better and worse options. I think this is an argument for less expert involvement, but I don't think it's an argument for greater populist involvement. The danger of inadvertently implying support for greater populism is such that I think whenever making an argument against experts, it is very important to stress that an argument against experts is not an argument in favor of populism.
The study you cite and its offspring is at the center of the "replicability crisis" in social psychology. ( http://www.slate.com/articl... )
Concidentally, this crisis is an excellent example of Robin's topic. There is little prestige in replicating the findings of others. All the fun (read signaling value) is in designing clever experiments, not redoing those designed by others.
It *is* famously difficult fto get health professionals to hand-wash diligently, though that is critical to their ostensible mission. I gather the same is true of checklist use. And come to think of it, the perverse, pervasive, persistent tradition of abysmally-scrawled prescriptions affords writer *and* reader a minor chance to show off with an apparently acceptable level of collateral injuries and deaths.
Do such things correlate with the individual health professional's level of prestige? Inversely with the patient's degree of connection to the professional?
PS. That said it's not all about prestige. I've got to believe that lots of these folks try harder to do a good job than necessary to maximize their prestige. And other extraneous factors can have big impacts too: Recall the 2011 peer-reviewed study of 1,000 Israeliparole applicants which found that “You are anywhere between two and six times aslikely to be released if you’re one of the first three prisoners considered[during a break-less session] versus the last three prisoners considered.”
If there are many prestigious people, they will be tempted to choose the part of the field where it is easier *in general* to show off. But if there are only a few prestigious people (in extreme case: only one), they may choose the part of the field where they *personally* have an advantage.
For example, a senior developer in an IT company may insist that everyone must use the technology the senior developer has most experience with, even when it does not fit the project's needs. (Even if it is *not* the technology that developers *in general* would use to show off.) Of course, if other developers will suggest another technology, it is easy to accuse them of doing exactly this.
I agree with you, and I also think your warning touches on a very specific case of a general problem: we tend to award discretion based on demonstrated levels of technical skill, rather than based on demonstrated levels of altruistic virtue. That works fine when the discretion is about a technical topic, e.g., how many milligrams of this drug should be prescribed -- but it works terribly when the discretion is about a social topic, e.g., who should have access to this drug.
I'm not sure how likely professionals are to abuse their discretion *specifically* for the purpose of showing off their high skill level, but I'm sure professionals abuse their discretion all the time for other reasons, because we barely have any controls designed to ensure that professionals who have the most discretion also have the highest ethical standards.
It seems like this problem is intensified (1) if the public doesn't have a good way to evaluate the properties it really wants, or (2) if the highest-prestige workers have extraordinarily higher productivity (and so are in a position to demand the work be structured in the way they want as part of their pay).
It could be worse though. We could let non experts judge. The problem is non experts are even in less position to judge. They won't know the problems or possible solutions, they won't know what to look at, what to measure, how to measure it, or whether it is indeed measured. The best one can hope for is multiple independent groups of experts with high ethics cross checking each other but able to pursue their own path.
We need great metrics to measure performance. Sometimes (often in fact) we need to put a lot of work into developing and testing metrics for relevance and accuracy. Too often the great metrics do not get developed and poor metrics create a misleading picture.
We run into the very big problem that things that get measured end up skewing incentives toward optimizing those measures at the expensive of other aspects of the system not so easily measured. This has led many companies down.
Ditto academic institutions. Look at the journals that exist to give academics places to publish. It is much easier to measure quantity than quality when every product is unique.
I think the problem is more fundamental; basic human prestige programming.
One must also look at the balance of influences. Even under the reign of experts, there are countervailing pressures to mediocrity.
Agreed. In some kinds of institutions the biggest obstacle is an unwillingness to use metrics. When is it a sign of unfortunate incentives and when it is a sign of innumeracy and failure to appreciate the possibilities?
To my knowledge, no one has attempted to replicate the parole study. But the theoretical basis for interpreting it is thrown into question by the failure of replication attempts of studies supporting the ego-depletion/decision-fatigue concept.
Thanks Stephen. Aware of the replicability issue generally (and how enticing a meme the study had produced) I did a cursory search for followups found only noncritical summaries. I *was* impressed that the article was edited by Daniel Kahneman (poke at myself there:-)
The Slate article is fascinating but doesn't seem to mention the parole study. Prompted by your comment I did find a critical response to the parole study by Weinshall-Margell and Shapard, and a response to that by the original study authors (Danzinger et al), both in 2011. But I gather from your comment that there's more than just that. Do you have any pointers?
This might be one of the reasons why so many software developers prefer to write "interesting" tools themselves, especially compilers, instead of reusing them. "I wrote a compiler" is such a great claim to make, even if the compiler was written (and a programming language invented) as part of something like developing an accounting app.
I believe you are referring to algorithm aversion with that statement.
I strongly agree with this. The alternative to experts is often laymen who often have neither the incentive nor the ability to discriminate between better and worse options. I think this is an argument for less expert involvement, but I don't think it's an argument for greater populist involvement. The danger of inadvertently implying support for greater populism is such that I think whenever making an argument against experts, it is very important to stress that an argument against experts is not an argument in favor of populism.
The study you cite and its offspring is at the center of the "replicability crisis" in social psychology. ( http://www.slate.com/articl... )
Concidentally, this crisis is an excellent example of Robin's topic. There is little prestige in replicating the findings of others. All the fun (read signaling value) is in designing clever experiments, not redoing those designed by others.
It *is* famously difficult fto get health professionals to hand-wash diligently, though that is critical to their ostensible mission. I gather the same is true of checklist use. And come to think of it, the perverse, pervasive, persistent tradition of abysmally-scrawled prescriptions affords writer *and* reader a minor chance to show off with an apparently acceptable level of collateral injuries and deaths.
Do such things correlate with the individual health professional's level of prestige? Inversely with the patient's degree of connection to the professional?
PS. That said it's not all about prestige. I've got to believe that lots of these folks try harder to do a good job than necessary to maximize their prestige. And other extraneous factors can have big impacts too: Recall the 2011 peer-reviewed study of 1,000 Israeliparole applicants which found that “You are anywhere between two and six times aslikely to be released if you’re one of the first three prisoners considered[during a break-less session] versus the last three prisoners considered.”
If there are many prestigious people, they will be tempted to choose the part of the field where it is easier *in general* to show off. But if there are only a few prestigious people (in extreme case: only one), they may choose the part of the field where they *personally* have an advantage.
For example, a senior developer in an IT company may insist that everyone must use the technology the senior developer has most experience with, even when it does not fit the project's needs. (Even if it is *not* the technology that developers *in general* would use to show off.) Of course, if other developers will suggest another technology, it is easy to accuse them of doing exactly this.
I agree with you, and I also think your warning touches on a very specific case of a general problem: we tend to award discretion based on demonstrated levels of technical skill, rather than based on demonstrated levels of altruistic virtue. That works fine when the discretion is about a technical topic, e.g., how many milligrams of this drug should be prescribed -- but it works terribly when the discretion is about a social topic, e.g., who should have access to this drug.
I'm not sure how likely professionals are to abuse their discretion *specifically* for the purpose of showing off their high skill level, but I'm sure professionals abuse their discretion all the time for other reasons, because we barely have any controls designed to ensure that professionals who have the most discretion also have the highest ethical standards.
It seems like this problem is intensified (1) if the public doesn't have a good way to evaluate the properties it really wants, or (2) if the highest-prestige workers have extraordinarily higher productivity (and so are in a position to demand the work be structured in the way they want as part of their pay).
But we often have plenty good metrics that people don't want to use.
It could be worse though. We could let non experts judge. The problem is non experts are even in less position to judge. They won't know the problems or possible solutions, they won't know what to look at, what to measure, how to measure it, or whether it is indeed measured. The best one can hope for is multiple independent groups of experts with high ethics cross checking each other but able to pursue their own path.
We need great metrics to measure performance. Sometimes (often in fact) we need to put a lot of work into developing and testing metrics for relevance and accuracy. Too often the great metrics do not get developed and poor metrics create a misleading picture.
We run into the very big problem that things that get measured end up skewing incentives toward optimizing those measures at the expensive of other aspects of the system not so easily measured. This has led many companies down.
Ditto academic institutions. Look at the journals that exist to give academics places to publish. It is much easier to measure quantity than quality when every product is unique.
Metrics are hard.