I actually feel that it does, because I feel the medical market is quite close to the description I gave of it. The reasons for my belief are all subjective (personal experiences, friend’s experiences, and doctors-who-are-friends experiences, and – even worse – newspaper reports), so my belief isn’t very strong, but it is there (and very specific to the perverse-incentive aspect of noisy measures). Anders’ example has undermined my belief to some extent though.

I strongly doubt, however, that those advocating suppression of noisy measures have a proper economic model. They seem to have much more intuitive models, and focus only on the cost, not the benefits.

And, I forgot to add, the marginal medical loss for transferring effort.

well, the main noisy piece of info that we already have is government approval or registration. This involves a lot of studying, and work experience at low pay. Some of this will be signaling, some will improve doctor medical competence.

The cost of this sort of noisy signal is to reduce the pool of available doctors, and the benefit is to filter out some medically unfit doctors at the start.

Now consider the noisy signal of a biased medical league table. Add the following premises:
1) A given doctor can improve in the league table not by becoming better, but by transferring efforts from one domain to the other.
2) This transfer is medically detrimental.
3) People seek the first available tolerable doctor.

To simplify the model, assume a level of “tolerability” going from 0 to 100, with 50 being tolerable. Assume the transfer of effort costs the doctor 5 points on the tolerable scale.

Then any doctor with a tolerability above 55 should opt for the medically detrimental transfer of effort, since it will not cost him, and will benefit him if anyone is paying attention to the noisy measure of quality. Only doctors with tolerability in the 50-55 range will be motivated to actually improve. (This is wildly over-simplistic model, but it does capture the essence of what could happen, if the premises are reasonably correct. A more sophisticated and realistic analysis would rest on the marginal gain of transferring effort, compared with the marginal loss of reduced patient experience).

Compare that with the one-off cost of government approval. Its main detrimental effect is to increase costs. Its main benefit is to reduce gross medical error, and reduce (somewhat) the efforts a patient needs to furnish to find a tolerable doctor. Depending on the cost/benefit there, this could result in a net good (as the cost of a gross medical error is probably much higher than some extra monetary cost).

So to sumarise: there exists market structures that can make some noisy signals beneficial (mainly one-off signals), and others detrimental (updated biased league tables). That market structure does not seem totally unreasonable. Therefore we need to look at the data to see if the market actually has that structure. Anders’ example is an argument against this.

My personal prediction is that biased league tables will result in an increase in quality among bad doctors, and a decrease among good doctors. Anyone know if this prediction is born out or refuted by the evidence?

PS: this argument probably falls apart (even in my simple model) if one considers biased league tables in the absence of a governmental approval scheme. There, the amount of bad doctors in the system will be much higher, so the benefits of the league tables will become substantially higher than their drawbacks.

I am assuming that since that put misinterpreted in quotes, you don’t see this as a valid argument against publishing such stats.

The problem with a single summary statistic such as death rate is that it is not very meaningful without more information concerning potential confounders such as percentage of high-risk patients. Of course, I am sure that you wouldn’t make such an error, but what about the general public who isn’t very well-versed in statistics. I know that the general public thinks that they want info and stats about everything, but there are some statistics which can be EXTREMELY misleading to someone who doesn’t know any better… and they wouldn’t have a clue.

All, every product where we get quality clues can suffer the same problems. Yet we usually allow people access to many quality clues. So,

Michael, why would base-rate neglect be a worse problem in medicine?

Jor, why would perverse incentives from quality measures be a bigger problem in medicine? (And as I asked before, why do added clues have worse problems than the basic clues?)

Perverse incentives are a big problem in medicine, and I don’t know if we need another source for metrics that might be of questionable utility and easily distorted. If you need an example of perverse incentives, just look at some of the financial incentives in medicine, and you can easily see the over-proceduralization and excess diagnostic studies performed due in part to the absurd re-imbursement system we have in place.

If you are dying, do you prefer to have Dr. House or a charming quack?

I’ll take Dr House, anytime.