Top Docs No Healthier

My two years a RWJF Health Policy Scholar exposed me to enough data to make me a skeptic on the marginal aggregate health value of medicine.  But where data is silent I try to give medicine the benefit of the doubt, such as in assuming average values are higher than marginal values, and that top med school docs give more value than others.   So I am shocked to report that in a randomized trial of 72,000 hospital stays by 30,000 patients, patients of top med school docs were no healthier:

The school affiliated with Program A is the top school in the nation when ranked by the incoming students’ MCAT scores, and it is always near the top. In comparison, the lower-ranked program [B] that serves this VA hospital is near the median of medical schools. … [Added: other ways A beats B here.] Patients treated by the two teams have identical observable characteristics and have access to a single set of facilities and ancillary staff. …

Health outcomes are not related to the physician team assignment. … Program B is associated with … a 0.3 percentage-point reduction in 5-year mortality (or 0.6% of the mean).  … The confidence interval is [-0.0162, 0.0106]. …


Those treated by physicians from [A] … have 10-25% shorter and less expensive stays than patients assigned to the lower-ranked institution. .. For example, patients assigned to physicians from Program B are more likely to undergo diagnostic tests compared to patients treated by Program A (73% vs. 68%).

Procedure differences across the teams are consistent with the ability of physicians in the lower-ranked institution to substitute time and diagnostic tests for the faster judgments of physicians from the top-ranked institution. … This is consistent with a group that is either more careful or a group that requires more time and information to understand the nature of the condition.

This ignores the possibility that lower rank docs order more tests because they rightly fear more malpractice suits.  After all, the paper notes:

Hartz et al. (1999) show that surgeons are more likely to be regarded as a "best doctor" in these community surveys if they trained at a prestigious residency or fellowship program. They note that treatment by physicians trained at prestigious programs is not related to mortality, however.

And a malpractice suit study says: 

The Medical Malpractice Project attempted to review every malpractice suit filed in North Carolina between July 1, 1984, and June 30, 1987 – 895 cases.  … The central issue in many of the trials … did not involve technical issues, despite a lot of testimony on these subjects, but rather the credibility of the doctor or other health care providers compared with the credibility of the plaintiff.   

This all suggests that while med schools may feel competitive pressure to achieve prestige, there isn’t much pressure to teach students how to make patients healthier.  HT to Michael Cannon on the main study, retired urologist on the malpractice study.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • Phil

    Or perhaps the quality of the doctors doesn’t correlate with their MCAT scores or the perceived quality of their schools.

    Or, the quality of medical care is not highly correlated with the aptitude of the doctor. Other characteristics might be important, such as doctor workload, nurse quality, institutional characteristics, hand washing practices, and so on.

    I have no evidence, but believe both of the above are true. If I need medical care, I want a better doctor, but MCAT score would not be how I defined “better”.

  • Stephen Day

    Do you think the results would be different if the study was done with information from physicians in private practice?

  • http://drchip.wordpress.com/ retired urologist

    Disclaimer: I have neither conducted nor knowingly participated in any studies about the quality of medical care. I am a qualified reader (in English).

    This study, I suspect conducted in the Boston VA system (comparing Harvard with U Mass?), confirms what most US physicians in private practice have anecdotally concluded long ago: in the aggregate, American doctors deliver about the same level of care, regardless of US medical background. The first paragraph in this post gives some reasons for this. Because of the US training system, there are no “bad” US medical schools, or at least they are all about the same level of “bad” or “good”, regardless of “prestige”.

    Program A is the top school in the nation when ranked by the incoming students’ MCAT scores.

    This is a silly way to rank medical schools; it uses a score that is achieved before the students’ exposures to the schools themselves, rather than after. In a general sense, MCAT scores are possibly related to IQ scores, but apparently, given the material that is to be mastered, there is a requisite level of intelligence required, beyond which other factors are more important than native intelligence. The study does not mention the medical-school education of the residents or the attendings, who actually determine the outcome. **Anecdote about “rankings”: In my day, residents in all urology programs took the “In-Service” exam, testing every member of the program on the same material, regardless of current level of training. Overall programs were then assigned rankings. In 1978, the top-ranking score was a community (non-academic) hospital program at Confederate Memorial Hospital in Shreveport, LA. Second was the U of Kentucky (my program). The “big” names were down the list. Interestingly, rankings were not announced after that.

    This is consistent with a group that is either more careful or a group that requires more time and information to understand the nature of the condition.

    The authors have the experience to know that this is only one possible (and I believe a less likely) explanantion of the hospital-stay times. University center medical care is a bureaucracy, pure and simple. It all flows downhill. The medical students make no decisions at all, the intern (noted in the article as the primary care-giver) makes mostly daily-work types of decisions, and the overall course of care is decided by the senior resident and the attending. The senior resident has never worked on his own, and surprisingly, in the university setting, usually the same is true of the attending: he has always practiced in a bureaucracy. This study shows that the upper-level bureaucrats at the prestige school are perhaps more efficient bureaucrats, and that better bureaucracy does not lead to better health.

    A much better study of the quality of training in the two programs (only hypothetical; I don’t suspect it could be done) would be to follow the physicians in purely private practice after their training, and compare the outcomes. Or, to get into the guts of the Rand Experiment, determining the source of medical training of those on the upper and lower ends of the data.

    This all suggests that while med schools may feel competitive pressure to achieve prestige, there isn’t much pressure to teach students how to make patients healthier.

    Indeed. People have long wondered what the US would be like if the 3 branches of government were comprised of people with proven practical success records, instead of career politicians. Medicine is the same. The “thought-leaders” in academia (the aggregate, not individuals) are exposing our doctors to this type of education.

  • http://hanson.gmu.edu Robin Hanson

    retired, the paper also says this to distinguish programs A, B:

    Another commonly used measure to compare medical schools is funding from the National Institutes of Health (NIH). This ranking identifies the major research-oriented medical schools, again with some of the most prestigious schools near the top. The medical school associated with Program A is again among the top schools in the U.S., whereas the lower-ranked program has an NIH funding level that is generally less than three out of every four medical schools.

    Second, each training program is affiliated with another teaching hospital in the same city, in addition to the VA hospital. Program A’s “parent hospital” is ranked among the top 10 hospitals in the country according the U.S. News and World Report Honor Roll rankings of hospitals. Out of 15 specialties ranked by U.S. News, Program A’s hospital is among the top 10 hospitals in the country for nearly half of them, and among the top 20 in nearly all of them (U.S. News & World Report, 2007). Meanwhile, Program B’s parent hospital is not a member of this Honor Roll overall or ranked among the top hospitals in terms of subspecialties. …

    Approximately 30% of residents who were trained in Program A received their M.D. from a medical school in the top 10 of the U.S. News and World Report rankings in 2004, compared to 3% of those trained in Program B. For top-25 medical schools, approximately half of Program A’s residents graduated from such a school, compared to less than 10% for Program B. Similar differences are seen when the residents’ medical schools are ranked by NIH funding levels. In addition, twice as many of Program B’s physicians earned their medical degree from a medical school outside of the U.S. …

    The [board-certification exam] pass rate for Internal Medicine is close to 100% for the residents in Program A compared to a pass rate of approximately 85% for Program B (a rate that is in the bottom quartile of the 391 programs listed). The pass rate for General Surgery is lower, 85% for Program A and 60% for Program B. These scores place Program A in the top quartile and Program B in the bottom quartile of residency programs in the U.S. …

    A survey in the early 1970s asking medical school faculty to rank programs included Program A in its top 10, whereas Program B was ranked near the median of the rankings.

  • http://drchip.wordpress.com/ retired urologist

    Robin Hanson: …NIH grants.

    I didn’t want to increase the length of the post, so I didn’t mention that. NIH grants, of course, go to the professors, who supposedly are already trained. In my experience (there’s that anecdote again), the grants have a net negative effect both on the training of the housestaff and on the care received by the patients, because the emphasis is on enrolling patients in studies (hammer/nail syndrome). This attracts patients with unusual maladies or patients who have reached the limit of perceived effective care, and are present to be enrolled in an experimental study. In my own field, for instance, patients at a nearby “prestigious medical school” who were candidates for proven effective therapy for ED, fully covered by their insurance, were instead steered away from that toward a nonsense study for which the professor had received a grant from private industry to evaluate a penile ointment. (for much more info about such practices, email me).

    Second

    I don’t understand the criteria for such rankings. Is it outcomes, or is it degrees, honors, test scores, name recognition, or what? A local for-profit cardiovascular hospital in my town had to prove better outcomes at less expense in order to continue existing. Is this what is meant by the ranking?

    My personal bias is that I agreed with article’s conclusions before I knew it existed. Talk about false medical economy: the annual tuition/fees at Harvard Medical School this year are $41,861. At the Medical College of Georgia (my medical school) tuition is $4334. Yet, both institutions offer the same “union card” and, as this article shows, the same product. I may be a good ole Southern boy, but I wouldn’t pay $10000 at Bloomingdales (if they sold such) for the exact lawnmower I could get at Tractor Supply for $500.

  • http://www.iphonefreak.com frelkins

    Who wants a doctor anyway, when I could have the robot?

    “extensive studies at Guy’s Hospital in London, among others, have shown that robots significantly improve surgical accuracy, particularly among the less-skilled.”

    Perhaps you left at the right time, RU, as it seems these techniques are especially suited to urology.

  • http://neuraltransmissions.wordpress.com MZ

    retired urologist:

    Program A is the top school in the nation when ranked by the incoming students’ MCAT scores.

    This is a silly way to rank medical schools; it uses a score that is achieved before the students’ exposures to the schools themselves, rather than after.

    Indeed. Why not use USMLE scores, or the scores from board certification exams, if the physicians have gotten that far.

    My personal bias is that I agreed with article’s conclusions before I knew it existed. Talk about false medical economy: the annual tuition/fees at Harvard Medical School this year are $41,861. At the Medical College of Georgia (my medical school) tuition is $4334. Yet, both institutions offer the same “union card” and, as this article shows, the same product. I may be a good ole Southern boy, but I wouldn’t pay $10000 at Bloomingdales (if they sold such) for the exact lawnmower I could get at Tractor Supply for $500.

    I think this applies to all academic institutions from medical schools down to elementary schools. When my mother brought up the possibility of sending me to a private high school, my wise father responded, “It doesn’t matter where you study. All that matters is what’s between your ears.”

    I was the kind of kid who, in middle school, would pull volumes of the Encyclopedia Britannica off the shelf and read random articles (this was before the internet). I was *intellectually curious* and capable of educating myself, and that was more valuable than any Ivy League education. I went to public schools, public universities, and turned out just fine.

    These days, anyone can get a Harvard education for free on the internet. Granted, you can’t learn procedural knowledge like flying an airplane or performing a quadruple bypass through the internet, but you can acquire academic knowledge. The tools are there for everyone. It doesn’t matter which school you go to.

    Ultimately, the degree is just a certificate that gets your foot in the order because society needs SOME cheap and easy way to determine who really bothered to learn the requisite material (they offload the cost onto you, the prospective employee, who had to pay the tuition for the degree).

    And what you really pay for at an Ivy League school is the entrance fee to an *alumni network* that can get you a better paying job. But that’s not how Rational Actors SHOULD behave.

  • http://drchip.wordpress.com/ retired urologist

    Did some more research on rankings, and possible explanations for care-levels.

    *interesting statistic: Physicians and surgeons held about 633,000 jobs in 2006; approximately 15 percent were self-employed. Why would a non-research doctor who considers himself competent want to be a salaried employee (I realize there are valid exceptions, such as ER medicine)? I’d like to see the Rand Experiment addressed to this 15% of docs.

    *There are 16000 US medical graduates annually, but there are 24000 US slots for residency training, ref.. This means at least 8,000 residency program graduates annually that are not trained in US medical schools (note that the board exams are, for the most part, written). It also means that almost any US medical graduate that wants a residnecy can get one. Once in, you must do something really bad to get kicked out: someone’s got to do the wrok.

    *Harvard has the same pass rate on the internal medicine boards as Caritas Carney, a community-based Massachusetts program.

    *Top-ranked medical schools fall into two categories, Research and Primary Care, as noted here. There is only one top-10 research school that appears in the top 10 of primary care. Harvard isn’t one of them. Best overall ranking: University of Washington. Choose wisely.

    @frelkins: robot

    Yeah, we’ve had the DaVinci robot here in this little ol’ Southern town for several years.

  • Constantconstant

    you can’t learn procedural knowledge like flying an airplane or performing a quadruple bypass through the internet

    Even this may be possible in principle, with video games. There is still probably going to be a gap (e.g. without an expensive simulator the student isn’t going to feel the acceleration of the plane) but the gap may be small.

  • steven

    That looks like a pretty big confidence interval. Not finding an effect isn’t the same thing as finding the lack of an effect.

  • Thanatos Savehn

    I’m not a doctor, but I cross examine them all the time and I take a very dim view of the “art” part of their craft.

    When I started practicing doctors were still treated with great reverence. Under the law, if a doctor said it, it was admissible and could support any verdict – however absurd “it” was. In one of my very first trials, second chair (i.e. box carrier/gopher) I got to see an M.D. testify that a particular chemical made by our client was metabolized into a carcinogen and so caused the plaintiff’s cancer. I knew the testimony was coming and so had my Partner ready with Morrison & Boyd’s “Organic Chemistry” – the plan was to mount a theater of the absurd defense – demonstrating that the claimed reaction was highly unlikely since it would have required heating the plaintiff to 400 degrees and treating him with a platinum catalyst – a circumstance that should have proved fatal 25 years before – during his exposure. Instead the jury found in favor of the plaintiff – science be damned. Interviews with the jurors revealed their awe of doctors – doctors were to them modern day Gandalf’s blessed with a deeper knowledge than chemistry textbooks.

    The next 15 years were spent helping to pull the curtain aside and revealing the men behind it. The revelation that a not insubstantial number of doctors were just making it up as they went along followed. The profession never recovered. The “evidence-based medicine” movement has only exacerbated the problem.

    Now it’s pretty easy to narrow down your likely diagnosis and treatment before you go to the doctor. For example, my Dad was having problems with syncope recently. I googled his symptoms, past exams and treatments, and got him an appointment with the DeBakey heart folks here. I told my Dad the first thing they’d likely do would be a tilt table test and eventually recommend a dual chamber pacemaker unless they found something neurological. And danged if that’s not exactly what happened.

    I don’t mean to dis an entire profession because God knows there are lots of doctors doing one helluva job out there. But more and more it’s like a medieval guild whereby the work ain’t all that hard, it’s getting the plumbers’ license what’s the hard part.

  • Jor

    30,000 pts were randomized in an ER to one of two different hospitals and then subsequently transferred to one of those hospitals? HEH, for all the concern about the medical profession by economists, its funny how their journal articles have piss-poor figures and tables :P.

  • londenio

    Dear Thanatos,

    Metabolic enzymes can catalyze reactions that would otherwise need extreme conditions if performed in a container in a lab. I know it is not the main point you are trying to make, but I think it is important to mention this.

  • http://drchip.wordpress.com/ retired urologist

    @Tanatos (regarding the medical profession): more and more it’s like a medieval guild whereby the work ain’t all that hard, it’s getting the plumbers’ license what’s the hard part..

    We already know that doctors statistically suck, but just wait until Robin Hanson gets around to publishing his series on the marginal value of attorneys in the US economy.

  • Thanatos Savehn

    londenio: You’re quite right. In my case the plaintiff’s lawyer was trying to wire around the fact that IARC et al had found no evidence for carcinogenicity of the solvent in humans and only weak evidence in rats so his expert was testifying as to “plausibility”. The evidence that the plaintiff so metabolized the solvent was his cancer. Circular reasoning at its worst. After Daubert and its progeny, evidence that the solvent was not metabolized into the putative carcinogen put an end to the litigation but not until many millions had changed hands.

    retired urologist: I’ve been swimming in this cesspool a long time. Doctors are, or at least have been, net positives for society and the economy – and by a wide margin. Lawyers and law firms act either as protection rackets (think of criminal and divorce lawyers), guild agents and rent-seekers (corporate deal attorneys) or extortion rackets (litigation e.g. asbestos). I have seen things you cannot imagine. Fortunately, a little of it is finally beginning to come to light and some of the worst of the worst are heading off to prison. I see the profession as a net drag on the economy, a net corruptor of science and its perfection of sophistry a net negative for society – and by a wide margin.

  • http://www.bizop.ca/blog2/due-diligence/are-you-too-easily-impressed.html THE BIZOP NEWS

    Are You Too Easily Impressed?

    Image via Wikipedia Robin Hanson certainly thinks so.He has provocative post at Overcoming Bias about the relationship between expertise, credentials and rationality.”Yesterday I reported that top med…

  • http://profile.typekey.com/jhertzli/ Joseph Hertzlinger

    Let’s see. The top doctors don’t cure patients more often but they do save money.

    I suppose another NBER paper showed that top financial advisors don’t beat the market but they can perform better first aid.

  • David J. Balan

    Robin is right that the finding that second-tier docs take more time and do more tests may be because those docs’ lower status makes them more vulnerable to malpractice lawsuits. But this seems like quite a stretch. It seems much more plausible to me that the obvious interpretation is the correct one: the second-tier docs are somewhat inferior, but that inferiority takes the form of needing more time and tests to reach the same conclusion, rather than the form of reaching a different conclusion. That is, it sounds to me like the first tier docs really are better in some meaningful sense.

    Retired, a big part of the reason that the Medical College of Georgia is so much cheaper than Harvard is (I assume) that the former is a public school that is heavily subsidized by the state. It’s not anything like that much cheaper (though still probably a lot cheaper) when measured in the total cost per student.

  • Jeff Smith

    If I only have the following two pieces of info to eval a doctor, which is more important?

    Tier B med school, but Board Certified
    Tier A med school, but no Board Certification

    An interested layman