Comments on: Medical Study Biases http://www.overcomingbias.com/2007/02/medical_study_b.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Thu, 18 Mar 2010 19:46:22 -0400 http://wordpress.org/?v=2.8.5 hourly 1 By: Rick Davidson http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421589 Rick Davidson Sun, 04 Mar 2007 11:32:37 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421589 Nope, I know it doesn't happen on a large scale. It doesn't even happen in the academic medical center where I practice, at least much of the time. Just read the article quoted in the other EBM thread about Dan Merenstein if you have any doubts...but since my job is to convince medical students that they need to practice that way, all I can do is point them in the right direction. Introduce them to evidence-based clinical guidelines, teach them how to critically read and be skeptical...the fact is, right or wrong, that variability in practice may be disappearing for the wrong reason...through issues of managed care and cost control. But decreasing the variability of practice patterns, using the Dartmouth index effectively, and paying attention to current literature will improve the quality of medical care. It's the "one day" I'm aiming for...but I do believe that the situation has improved a great deal in the 25 years I've been teaching. In my course we use small group teaching for half the content, reading current articles. Among my small group teachers are the head of the lung transplant program, the division chief of GI, a former dean of the medical school, the department chair of family medicine, and assorted hematologists, rheumatologists, a urologist, and a medicine chief resident. Not one of these teachers have had formal training in EBM....they all are committed to critical reading of the literature and implementing it in their practice. That has an effect on students regarding the relevance of the content. I frequently have graduates come up and tell me that they have continued reading in their practices. I think things are changing. Nope, I know it doesn’t happen on a large scale. It doesn’t even happen in the academic medical center where I practice, at least much of the time. Just read the article quoted in the other EBM thread about Dan Merenstein if you have any doubts…but since my job is to convince medical students that they need to practice that way, all I can do is point them in the right direction. Introduce them to evidence-based clinical guidelines, teach them how to critically read and be skeptical…the fact is, right or wrong, that variability in practice may be disappearing for the wrong reason…through issues of managed care and cost control. But decreasing the variability of practice patterns, using the Dartmouth index effectively, and paying attention to current literature will improve the quality of medical care. It’s the “one day” I’m aiming for…but I do believe that the situation has improved a great deal in the 25 years I’ve been teaching. In my course we use small group teaching for half the content, reading current articles. Among my small group teachers are the head of the lung transplant program, the division chief of GI, a former dean of the medical school, the department chair of family medicine, and assorted hematologists, rheumatologists, a urologist, and a medicine chief resident. Not one of these teachers have had formal training in EBM….they all are committed to critical reading of the literature and implementing it in their practice. That has an effect on students regarding the relevance of the content. I frequently have graduates come up and tell me that they have continued reading in their practices. I think things are changing.

]]> By: david http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421588 david Sun, 04 Mar 2007 04:34:36 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421588 I'm sorry, were you laboring under the impression that the actual practice of medicine by physicians is based on science, evidence, and empirical thinking? That would be a revolution. In practice, what a doc does to determine how to treat a patient does not involve a hardcore look into the data or results of the clinical studies. I mean, there's some communication pipeline that tells docs how they should diagnose things and treat things under certain circumstances, but it is not a critical lens around the clinical trials. Did you really think that that happens on a large scale? Are you joking? Maybe one day. I’m sorry, were you laboring under the impression that the actual practice of medicine by physicians
is based on science, evidence, and empirical thinking? That would be a revolution. In practice,
what a doc does to determine how to treat a patient does not involve a hardcore look into the data
or results of the clinical studies. I mean, there’s some communication pipeline that tells docs how
they should diagnose things and treat things under certain circumstances, but it is not a critical
lens around the clinical trials. Did you really think that that happens on a large scale? Are you
joking? Maybe one day.

]]> By: Rick Davidson http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421587 Rick Davidson Sat, 03 Mar 2007 11:08:37 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421587 I suppose that if you could find a large enough number of non-randomized trials, or unblinded trials, you could construct some kind of summary "error"...and in fact, just like the financial support issue, the results would likely show a systematic error, as opposed to a random one. I remember studies done at least twenty years ago that showed these errors with controlled vs uncontrolled studies, and they ALWAYS show a greater benefit without controls. Interesting concept....not sure how feasible, but it would be interesting to try. I suppose that if you could find a large enough number of non-randomized trials, or unblinded trials, you could construct some kind of summary “error”…and in fact, just like the financial support issue, the results would likely show a systematic error, as opposed to a random one. I remember studies done at least twenty years ago that showed these errors with controlled vs uncontrolled studies, and they ALWAYS show a greater benefit without controls. Interesting concept….not sure how feasible, but it would be interesting to try.

]]> By: Robin Hanson http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421586 Robin Hanson Fri, 02 Mar 2007 13:54:14 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421586 Rick, your comments better address correlations of results with funding source than correlations of results with room for fudging, or the high rate of failure to replicate. Rick, your comments better address correlations of results with funding source than correlations of results with room for fudging, or the high rate of failure to replicate.

]]> By: Rick Davidson http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421585 Rick Davidson Fri, 02 Mar 2007 10:08:54 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421585 To the best of my knowledge, I published the first study to look at the association of industry funding with the outcome of published studies (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=3772583&query_hl=1&itool=pubmed_DocSum). I have spoken on this issue for years at workshops and academic health centers. From the practical standpoint, the best I can do is alert students to the significant risk of bias. The literature has improved since the mid '80's in terms of required acknowledgments and more careful attention to the problem. There are alternative explanations for the associations noted, other than purposeful (or unconcious) bias. Many drug studies involve drugs that have already been investigated in other countries and have been found to be effective. Studies that appear to be heading in the wrong direction may be discontinued by the company before the sample size is adequate to make the results publishable. There is always the possibility of publication bias that keeps negative studies out of the literature. Many investigators who undertake drug studies for companies have no particular interest in publishing the results. And, there are documented examples of investigators who were pressured by companies to not publish negative results. Trying to put a corrective factor on the amount of bias involved is an interesting suggestion....and could probably be determined for a large number of studies. The results, just like all grouped results, could not determine if an individual study is biased in this manner, but could provide an estimate for general use. To the best of my knowledge, I published the first study to look at the association of industry funding with the outcome of published studies (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=3772583&query_hl=1&itool=pubmed_DocSum). I have spoken on this issue for years at workshops and academic health centers. From the practical standpoint, the best I can do is alert students to the significant risk of bias. The literature has improved since the mid ’80’s in terms of required acknowledgments and more careful attention to the problem. There are alternative explanations for the associations noted, other than purposeful (or unconcious) bias. Many drug studies involve drugs that have already been investigated in other countries and have been found to be effective. Studies that appear to be heading in the wrong direction may be discontinued by the company before the sample size is adequate to make the results publishable. There is always the possibility of publication bias that keeps negative studies out of the literature. Many investigators who undertake drug studies for companies have no particular interest in publishing the results. And, there are documented examples of investigators who were pressured by companies to not publish negative results. Trying to put a corrective factor on the amount of bias involved is an interesting suggestion….and could probably be determined for a large number of studies. The results, just like all grouped results, could not determine if an individual study is biased in this manner, but could provide an estimate for general use.

]]> By: Douglas Knight http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421584 Douglas Knight Fri, 02 Mar 2007 02:58:47 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421584 <em>moved the significance level to 0.001</em> I think a suggestion like that is pretty much meaningless in isolation. The result of such a change would be heavily dependent on how it came about. Two failure modes are increased fraud and the cessation of medical research publication (and/or FDA approval). Done right, it could improve things. But simply convincing people that such significant studies on things people think they know would be an improvement, without touching the bad studies. moved the significance level to 0.001

I think a suggestion like that is pretty much meaningless in isolation. The result of such a change would be heavily dependent on how it came about. Two failure modes are increased fraud and the cessation of medical research publication (and/or FDA approval). Done right, it could improve things. But simply convincing people that such significant studies on things people think they know would be an improvement, without touching the bad studies.

]]> By: pdf23ds http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421583 pdf23ds Fri, 02 Mar 2007 01:56:57 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421583 "Bruce, for any biased estimate of anything, if you know something about the sign of the bias you should be able to correct it to produce a less biased estimate." You also have to have some idea about the magnitude of the bias. Otherwise you have no idea whether you're overcorrecting or not. “Bruce, for any biased estimate of anything, if you know something about the sign of the bias you should be able to correct it to produce a less biased estimate.”

You also have to have some idea about the magnitude of the bias. Otherwise you have no idea whether you’re overcorrecting or not.

]]> By: Doug S. http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421582 Doug S. Thu, 01 Mar 2007 17:59:57 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421582 Yeah, one in twenty medical studies will produce statistically significant results purely by chance. Yeah, one in twenty medical studies will produce statistically significant results purely by chance.

]]> By: dearieme http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421581 dearieme Thu, 01 Mar 2007 16:41:44 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421581 A friend who spent most of his career in science at Cambridge advocates the rule of thumb "Medical research is rubbish". A friend who spent most of his career in science at Cambridge advocates the rule of thumb “Medical research is rubbish”.

]]> By: Eliezer Yudkowsky http://www.overcomingbias.com/2007/02/medical_study_b.html#comment-421580 Eliezer Yudkowsky Thu, 01 Mar 2007 02:02:09 +0000 http://prod.ob.trike.com.au/2007/02/medical-study-biases.html#comment-421580 One gets the impression that an awful lot of sociological problems in science would go away if we moved the significance level to 0.001. Yes, I know it'd be just as arbitrary, but still. One gets the impression that an awful lot of sociological problems in science would go away if we moved the significance level to 0.001. Yes, I know it’d be just as arbitrary, but still.

]]>