Flu Shot Doubts

Years ago my wife and I and our two young kids were visiting our families.  The others came down with the flu, and lay about in misery in the hotel room for days.  But I had had a flu shot, and stayed well, which convinced me to get a flu shot every year since.  While I’m generally a med skeptic, I’ve figured we can see at least a few identifiable areas where med seems most worth the cost, including infant care, trauma care, and vaccines.   So even I’m surprised to see just how weak is the case for flu shots.  Brownlee in the Atlantic:

Some top flu researchers are deeply skeptical of both flu vaccines and antivirals. Like the engineers who warned for years about the levees of New Orleans, these experts caution that our defenses may be flawed, and quite possibly useless against a truly lethal flu. …

We think we have the flu anytime we fall ill with an ailment that brings on headache, malaise, fever, coughing, sneezing, and that achy feeling …  but researchers have found that at most half, and perhaps as few as 7 or 8 percent, of such cases are actually caused by an influenza virus in any given year. …

Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did. .. [Thus] the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all.

The results were also so unexpected that many experts simply refused to believe them. Jackson’s papers were turned down for publication in the top-ranked medical journals. … When the papers were finally published in 2006, … they were largely ignored by doctors and public-health officials. …

The history of flu vaccination suggests other reasons to doubt claims that it dramatically reduces mortality. In 2004, for example, vaccine production fell behind, causing a 40 percent drop in immunization rates. Yet mortality did not rise. In addition, vaccine “mismatches” occurred in 1968 and 1997: in both years, the vaccine that had been produced in the summer protected against one set of viruses, but come winter, a different set was circulating. In effect, nobody was vaccinated. Yet death rates from all causes, including flu and the various illnesses it can exacerbate, did not budge. …

Rising rates of vaccination of the elderly over the past two decades have not coincided with a lower overall mortality rate. In 1989, only 15 percent of people over age 65 in the U.S. and Canada were vaccinated against flu. Today, more than 65 percent are immunized. Yet death rates among the elderly during flu season have increased rather than decreased. ….

Jefferson … [has] combed through hundreds of flu-vaccine studies. … Only four studies were properly designed to pin down the effectiveness of flu vaccine, he says, and two of those showed that it might be effective in certain groups of patients, such as school-age children with no underlying health issues like asthma. The other two showed equivocal results or no benefit. …

In the flu-vaccine world, Jefferson’s call for placebo-controlled studies is considered so radical that even some of his fellow skeptics oppose it. … Everybody … feel[s] strongly that vaccine has been shown to be effective and that a sham vaccine would put test subjects at unnecessary risk of getting a serious case of the flu. …

[But] the annals of medicine are littered with treatments and tests that became medical doctrine on the slimmest of evidence, and were then declared sacrosanct and beyond scientific investigation. In the 1980s and ’90s, for example, cancer specialists were convinced that high-dose chemotherapy followed by a bone-marrow transplant was the best hope for women with advanced breast cancer, and many refused to enroll their patients in randomized clinical trials that were designed to test transplants against the standard—and far less toxic—therapy. The trials, they said, were unethical, because they knew transplants worked. When the studies were concluded, in 1999 and 2000, it turned out that bone-marrow transplants were killing patients.

Another recent example involves drugs related to the analgesic lidocaine. In the 1970s, doctors noticed that the drugs seemed to make the heart beat rhythmically, and they began prescribing them to patients suffering from irregular heartbeats, assuming that restoring a proper rhythm would reduce the patient’s risk of dying. Prominent cardiologists for years opposed clinical trials of the drugs, saying it would be medical malpractice to withhold them from patients in a control group. The drugs were widely used for two decades, until a government-sponsored study showed in 1989 that patients who were prescribed the medicine were three and a half times as likely to die as those given a placebo. …

As it stands, more than 50 percent of health-care workers say they do not intend to get vaccinated for swine flu and don’t routinely get their shots for seasonal flu, in part because many of them doubt the vaccines’ efficacy. … In the U.S. … our reliance on vaccination may have the opposite effect: breeding feelings of invulnerability, and leading some people to ignore simple measures like better-than-normal hygiene, staying away from those who are sick, and staying home when they feel ill.

I’ll join Jackson in calling for randomized placebo-controlled trials of flu shots. More from Jackson:

Seasonal influenza is a relatively rare and benign condition, with an incidence not exceeding 1% in the general population during autumn and winter months.

Hat tip to Carl Shulman.

GD Star Rating
Tagged as: ,
Trackback URL:
  • In the flu-vaccine world, Jefferson’s call for placebo-controlled studies is considered so radical that even some of his fellow skeptics oppose it. … Everybody … feel[s] strongly that vaccine has been shown to be effective and that a sham vaccine would put test subjects at unnecessary risk of getting a serious case of the flu.

    Pascal’s Wager, anyone?

    • Jake

      In drawing the analogy to Pascal’s Wager, I suspect you are making one of the same errors as Pascal: in this case, assuming that the risks associated with opting for vaccination are zero or at least negligible. There is little basis for this assumption.

      I had naively assumed that the efficacy of flu vaccination has been backed up by clinical trials for some time. I am very surprised to learn that this is not the case.

  • Rob
  • Joborg

    For the establishment view, see e.g.,
    commenting directly on Jefferson:

    Vaccines for seasonal flu have at least 50% likelihood of preventing illness given exposure. Arguing that exposure is unlikely or that the flu is not a serious illness is a different issue.

    Pandemic flu is an entirely different problem.

    • I don’t doubt that specific narrow studies have found some positive results, but there seems to be a huge disconnect between those and the overall outcomes we most care about. None of those critiques seem to question the claim that mortality didn’t change in years when flu shots were ineffective.

      • I’ve read (and I can’t find the source immediately, but probably someone on SBM) that even a poorly matched flu vaccine can have some efficacy against total mortality. The misconception is that the vaccine operate sas a binary switch (protected from flu true/false). Instead, it is a very analog education of the immune system. It’s plausible that a poorly matched shot can still help enough that you don’t end up with as many severe complications.

        Jefferson makes a good point that there could be confounding bias in the non-blinded trials. But this is NOT the same thing as appealing to non-blinded trials of (say) acupuncture. One difference is that it’s very difficult to design a perfect study here (at least, given constraints of ethics). Although as Mark Crislip notes, there have been some good ones that show efficacy.

        So, we can’t yet demonstrate to our satisfaction that flu vaccines definitely work. Our best guess so far is that they work, but not perfectly. But we have shown they are definitely safe. Given that there are lives at stake, I claim we should proceed as best we can, while still trying to think up better ways to control for bias in future studies.

        I think this is one of the two prominent differences between Evidence Based Medicine and Science Based Medicine: In SBM, you are allowed to do what the best available science reasonably suggests is efficacious, as long as it’s safe. But you should still be humble and clever and do everything you can to rule out your bias.

        (For completeness: the other difference is that EBM can be made to show efficacy for unscientific things, by subtly assuming them to be valid in the formulation of the study. For example if you assume acupuncture can have an effect on pain, you can use EBM to quantify that effect. But you have implicitly facepalmed because your control is too different to rule out bias and placebo effects)

  • Pingback: Untaken Road » Blog Archive » Flu Shot Doubts()

  • Bill

    If you are vacinated and have an exposure to the virus and are later exposed to me, the benefit of your vacination is to me as well, even if I do not get vacinated.

    I would think that you could model the optimal size of population that should be vacinated, given that there may be some risks of vacination. Should the non-vacinated pay the vacinated for their decision which benefits them.

    • The effect you are claiming here would increase the benefit of the vaccine. The article is claiming there is no visible benefit. So you are basically not only claiming the article is wrong, you are claiming that the exact opposite is true, that the benefit is even greater, rather than missing. Where’s your evidence?


        Am I missing something? Why do you assume that this comment is anti-vaccine??

      • The comment is pro-vaccine, but the original post says there is no evidence of benefit from the flu vaccines. I was requesting evidence for his pro-vaccine claims.

      • Bill

        I am pro-vaccine based on evidence of vaccination limiting transmissability, but, like everything else, there is a balancing of risks to benefits. The question was posed as determining the optimal degree of vaccination, given risks and benefits. I read the Atlantic article, but also the other ones critical of it and their conclusions:

        More important, though, vaccines are hardly ever 100% effective, and this is especially true of flu vaccine. You can get vaccinated and still get the flu, or get infected, or be contagious, or whatever the endpoint is. It is not a guarantee for an individual. But what we know from the public health standpoint, where the measure is the population, the flu vaccine has a major effect on influenza in the community. We know this from numerous studies that have been done demonstrating it. I’ve hardly scratched the surface of this business. If you want to pursue it you could start with the Basta, Halloran et al. paper in the American Journal of Epidemiology (Am J Epidemiol. 2008 Dec 15;168(12):1343-52) and move on to look at individual studies.

  • Mark

    Unfortunately, this post has some factual shortcomings.

    I would suggest getting information about vaccines (and medicine) from a better source, such as http://www.sciencebasedmedicine.org. A good starting point would be “http://www.sciencebasedmedicine.org/?p=2040&cpage=1” which talks about flu vaccine efficiency. On a good day a good reporter can mangle the excellent research of a great researcher. A reporter with little knowledge of the field reporting on less than excellent research from less than great researchers….
    Second, as the link will note, trials have been done and ample evidence for efficacy exists. It isn’t as good a vaccine as we would like (mostly due to the nature of the flu itself) but it works.
    Finally, a 1% incidence in a population of 300 million people is 3 million. While that may be “rare”, it is also a large number of people. Based on that definition, death is also rare…. It also should be noted that the Centers for Disease Control estimate a 5 to 20% incidence of seasonal flu, rather greater than Jackson’s numbers….

    • It is easy to understand why we see no effect on aggregate mortality if the incidence is <1%, but harder to see if it is 5-20%. At a <1% level it doesn't seem worth it for me to get a flu shot, especially given that it seems less effective for those with a higher mortality risk. So this is valuable news to me, news that wasn't made clear in the sources you prefer.

      • Doug S.

        I thought the primary benefit of getting a seasonal flu shot is avoiding the inconvenience and unpleasant experience of having the flu, not avoiding a flu-related death. (I imagine that the flu is more likely to kill someone that has a compromised immune system, and that vaccines are less effective for people with compromised immune systems…)

      • Mike7106

        Compromised immune system. A term that was relatively unheard of before the vaccine tsunami.

      • Ronfar

        Well, it wasn’t a term that would apply to many people before organ transplants gave people a reason to take immunosuppressant drugs and before HIV started spreading.

      • mattmc

        I find this statistic, presented barely, to be shockingly high.
        “Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did.”

        I wonder if confining his research to seniors had any effect, I have a very high suspicion of sample bias here.

    • In the BMJ paper that Robin linked to, Jackson describes in detail how he arrived at his 1% incidence number, and it seems pretty convincing. Does anyone know how the CDC computed the 5 to 20% figure?

      Robin, I wonder why it doesn’t seem worth it to you to get a flu shot at the 1% level. Is it mainly the time and hassle involved? If so, would you get a shot if that cost could be minimized, for example if there were traveling vaccination teams that went to workplaces to vaccinate anyone who wanted a shot?

      • It takes me 1.5 hours to get a flu shot, but having the flu costs me less than 150 hours in lost time.

      • If time cost is the main consideration, why not call for making flu shots cheaper (in time)? It seems like we have enough evidence that the flu shot does protect against the influenza virus, but the virus isn’t common enough to make it clearly worth the cost.

        BTW, in the benefits column, I think you forgot to account for:

        eliminating “pain and suffering” caused by the flu
        not transmitting the flu to your family and others
        decreased probability of contracting other diseases, such as pneumonia
        decreased probability of death

      • I call in general for costs to be lower, but not below prices.

  • Constant

    The easiest way to test a vaccine is to vaccinate a volunteer healthy subject and then deliberately expose him to the virus. My company did that for a drug it was testing as a possible prophylactic or treatment for malaria (it was actually the military that tested the drug, and it wasn’t a vaccine). A cup with malaria-infested flies was placed against the soldier’s skin until he was definitely exposed. The drug didn’t work.

    The study was not randomized, was not blinded, did not have a placebo arm. It didn’t matter, because it’s hard to mistake malaria with phantom malaria, hard to mistake an actual malaria cure with the mere placebo-induced psychological feeling that one is cured.

    You don’t always need placebos, or randomization, or blinding.

    • Constant

      I mean malaria-infested mosquitoes.

    • Constant

      “The easiest way to test a vaccine” – sorry, I said too much. I meant only that I would imagine that one could test a vaccine this way. My example is only analogous – malaria is not a virus and the drug was not a vaccine.

    • Alan

      Constant – this kind of direct exposure study has been done for flu – the results are compelling in that protection was good (60-80% if I recall, higher if you included only those who actually responded to vaccine), provided the exposure test strain matched the vaccine strain. In years where there is a vaccine mismatch, the shots are probably less effective. However, there may be residual benefit due to low-level cross-protection across strains.
      Also, there may be benefit in the young by reducing asymptomatic carriage and thus reducing spread to the elderly and other death-susceptible populations, however the relatively low rates of flu vaccination make this hard to study, since you don’t expect to see herd immunity effects until vaccination rates are very high.
      You also have to remember that much of the data measures “influenza-like illness” and not flu per se. It is unsurprising that vaccination rates/effectiveness have little effect upon influenza-like illness, since they only protect against the true ‘flu.

      Also 1% of the population is a huge number of people, especially for something that happens *every year*. I don’t think that a rate of 1% a year is “relatively rare”, I would say that that is pretty high. For comparison HIV incidence is estimated at 0.02% in the USA (my own envelope calc from CDC and census stats).

    • Eric Johnson

      Actually, if you are going to do a Geimsa smear or a PCR or whatever to examine for malaria, it would be an essential practice to have the technician blinded in order to avoid bias. It’s true that the patient and physician don’t need to be blinded, unless you fear that social interactions could influence the immune response to malaria. This is an unlikely theory but it is not absolutely out of the question. The nice thing about a blinded controlled trial, especially if you are trying to produce authoritative info rather than an intriguing pilot study, is that it takes care of virtually /every/ epistemological problem – it is extremely clean epistemologically. The only big worry I know of (this is not my area of specialization) is unblinding – via side effects, etc.

  • mjgeddes

    And more generally, how many other popular health fads don’t stack up? This is an extract from an article that appeared in my local newspaper over the weeked:

    ‘Taking vitamin supplements can be as danagerous as passive smoking, an Auckland medical school professor says (Head of epistemology and biostatics, Auckland University)

    A major analysis in 2008 of a large number of trials into the benefits and dangers of vitamin supplements showed that when taken by people with no diagnosed deficiencies, vitamin A increases the risk of premature death by 16%, beta-carotence by 7%, vitamin E by 4%. Studies into vitamin C were not large enough to be conclusive, but appear to show a 6% increase in risk nof premature death. The analysis was performed by the esteemed medical analysis organization the Cochrane Collaboration’

    Let this be a lesson to all budding ‘immortalists’ and irrational exuberants hoping to live to 1000 by vitamin popping: This is what happens when you deviate from strict rationality and start making stuff up rather than following evidence: the universe will bite your head off and you will get horribly burned every single time.

    • Doug S.

      I think I’ve heard that kind of thing mostly applies to “megavitamin” supplements, those that contain a much bigger dose than people would ever get by eating food. A standard “one a day” vitamin shouldn’t be enough to overdose on, but I could be wrong about this.

      Also, note that Vitamin D was not part of this meta-analysis, is not an antioxidant, and low blood levels of it are associated with an increased risk of death.

  • The problems with the Atlantic article are covered extensively here: http://scienceblogs.com/effectmeasure/2009/10/journalists_sink_in_the_atlant.php . In short, the article exacerbates legitimate concerns and doubts to an extent that those concerns are no longer legitimate.

    Re-posting this is spreading something that’s uncomfortable close to FUD. If you’re going to feature a long blockquote from it, you ought to also put the article above on the front page.

    • You are the third commenter to point to that same blog post as if it had some definitive treatment. But it doesn’t address what to me is the key issue: the lack of a noticeable net mortality effect.

      • lemmy caution

        A key distinction is between seasonal and pandemic influenza. There may be no noticeable net mortality effect for seasonal flu shots (assuming the flu doesn’t kill youngsters and the shot is ineffective for oldsters). This does not mean that shots won’t help in a pandemic in which youngsters die more at much higher rates than by seasonal flu.


      • But it doesn’t address what to me is the key issue: the lack of a noticeable net mortality effect.

        True, but I’m not convinced there is a key issue in the flu vaccine question: there are a suite of issues, which are unfairly jumbled together in the Atlantic article.

  • Michael Turner

    “In addition, vaccine ‘mismatches’ occurred in 1968 and 1997: in both years, the vaccine that had been produced in the summer protected against one set of viruses, but come winter, a different set was circulating. In effect, nobody was vaccinated. Yet death rates from all causes, including flu and the various illnesses it can exacerbate, did not budge….”

    Some variables that are going unaccounted for, here: degree of lethality, and possible dependence of that degree of lethality on time-to-mutate. In both cases, time-to-mutate (summer-to-winter) was low — perhaps the main change in the viral mix was simply in its increased resistance to antibodies created by the vaccination program. In any case, degree of lethality varies dramatically, for influenza.

    We don’t exactly have a slam-dunk sample size, with two years separated by almost three decades. Concluding anything from this might be rather like concluding that, since a few turn in Russian roulette hadn’t killed you yet, the risk of continuing the game must be zero.

  • John Smoltz

    Hi Robin,

    I’m just wondering what you mean by the term “med skeptic”? Would you care to elaborate? What exactly are you skeptical about?


    • Robin is a “med skeptic” in the sense that he recognizes that spending on medicine has (in aggregate) a stunningly weak correlation to improved health. Robin and several health policy experts discussed this in Cato Unbound, and I’m sure you can find some posts at OB on the topic too.

  • retired phlebotomist


    I hope some day you’ll write a popular book. While it’s no doubt higher status to flesh out the impossibility of disagreement in a tome made to molder in academic libraries and LW crowd bookshelves, you have some fascinating ideas that should get out there.

  • I enjoy reading overcomingbias.com and lesswrong.com, they are indeed my favorite resources when it comes to food for thought. I’m not very educated and so I tend not to participate actively.

    I live in Germany where we are able to be vaccinated against swine flu next week with Pandemrix from GlaxoSmithKline which contains the immunologic adjuvant AS03. After reading quite a few article about vaccination over the past few months on scienceblogs.com and elsewhere I came to the conclusion that it might be rational to be vaccinated myself. Though today I came across this post.

    Now I’m simply not sure anymore. Some smart people seem to be in favor of it. But on the other hand, most health-care workers here in Germany are not going to be vaccinated. So how could someone like me know what to do? This topic is loaded with misinformation. And for me it is almost impossible to verify and evaluate all evidence before it is too late.

    What would you people suggest? Should I be vaccinated or not? I’d love to hear what you think.


    Get the H1N1 Vaccine When It is Available For You http://ff.im/-aiYuz
    Why you should get vaccinated against the seasonal flu anyway, despite H1N1 http://ff.im/-8hOTv
    Seasonal flu: Why I got vaccinated http://scienceblogs.com/effectmeasure/2009/09/seasonal_flu_why_i_got_vaccina.php

  • gwern

    Alexander: I wouldn’t. If much of Germany is getting vaccinated, you can just be a free-rider. I have heard that it’s safer to be unvaccinated, protected by herd immunity, than to be vaccinated without herd immunity.

  • mattmc

    The aggregate studies presented here would seem to be less accurate than the regular testing of vaccines that goes on. Simple tests where people are exposed to the virus, with the vaccine at varying levels, are given every year to determine effective dosing. It works!

    Using these aggregate numbers it’s just too hard to figure out what is causing the correlations.

  • This was the key sentence in the Atlantic article for me:

    “Studies show that young, healthy people mount a glorious immune response to seasonal flu vaccine, and their response reduces their chances of getting the flu and may lessen the severity of symptoms if they do get it.”

    I HATE having the flu; if the cost is 1.5 hours or my time but the benefit is a 1% chance of avoiding five days of being miserable in bed, then that’s worth it to me. I’d give it about a three-to-one ratio– I’d rather spend three hours waiting in the doctor’s office than one hour being miserable and feverish in bed.

    Also, I have two school-aged children, so I bet my chances of getting the flu are higher than the overall 1% incidence figure.

  • Just A Public Health Worker

    I think there might be a disconnect here between what positive effects we should be looking for, and how they are measured. I do not work in the field of flu vaccination, but I have enough epidemiology education that I can tell that the main good points of a vaccine are not being articulated by the CDC, etc.

    While a vaccine does theoretically protect an individual, one of the major reasons behind vaccination campaigns is to prevent or slow *an epidemic* — or the spread of a disease through a group, instead of preventing one person from getting sick. This is a little like voting, because part of the point of democracy is to keep crazy despots from coming into power and wrecking the country, instead of specifically protecting one person’s interests.

    The economists will tell you that, in a country of hundreds of millions of people, voting isn’t worth the time or expense on an individual level, but if not enough people vote (or vote the “correct” way, whichever way you think that is), we could end up with someone running the country who can destroy it.

    Similarly, when enough people get vaccinated against the flu, it should slow down the spread of disease enough that, yes, perhaps fewer people get the disease, or it just spreads more slowly. When the flu spreads more slowly, that should decrease 1)the number of opportunities the flu has to mutate into a more deadly flu at any one time, or 2)the chances that so many people will be sick at one time that healthcare facilities are overwhelmed or basic services (like police) cannot be fully staffed because too many people are sick.

    We are very lucky today to not understand a deadly epidemic. The flu epidemic in 1918 didn’t just kill thousands of Americans–it brought normal life to a standstill, including much economic activity. Other, far deadlier epidemics, such as yellow fever and bubonic plague, resulted not only in unprecedented numbers of individual deaths, but also in panic and disorder that brought down cities. I do not say this to scare people (don’t be scared!) but to try to articulate that a true epidemic is not just about sickness but about preventing chaos that can lead to economic and physical safety problems. It is far easier to prevent people from getting a disease through vaccination than it is to prevent people from panicking when a disease hits.

    While we haven’t had a true flu epidemic in many decades, I don’t think we’ve ever gotten enough people (or the right people–that would be children, food service workers, healthcare providers, etc) vaccinated to actually prevent a flu season, and that has translated to what (I believe) the CDC has estimated as 30,000 annual flu-related deaths a year, mostly among elderly people.

    But these costs and benefits are difficult to measure. Since we haven’t had an epidemic in years, we don’t see the possibilities that vaccination could be preventing–it’s like how we don’t appreciate how much we need clean water, because we are not routinely sick from not having it. Because we don’t have two identical Earths hanging out, we can’t tell if our vaccination campaigns slowed down the development of a new flu pandemic, by preventing many chances for different strains to mutate or come in contact with one another. (This is important, not just in delaying a bad event, but in allowing people to develop better Flu treatments. Anti-viral medications have come a long way in the past twenty years.) Finally, those thirty thousand estimated flu deaths may not show up on death certificates, where cause of death will often be the complications of flu, like pneumonia or dehydration.

    I completely support people’s rights to refuse vaccination, and I hope that I’ve provided a coherent explanation of the public health reasons for flu vaccination, whether you agree with me or not 🙂

  • Max

    Mark Crislip responded to the Atlantic article point-by-point.

    A randomized placebo-controlled trial shouldn’t be unethical.
    They did it to test polio vaccine.
    “In placebo control areas, where vaccine was interchanged with an inert substance, 428 out of 749,236 children contracted the disease.”

    They’re doing it to test malaria vaccine on children.
    “The latest study will include two age groups: infants and children five to 17 months old. Half of each group will get the vaccine and half a placebo… Then they will be followed over two years for signs of clinical malaria.”

    Suppose the vaccine is 60% effective, but there’s a 50% chance of getting the placebo. That’s equivalent to getting a vaccine that’s 30% effective, which is better than nothing.