Science ROI Hype

Years ago as a researcher at NASA Ames, I considered returning to grad school. Thinking about where I might study prediction markets as applied to academia, economics of science looked promising, especially as Paul David headed an econ of sci group nearby at Stanford. But reading the literature I got a bad feeling – authors seemed to be dishonestly trying to help research agencies justify funding. So I instead when to Caltech to do experimental econ, whose results I trusted more. My distrust is confirmed in a recent three page Nature article:

Spending on science is one of the best ways to generate jobs and economic growth, say research advocates. But … the evidence behind such claims is patchy.

The number one current rationale for extra research investment is that it will generate badly needed economic growth. … Heeding such arguments, governments in Germany, Sweden, Canada and Australia, as well as the United States, have increased research spending as part of stimulus packages …  Beneath the rhetoric, however, there is considerable unease that the economic benefits of science spending are being oversold. … The problem, economists say, is that the numbers attached to widely quoted economic benefits of research have been extrapolated from a small number of studies, many of which were undertaken with the explicit aim of building support for research investment, rather than being objective assessments. … “Too much of what has been done, has been done as a process of advocacy.” …

In one of the bedrock papers in this field, Edwin Mansfield, the late University of Pennsylvania economist, estimated that academic research delivered an annual rate of return of 28% (E. Mansfield Research Policy 20, 1–12; 1991). The figure has been widely quoted ever since. But Mansfield reached this estimate by interviewing chief executives, asking them what proportion of their companies’ innovation was derived from university research and, in effect, demanding that they come up with a number. “He was asking an impossible question.” …

Measuring the ROI from research has proved tough, and has produced a wide range of values (see table). Some … [ask] what contribution did a dozen neuroscience grants received by the University of Cambridge in 1972 eventually make to drug development? Such efforts are complicated, however, by the difficulties of attributing credit for any given drug to the numerous research teams involved over time. … “It is fair to say that this is an analytical dead end.” …

This [PR] influence derives in part from the activities of US medical research lobbyists. An example is the 2000 report Exceptional Returns: The Economic Value of America’s Investment in Medical Research by … Mary Woodard Lasker Charitable Trust that advocated biomedical research spending. … The document estimated that the steep decline in cardiovascular deaths in the United States between 1970 and 1990 has an economic value of $1.5 trillion annually, and deduced that one-third of this — $500 billion a year — could be attributed to medical research that led to new procedures and drugs. … Robert Topel, … whose work was cited in the report, distances himself from some of its claims. “Probably only a little of the fall in the cardiovascular death rate has to do with surgery and beta-blockers,” he says. …

Research agencies have no interest in assessing the costs of research fairly, says Barry Bozeman, a science-policy specialist. … “Honest clients are in short supply” for research in this field, he says. “Most of them think they already have the answers, and want someone to find the numbers to prove them right.”

GD Star Rating
Tagged as: , ,
Trackback URL:
  • Not to go off-topic, but:

    The document estimated that the steep decline in cardiovascular deaths in the United
    States between 1970 and 1990 has an economic value of $1.5 trillion annually, and deduced that
    one-third of this — $500 billion a year — could be attributed to medical research that led to new
    procedures and drugs

    What else might have caused it? (not rhetorical) Earlier detection? Changed behavior?

    • Just list a few:

      More people using a low does aspirin regiment
      The fitness movement
      Less stressful work
      Less smoking
      Fewer old people shoveling snow

  • It’s misleading to focus on deaths due to heart disease if we’re talking about the full costs of the burden — what about people who suffer from bad enough symptoms to have to periodically check into a hospital throughout life, but who don’t actually die from it right away?

    In fact, that has been going up since the 1970s, because of the massive switch in our diet away from animal products ca. the 1950s and ’60s and toward carb-filled junk that the government and experts scared us into eating.

    The American Heart Association doesn’t want you to see it because it would embarrass their field, but they do include estimates of how prevalent heart disease is — not just deaths — in their official reports. Here’s one whose data come from the National Hospital Discharge Survey:

    Those data are counts, but still by 2000 they are double the 1970 value, and population did not double in that time. So the rate has only gone up, not down.

    Read Gary Taubes’ sociology of science book Good Calories, Bad Calories on the mechanism linking more carbs in the diet to higher risk of heart disease. Search the index for Krauss, Ronald (a pioneering researcher). Or google around with the obvious keywords and you’ll find something.

  • Constant

    Issue seems really to be ROI from government funded research.

  • This post reminded me of a book I read a while back, The Economic Laws of Scientific Research by Terrence Kealey. He made a free market argument against government funding of science. I was wondering if you’ve read it, and if so what you thought of his argument.

  • Pingback: Tweets that mention Overcoming Bias : Science ROI Hype --

  • More medical pseudoscience Schneier on Security – Hot Dog Security

  • Proper Dave

    Isn’t science a public good? The ROI is very difficult to compute, that is why no one wants to fund it privately, except as philanthropy.
    That is why the it is government funded.

    I do think allot of the low hanging fruit have been picked though. So the ROI may credibly decline.
    The problem is that no developed economy can afford to not, and they can afford it because they are already rich.

  • Brian Moore wrote “What else might have caused it? (not rhetorical) Earlier detection? Changed behavior?”

    I expect you’ll get a more complete answer from Prof. Hanson presently, but off the top of my head let me add one seldom-mentioned cause to that list of candidate causes. Fogel’s _The Escape from Hunger and Premature Death_ makes a plausible-to-me case for well-nourished undiseased childhoods and early adulthoods causing substantial general health benefits throughout the rest of life. To crudely simplify his thesis into concrete terms, if a subpopulation is significantly taller than the previous generation of that subpopulation, then it’s unsurprising that its death rate from many problems (specifically including heart attack IIRC) is significantly lower.

  • What makes this so frustrating is that massive numbers are not obviously wrong.

    It really looks like there is a relationship between science, basic science, and Total Factor Productivity growth.

    I suppose it could be overwhelmingly incrementalism. Small firm based improvements adding up over time.

    I just have no idea how we would know.

  • tonyf

    Brian Moore: What else might have changed it? — Yes, changed behaviour. Probably to a large extent much fewer smokers during the last half-century. (Although, say, 1960-1980 is maybe the more relevant for deaths 1970-1990. Probably fewer smokers also for that period, but I won’t check it.) And then the reason for that is medical research that showed the health hazards of smoking. Clearly showing that research is worth its cost also purely economically, and the large advantages brought to health by scientific medicin.

  • Jim Purdy

    “Most of them think they already have the answers, and want someone to find the numbers to prove them right.”

    Scientists, Faux News viewers, Tea Partiers, creationists … they all just want confirmation for their biases, not objectivity. It’s just human nature.

    Jim Purdy

  • Owen

    I showed this to someone in the evaluation field, and his comments were:
    “I’m bemused by the reported comments from Buxton, Martin and Grant as these are the main guys doing ROI-ish reports on research:

    Buxton is the creator of the “Buxton-Hanney Payback Model” which is purely an accountablilty effort; Martin runs with the SPRU crowd and produces such gems as “The economic benefits of publicly funded basic research: a critical review.”, and Grant’s major report in recent history was for the Arthritis Research Campaign UK, “Reviewing the returns of research: Capturing payback from funding by the Arthritis Research Campaign”.

    They each consult/contract to produce these kinds of reports, so I’m at a loss to explain them dissing them.

    ‘d add that it’s the very difficulty in determining direct economic benefits from research that make it difficult for industry to fund it; not actually an argument to stop public funding as some have taken it.”

  • Pingback: Cobb()