Radical Research Evaluation

The NIH says it wants to hear, by August 17, radical suggestions for changing how they evaluate research: 

The NIH is seeking comments regarding NIH’s support of the biomedical and behavioral research, including peer review, with the goal of examining the current system to optimize its efficiency and effectiveness. The NIH is especially interested in creative suggestions, even if they involve radical changes to the current approach. … [to deal with] an increased load on the peer review system resulting from a steady rise in applications and the increased complexity of biomedical and behavioral science. Ultimately, NIH wants to ensure that the most meritorious science is supported while minimizing bureaucratic burden on applicants and the NIH itself.

I’d like to be hopeful, but the recent NSF experience leaves me jaded.  From Science in January:

In 2005, Representative Frank Wolf (R-VA), then chair of the spending panel that sets NSF’s budget, asked the agency to think about offering megaprizes for solutions to important scientific challenges with societal implications. At one hearing, for example, Wolf speculated about how a $1 billion prize could yield an elegant technological solution to the nation’s dependence on foreign oil. Last summer, NSF officials turned for guidance to the National Academies, which this week issued a 44-page report urging NSF to adopt "an experimental approach" by piloting a handful of small prizes before considering spending big bucks. …

It mentioned areas such as nano self-assembly, green chemistry, low-carbon energy technologies, and teaching software. The report suggests that NSF should start with a handful of prizes ranging from $200,000 to $2 million and raise the payout to as much as $30 million if the concept proves successful. Myers says Congress should increase NSF’s budget to make room for the program so that its cost doesn’t undermine ongoing activities. 

The report recommends starting the prizes this year, but with NSF facing a budget freeze … that’s not likely to happen. " … An even bigger obstacle, however, may be the recent changes on Capitol Hill. Wolf is no longer on the spending subpanel, and House Republicans are in the minority. A Democratic aide … says that prizes probably make more sense at a mission-oriented agency such as NASA or the Defense Department. And the aide questions the idea that a prize will attract new players into the research game.

Translation: NSF pulled the usual agency trick of stalling until Congressional pressure faded. and those who get the usual money the usual way made sure the official recommendation was for no change in that; only a little money, and only new money, should go to new methods.

I’d love for the NIH to get an avalanche of recommendations to try more accomplishment prizes and information prizes (tied to prediction markets), but honestly I doubt it would make much difference.  Hat tip to Rafe Furst.

GD Star Rating
Tagged as:
Trackback URL:
  • Hopefully Anonymous

    I think what help may be naming specific decision makers and holding them accountable. These aren’t bureacracies of nameless individuals, although they may like to present themselves that way.

  • Nathan Iver O’Sullivan

    It has always seemed to me that a prize system would require reams of rules and regulations about who gets the money when the prize’s conditions are met. Great scientific breakthroughs are rarely accomplished by one man or a few researchers alone; science is a collaborative process, and breakthroughs often involve tens or even hundreds of researchers (some more important than others). Moreover, if the rules were written in such a way that collaborators earned a fraction of the prize, researchers would have an incentive to work alone, which is probably inefficient.

  • Nathan Iver O’Sullivan

    Also, Robin, are you planning to submit any comments or proposals? If so, maybe you could post them on the blog.

  • Hopefully Anonymous

    Alternatively it may enhance efficiency, since researchers would be motivated to work with the minimum sized team necessary to accomplish the research goal.

  • Plinius

    But what if biomedical research needs funding before the discoveries are made rather than after?

  • Hopefully Anonymous

    I didn’t see anyone suggest a complete replacement of grants with prizes, but for the sake of argument, promising research teams could secure loans or incorporate and sell equity stakes of their incorporation in the anticipation of future prize money earnings. Practically, I suspect prize money is a good supplement to other ways of funding, rewarding, and incentivizing the best research, including the best “basic” research.

  • Carl Shulman


    The NIH spends $28 billion per year. The genomics X-prize is $10 million, and a winning team will have delivered a major medical advance. It seems highly improbable that having well under 1% of federal funding actually be performance-based prizes is optimal.

    On the issue of not having funds to get research started, prizes don’t have to be enormous. Imagine the NIH allocating a fifth of its budget to endowing 5600 new $1 million prizes each year. The volume is sufficiently large that a medical school, biology department, or profit-making biotechnology company could compete for many prizes simultaneously, diversifying its research portfolio to earn a fairly steady stream of revenues, but with new incentives to conduct cost-*effective* research.

  • It’s hard to see what motive any bureaucrat has for putting real change in place, when they can look sexy and edgy and dangerous just by calling for dangerous proposals. It’s actually putting money in place that would involve the career risk – and that, of course, can be left out at little cost to reputation.

  • dearieme

    Put 5% of funds aside to be allocated by lot. Every applicant is appraised every (say) 5 years to confirm that he’s qualified to be an entrant, and then chance rules.

  • Hopefully Anonymous

    Great point and interesting idea.

  • Carl,

    I see your point about the biotechnology company, but would academic institutes have enough resources to put the money up in advance? My understanding was that university scientists practically always need outside funding for empirical research projects.

    I also don’t think venture capitalists would be willing to put up lots of money for the simple reason that they cannot tell how big of a challenge the reasearchers would face.

  • Carl Shulman


    “but would academic institutes have enough resources to put the money up in advance? My understanding was that university scientists practically always need outside funding for empirical research projects.”

    They need to get outside funding in substantial part because ordinarily such research is a cost to the university, so other constituencies eager for budget prevent such allocation. The situation is different for a profit center: universities invest billions in alternative investments, from hedge funds to timber lots. Investing in efficient research aimed at winning (abundant) prizes would be a high-return investment, as prize money (unlike grants) could be channeled to fund other activities.

  • joe

    Unfortunately it could also be a high-risk investment if you have many institutions vying for the same prize, and there is no guarantee that anyone will find an answer in a reasonable amount of time if ever. For many institutions, a consistent yearly return on an investment might be preferred.

  • Hopefully Anonymous

    Joe, I think you’re describing a problem with poor incentive prize construction, rather than an inherent problem with substantial funding of research through prize incentives. You’re pointing out the importance sufficient # of prizes and size of prizes, and pegging the prizes to reasonably achievable goals, all important. The MPrize seems to have moved strongly in that direction, and I imagine this whole area (encompassing how salary and bonus incentives work) has been significantly studied by economists and management researchers.

  • Pingback: Overcoming Bias : Bungling Bounties()