Changes to NIH Reviews: new scoring procedures, new review criteria, and shorter applications.

The NIH has released the next three stages of their plan to improve application review: new scoring procedures for evaluation, changes to the review criteria, and a new timeline for implementing shorter applications.

New Scoring Procedures

As of May 2009 review meetings, reviewers will be using a 9-point, whole-number rating system (1 excellent, 9 poor). Applications will receive a preliminary score to determine which ones will be discussed during the review session; however, all applicants will receive their scores, even if their applications are not discussed. Individual scores will be given for all five core review criteria. Thus, the old ‘triage’ system will change so that no one will go ‘unscored.’ This will be useful information, since applicants will have only one chance to resubmit, and the scores should direct that resubmission. The NIH believes that there will be more chances for tied scores under this system, and therefore, reviewers will consider factors such as mission relevance or portfolio balance when impact alone isn’t enough to decisively determine which applications should be prioritized. Percentile rankings will also change. Perhaps the greatest change is the use of an ‘impact’ score to determine which applications are discussed–here is what NIH is saying about the major changes:

Priority Scores – Discussed Applications. Before the review meeting, each reviewer and discussant assigned to an application will give a preliminary impact score for that application. The preliminary impact scores will be used to determine which applications will be discussed.  For each application that is discussed, a final impact score will be given by each eligible committee member (without conflicts of interest).  Each member’s impact score will reflect his/her evaluation of the overall impact that the project is likely to have on the research field(s) involved, rather than a weighted average applied to the reviewer’s scores given to each criterion.

The overall impact score for each discussed application will be determined by calculating the mean score from all the eligible members’ impact scores, and multiplying the average by 10; the overall impact score will be reported on the summary statement.  Thus, the 81 possible overall impact scores will range from 10 – 90. (Overall impact scores will not be reported for applications that are not discussed.)

Other Review Criteria Changes

Beginning January 2009, review criteria will be changed–or at least the language describing them. The five core criteria–significance, investigators, innovation, approach, and environment–have been clarified.  For a side-by-side comparison of the old and new criteria, visit: http://grants.nih.gov/grants/peer/side_by_side_comparison.doc. The major change is a focus on the overall impact of your research–a focus that will be eventually reflected in the structure of the shorter application.

For more about the review changes in general, visit here.

Updated Timeline

Shorter applications will take effect beginning with applications due in January 2010 (for the 2011 fiscal year). The details of these applications are only now emerging, but we will keep you up to date as we learn more. The new timeline is as follows:

For applications due in January 2009 (for potential FY2010 funding), the following already-announced policies will take effect:

1. Early Stage Investigator (ESI) and New Investigator Policy (NOT-OD-09-013; http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-013.html) [Defines what constitutes a “new investigator” and what criteria establish a researcher as an “early stage investigator.”]

2. New NIH Policy on Resubmissions (NOT-OD-09-003: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-003.html) [States that starting with the January 25, 2009, deadline, original new applications—those that have never been submitted—and competing renewals will have only one chance to be revised, instead of two opportunities for revisions under the previous policy].

In May 2009 Review Meetings (for potential FY2010 funding), NIH will implement:

1. 9-Point Scoring System

2. Enhanced Review Criteria

3. Formatted Reviewer Critiques

4. Scoring of Individual Review Criteria

5. Clustering of New Investigator Applications During Review

For applications due in January 2010 (for potential FY2011 funding), the following will be implemented:

1. Shorter Applications for R01s and Other Mechanisms

2. Restructured Applications to Align with Review Criteria