NIH Peer Review: Continuity and Change, a ... - DC Principles

boardsimplisticΒιοτεχνολογία

20 Φεβ 2013 (πριν από 4 χρόνια και 3 μήνες)

144 εμφανίσεις

Center for Scientific Review

National Institutes of Health

Department of Health and Human Services

Toni Scarpa

NIH Peer Review:

Continuity and Change

APS/Cardiovascular Section

Salt Francisco, April 2006

Peer Review: An N.I.H. “Conception”


Is the heart and soul of NIH


Has produced an effective partnership between the
federal government and research institutions


Has created the best academic medical centers, the best
biomedical/behavioral research and biotechnology


Has made possible the best cures and the best prevention


Has been admired and imitated here and abroad


Has protected NIH against outside influence

Center for Scientific Review

Applications received for all of NIH and applications
referred for CSR review, FY 1998
-
2004

0

20,000

40,000

60,000

80,000

1998

2000

2002

2004

Fiscal year

Number of applications

Applications received for all of NIH

Applications assigned for review by CSR

CSR 4 Review Divisions & 23 IRGs

Division of Biologic

Basis of Disease

Elliot Postow, Ph.D.

AIDS and Related

Research IRG (AARR)

Ranga V. Srinivas, Ph.D.


Endocrinology, Metabolism,

Nutrition, and Reproductive


Sciences (EMNR)

Sooja Kim, Ph.D.

Immunological Sciences

IRG (IMM)

Calbert Laing, Ph.D.

Oncological Sciences

IRG (ONC)

Syed Quadri, Ph.D.

Infectious Diseases and

Microbiology IRG (IDM)

Alex Politis, Ph.D.

Division of Physiology

and Pathology

Michael Martin, Ph.D.

Cardiovascular Sciences

IRG (CVS)

Joyce Gibson, D.Sc.

Integrative, Functional

and Cognitive Neuroscience

IRG (IFCN)

Christine Melchior, Ph.D.


Renal and Urological Sciences
IRG (RUS)

Daniel McDonald, Ph.D.

Hematology

IRG (HEME)

Joyce Gibson, D.Sc.

Digestive Sciences IRG

(DIG)

Mushtaq Khan, Ph.D., DVM

Division of Clinical and

Population
-
Based Studies

Anita Miller Sostek, Ph.D

Surgery, Radiology, and

Bioengineering IRG (SRB)

Eileen Bradley, D.Sc.

Health of the Population

IRG (HOP)

Robert Weller, Ph.D.

Risk, Prevention, and Health

Behavior IRG (RPHB)

Michael Micklin, Ph.D.

Brain Disorders and Clinical

Neuroscience IRG (BDCN)

Dana Plude, Ph.D.

Behavioral & Biobehavioral

Processes IRG (BBBP)

Karen Sirocco, Ph.D.

Division of Molecular

and Cellular Mechanisms

Donald Schneider, Ph.D.

Bioengineering Sciences and

Technologies IRG (BST)

Sally Amero, Ph.D.

Biology of Development and

and Aging (BDA)

Sherry Dupere, Ph.D.

Biological Chemistry and
Macromolecular Biophysics
IRG (BCMB)

John Bowers, Ph.D.

Cell Biology IRG (CB)

Marcia Steinberg, Ph.D.

Genes, Genomes, and

Genetics IRG (GGG)

Richard Panniers, Ph.D.

Molecular, Cellular and

Developmental

Neuroscience IRG (MDCN)

Carole Jelsema, Ph.D.

Respiratory Sciences IRG
(RES)

Mushtaq Khan, Ph.D., DVM

Musculoskeletal, Oral, and
Skin Sciences IRG (MOSS)

Daniel McDonald, Ph.D.

CSR Mission Statement

To see that NIH grant applications
receive fair, independent, expert,
and timely reviews
--

free from
inappropriate influences
--

so NIH
can fund the most promising
research.




Time

Necessary Changes in CSR Peer Review Operations


Increase communications between CSR, the
ICs, our reviewers and applicants



Increase uniformity



Increase efficiency



Facilitate work of IC program staff

Changes in CSR Operations

Potential of Knowledge Management Tools

for Peer Review

Collexis Software or Others




Knowledge management solutions


Fingerprinting and text retrieving


Disease coding


Benefits for Peer Review



Assigning applications to Integrated Review Groups or Study
Sections


Selecting reviewers (one application, multiple applications)


Nine pilots are underway to begin to assess these benefits


Study Section Realignment


Review of one IRG every month


Total review every 2 years

Required Changes in Current Systems



Shorten the review cycle



This is Not an Ford Assembly Line

Receipt

Refer

Evaluate Scientific

Merit of Applications

EnterpriseArchitecture@mail.nih.gov

Shortening the NIH Review Cycle, Initial Steps

For most research grants, we are posting
summary statements within one month after the
study section meeting instead of two to three
months after the meeting (effective Oct 05)


We are conducting a pilot study to speed the
review process for new investigators so they
may revise and resubmit for the very next
review cycle 4 months earlier than before
(effective Feb 06)

Desirable Changes in CSR Review



Shorten the review cycle



Address concern that clinical research is not
properly evaluated



Improve the assessment of innovative, high
-

risk/high
-
reward research



“The judging of grants has become a charade.”

The American Society for Cell
Biology

“The judging of grants has
become a charade. To be
funded, the experimental
plan has become a litany
of experiments already
accomplished so that
everything is feasible.
When grants come back
with unfundable scores,
new investigators may not
have sufficient resources
to do the experiments that
“show feasibility.”


Zena Werb

President, ASCB




Newsletter August 2005


Possible Changes in Current Systems



Shorten the review cycle



Address concern that clinical research is not
properly evaluated



Improve the assessment of innovative, high
-

risk/high
-
reward research



Do more to recruit and retain more high
-
quality
reviewers


Expanding Peer Review’s Platforms

Electronic Reviews


Telephone Enhanced Discussions


Video Enhanced Discussions


Asynchronous Electronic Discussions


Study Sections

Necessity





Clinical reviewers



Preference




Physicists, computational biologists



New Opportunities



Fogarty, International Reviewers

The First NIH Study Section



1945

The Last NIH Study Section



2005

If we didn’t have any peer
-
review
system and we had to design one
from scratch, what would it look like?

Possible New Systems

Questions

Applications


Should we have 3 or 365 deadlines for most
applications?







Should applications (Rs) be shortened? Should
appendices be eliminated or reduced in size?






Is there more value in having 2
-
3 reviewers
reading 25
-
page applications or 10
-
15 reviewers
reading 5
-
page applications?



Questions

Study Section Meetings


What is the ideal number of members to have
serve on a study section?







Is one study section with 50
-
70 reviewers
efficient?






What is the intellectual contribution of
individual reviewers in large study sections?



Questions

Scoring


Is it proper or desirable to have 50
-
70 reviewers
voting on priority scores for each application
referred to their study section?







Is consensus always good? Or should we focus
on score variance?







Applications Received for All of NIH FY 1998
-
2004

0

20,000

40,000

60,000

80,000

1998

2000

2002

2004

Fiscal year

Number of applications



Number of Research Grant Applications/Applicant

1.05
1.1
1.15
1.2
1.25
1.3
1.35
1.4
1.45
1980
1982
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004

CSR Applications Reviewed, Regular and SEP



May Council Only



0
2000
4000
6000
8000
10000
12000
14000
16000
1999
2000
2001
2002
2003
2004
2005
Applications Regular
Applications SEP


Study Section Application/Reviewer Ratio




October Council Only

0
0.5
1
1.5
2
2.5
3
3.5
4
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Regular
SEP
Two groups of challenges/opportunities


Reviewers


Decrease the number of reviewers and increase the
quality


Increase the number of applications reviewed
without extra workload


Recruit and retain the best reviewers


SRA


Increase efficiency


Recruit and train


Possible Short Term Approaches for Increasing
Efficiency for Reviewers and CSR


Replace Many SEPs with Smaller Parallel Study
Sections


Enlarge Study Section Membership and Decrease
Frequency of Participation


Pre Meeting Streamlining


Various Review Platforms


Hybrid Review Platforms


Staggering Application Deadlines


2 instead of 3 reviews


Shorten Applications


More Structured Applications and Reviews


Size of Grant Applications


RO1


Will increase number of applications reviewed by
reviewers


Will decrease the number of reviewers in a study section


May be combined with a change in format of the
application, more consonant to review criteria


May be combined with scoring individual criteria (i.e.
relevance, innovation, etc


Strong support by councils and scientific leadership


May result in more innovation


This is CSR

Coronary Heart Disease

Age
-
Adjusted Death Rates in U.S.:


Actual
(blue)

vs Expected (
yellow)


500

400

300

200

100

0

1950

1955

1960

1965

1970

1975

1980

1985

1990

1995

2000

Deaths per 100,000

Year

~ 514,000 Actual Deaths


in 2000

~ 1,329,000 Projected Deaths



in 2000

815,000 Deaths

Prevented in 2000

The True Value of Peer Review