Research Evaluation Supported by Automation - EuroCRIS

subduedlockΜηχανική

5 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

54 εμφανίσεις

©Keith G Jeffery/euroCRIS

www.eurocris.org

Research Evaluation
Supported by Automation

Keith G
J
effery

President, euroCRIS

k
eith.jeffery@stfc.ac.uk

euroCRIS Seminar 2010

1

©Keith G Jeffery/euroCRIS

www.eurocris.org

Why Research Evaluation


To protect from erroneous information /
knowledge;


To demonstrate value for money from public
funding;


To compare performance of research
institutions;


To compare judgement of proposal review
process in different funding organisations;


To promote the research outputs achieved;

euroCRIS Seminar 2010

2

©Keith G Jeffery/euroCRIS

www.eurocris.org

Kinds of Research Evaluation


The Research Grants process


Evaluation of proposals submitted for funding


Evaluation of research outputs from funded projects


The Research Communication process


Evaluation of scholarly publications submitted


The Formal Evaluation


Evaluation of research grant income


Evaluation of research outputs (particularly
publications but also products and patents)


Evaluation of contribution(s) to academic
communities

euroCRIS Seminar 2010

3

©Keith G Jeffery/euroCRIS

www.eurocris.org

How Research Evaluation


Commonly an intensive manual process involving


The funding organisation(s) defining a template


The research institutions struggling to locate, collate
and present the information


Along the way making strategic decisions on whom and what
to include and within which category to place the work


The funding organisations setting up evaluation panels
which effectively re
-
peer
-
review the submissions


Leading to allocation of funding

euroCRIS Seminar 2010

4

©Keith G Jeffery/euroCRIS

www.eurocris.org

Cost of Research Evaluation


According to PA Consulting the cost of UK
RAE2008 was > £47m


A recent report for JISC in UK estimates that with
automation this can be reduced by at least 25%


According to PA Consulting the cost of UK
submission and monitoring of research grants
is £85m p.a.


A recent report for JISC estimates that with
automation this can be reduced by 33%

euroCRIS Seminar 2010

5

©Keith G Jeffery/euroCRIS

www.eurocris.org

The Automation


The JISC
-
funded EXRI study recommended CERIF for research
information in UK


A group representing universities, research funders, Higher
education authority, research managers, charities convened and
approved the recommendation of CERIF for research information in
UK


The major automation envisaged is CERIF for


Making returns for research evaluation


Managing the research grant process


But there are additional benefits in managing research


Publicity and outreach including to innovators


Research collaborations


Finding reviewers


Strategic decision
-
making

euroCRIS Seminar 2010

6

©Keith G Jeffery/euroCRIS

www.eurocris.org

The message

From the JISC Report:


For any given institution the minimal annualised savings of owning
a modern CRIS using CERIF or a CERIF wrapper can be expressed as:


(X*225)+(y*3200)


Where


X is the number of researchers to be submitted to the REF and


Y is the number of grant applications per annum to Research Councils.


Using a typical institution with 1000 researchers and 1000 grant
applications this gives the minimal saving as £3.4m per institution
per annum



Balanced against the savings is the cost of either a CERIF wrapper
(£13,000) or CERIF CRIS (£10,450
-

£20,880) per annum


Which, frankly, is ‘lost in the noise’ of the benefit

euroCRIS Seminar 2010

7

©Keith G Jeffery/euroCRIS

www.eurocris.org

What Automation


Many research institutions have a mix of systems:
HR, Finance, webpages, institutional repository


Some have a Research Information Management
function


commonly by linking together
heterogeneous systems


Some have such a function by using a CRIS
-

and of
those many have a CERIF
-
CRIS


More organisations specify CERIF for their CRIS


ESF, ERC... Several UK universities


euroCRIS Seminar 2010

8

©Keith G Jeffery/euroCRIS

www.eurocris.org

Benefits of Automated Evaluation


Research Institution


Reduced cost


Increased accuracy


Information available for other purposes


Research Funder


Reduced cost


Increased accuracy


Information available for other purposes

euroCRIS Seminar 2010

9

©Keith G Jeffery/euroCRIS

www.eurocris.org

Benefits of Automated Evaluation


A clear transparent, reproducible process for
evaluation:


Based on open data that should already exist in a
well
-
managed research institution


Using algorithms not human opinions



Thus providing the public with assurance of
research output quality and value for money
from research funding

euroCRIS Seminar 2010

10