The importance of micro-data for the assessment of European research performance

hardtofindcurtainΠολεοδομικά Έργα

16 Νοε 2013 (πριν από 3 χρόνια και 10 μήνες)

73 εμφανίσεις

The importance of micro
-
data for the assessment of
European research performance

Emanuela

Reale

CERIS CNR

e.reale@ceris.cnr.it






European Parliament

STOA Science and Technology Options Assessment


Policy needs and opportunities


Brussels, March 26 2013

Content


Importance of micro data



Definition of micro data



Opportunities and constraints



Recently developed tools: EUMIDA and JOREP



Conclusions



2

3

Changing context and requests for STI

data and indicators



Differentiation

of

the

demand

side


Government

(Ministries,

Parliament,

local

government)


Intermediaries

(Funding

agencies,

QA

and

evaluation

agencies)


Research

organizations

(universities

and

PROs,

companies)


Stakeholders



Differentiation

on

the

supply

side


Statistical

offices


Indicator

producers


Specialized

institutes



Data

should

allow

tracing

the

effects

of

the

policies



on

the

beneficiaries

and

other

agents



on

the

context,

to

assess

the

R&D

performance

of

programmes

and

actors


New demands for new data and indicators


Demand for STI indicators become more differentiated and complex


Rise of evaluation


Importance of the evidence
-
based policy making (Sanderson, 2001)


O
vercoming the input
-
output framework linked to the linear model of
innovation


Emerging of new approaches (e.g. position indicators) for deepening the
processes that allow getting a certain performance or result



Demand for micro data emerges related to the need of investigating:


E
fficiency
, effectiveness and impact of different policy options,


T
he
performance of actors and systems in Europe,


T
he
choice of the funding instruments,


T
he
governance and steering approaches,


F
raming
conditions (enablers) for excellence and innovation,


D
eterminants
and constraints of economic
growth

4

What

are micro data for


The “we cannot manage what we cannot measure” paradigm



We need to measure performance (Wagener, 2008):


Effectiveness: doing right things (set the right targets to achieve an
overall goal)


Efficiency: doing things in the most economical way


Impact: the effect produced at a certain time and its social and
economic value


Sustainability: meeting the needs of the present without
compromising the ability of future generations to meet their own
needs



Systematic and organic linkages between policy and indicators at
European level under a comparative perspective


New policy requirements and changing organizational contexts



5

Data for STI investigations


Aggregated data


Based on large amount of input and output indicators


No distinguishing individual contributors and beneficiaries


S
ystematic accounting methodologies developed by international
organizations in order to represent financial, human resources and output



Case studies


Qualitative investigations


Deepening effectiveness, adequacy or equity of a programme, impact on
performers, social dynamics, processes



Administrative data


Information collected mainly for the management of administrative
procedures. They have a significant potential as research resource, especially
when linked with other sources (Jones and Elias, 2006)


6

Micro data


One can distinguish between (
Everitt
, 2006)


Census data: aimed to observe every member of a population


Survey: collect planned
i
nformation for a sample of respondents and estimate
characteristics, performance and effects



Data collected for statistical purposes are generally based on surveys


Survey of micro data
devoted to investigate
performance cannot be confused with
standardized surveys for aggregated data (e.g.
Frascati

Manual)



Advantages


Possibility to look at range, variance, skewedness, coefficients of variation


Aggregate statistics often do not allow answering policy questions



Limitations and problems


Design and size of the sample


Questions and definitions


Non
-
responses


Attrition problems in longitudinal surveys


Problems of large countries

7

Micro data problems


Availability



are data available at national level? Who own the data (government,
statistical offices, research performers, intermediaries)?



Comparability



how far are data comparable? What rules for data collection
should be designed? What is the perimeter we can investigate using micro data?
What is the underlying conceptual framework?




Confidentiality



How far confidentiality impedes/limits the data collection and
dissemination?



Updating

-

Data become old: how can we update data collection
? When data
collected become a matter for NSI?




Maintenance and management (infrastructure)

Who is in charge for
management of data and how the access to data should be regulated?

8

Two examples of micro data


Different layers in the European research policy system: performers, policy layers,
research funding


We
look at the mentioned problems using 2 key experiences of
micro
-
data
collection, on the performer and on the research funding


EUMIDA


Census data on Higher Education
Institutions


It
is the first attempt toward the foundations of a regular data collection
(Register)
by
national statistical institutes on individual higher education
institutions
in the
EU
-
27 Member States together with Norway and Switzerland


Large feasibility study on 27 European countries under the auspices of EC



JOREP


Database on characteristics of Joint and Open
Research
P
rogrammes


P
roviding
a
quantitative
basis for the monitoring of investments in joint and open
research programmes in EU countries, as well as empirical evidence of the policy
rationales and impacts of these programmes on the European Research Area
.


Experimental data collection and impact analysis on 11 European countries

9

Availability


EUMIDA


C
ore set of micro data largely available


Research active HEIs data more problematic


Reasons of non
-
availability (FR, p. 141)


Legal
issues (e.g. the statistical law explicitly forbids the publication of
micro data
)


Administrative
barriers (e.g. it would be possible to publish
micro data
but NSI depend on
administrative decisions from the Ministry)


Institutional
settings (e.g.
the
publication of
micro data
requires additional workload for which the
institution is not prepared, or it would be difficult to allocate the responsibility
internally

no official
sources to obtain output information on HEIs).


JOREP


Descriptors of joint programmes largely available


Openness of national programmes very difficult to understand


Funding data generally available but with some exceptions


Beneficiary data the most problematic issue


Reasons for non
-
availability


Funding agencies do not hold data based on the
Frascati

Manual performer sectors


Funding agencies record data on the basis of project decisions (trade off between funding volume
referred to the commitment year and funding decision refereed to the implementation years)


Loan repayments from private performers


10

Comparability


For both the programmes comparability was the most challenging issue


Building a conceptual framework of what census and survey shall include (it
means going inside the structure of HE systems and project funding systems
of European countries)


Decide the right level of analysis (HEIs
vs

Departments, Research Programmes
vs

Research Projects)


Define the perimeter of the data collection


Choice common definitions, modes of data collection, calculations, currency,
estimations, etc.


Disaggregation of data not always possible (e.g. beneficiaries in JOREP.
Current and capital expenditures in EUMIDA)


Missing data



Comparability often implies a work of social construction

11

Confidentiality


Problem with the data disclosure of research performers:



Agreement of the Authority in charge of education and research


Agreement of the HEIs that data as regards the expenditures (R&D
included) and income can be released



No problems (or very limited ones for the disclosure of data on project
funding
-
organizations’ name not included)

12

Updating


Both data collections relate on country correspondents (experts) for the coordination of the
contact with the relevant actors



Actors and roles in the data collection


Ministry of
Education and Research


National Statistical
Institutes


Funding Agencies (research performers acting as Agencies)


Other official
sources (Web sites, Reports)


HEIs for descriptors




Need to enlarge the coverage


Other EU and non
-
EU countries


Other project funding schemes


Problems linked to the changes of the national HE systems and PF systems, which can require
adaptation of the methodology



Important updating on a regular basis


13

Data infrastructure


Micro data
collection often requires activities that go beyond the NSI
functionalities



The
definition of a suitable organizational form and of procedures for regular data
collection is
important



It must takes care of two
characteristics:


The inclusion of both
expert
-
based descriptive information and statistical data
on funding flows;


Data
are owned by different subjects and at different institutional
levels


European agencies


European
Commission,


National States


N
ational
funding
agencies


14

Proposed organization for Joint programme data collection

15

Conclusions


Micro

data

are

essential

tools

for

the

assessment

of

European

performance


Overcoming

the

difficulties

linked

to

heterogeneous

populations

and

small

numbers


Large

access

for

different

type

of

users



Supporting

further

investigation

of

the

STI

systems

answering

relevant

policy

questions



Several

problems

affect

the

possibility

to

collect

and

to

maintain

databases

of

micro

data
:

data

need

investment


Availability,

Comparability,

Confidentiality,

Updating,

Infrastructure



The

RISIS

initiative

under

the

EUFP

framework

toward

the

building

of

research

infrastructures

for

STI

studies

shall

supply

an

important

step

forward

micro
-
data

exploration

and

exploitation




16