REF 2014 and the Business &

confidencetoughManagement

Nov 20, 2013 (3 years and 8 months ago)

86 views

REF 2014 and the Business &
Management community

Mike Pidd

Chair: sub
-
panel C19 (Business & Mgt Studies)

Department of Management Science

Lancaster University Management School

UK

M.Pidd@lancaster.ac.uk


Overview


Purposes of the REF (the
Research Excellence
Framework)


REF versus RAE 2008


Panels


Staffing


Outputs


Environment


Impact


Timetable


2

You can always blame the REF

REF purposes: official statement


FUNDING: inform research funding allocations by
the four UK HE funding bodies (approximately £2
billion per year)


ACCOUNTABILITY: for public funding of
research so as to demonstrate its benefits


QUALITY INDICATORS: provide benchmarks
and reputational yardsticks

3

REF principles: official statement


Expert, peer review panels


Equity: all types of research and forms of
output across all disciplines shall be assessed
on an equal basis


Equality: HEIs encouraged to submit the work
of all their excellent researchers


Transparency: assessment criteria and
procedures, and outcomes to be published in
full

4

5

RAE 2008: how did it work?

Research

environment
(20%)

Esteem &
impact
indicators
(10%)

Quality
profile

Weighted and aggregated across each submission

e.g.

Research income

PhD students

Staff development

Quality level

4*

3*

2*

1*

u/c

% research activity

10

40

35

15

0

Nossex
Business
School

Research outputs: 4 per person (70%)

Peer
review

6

REF 2014: how will it work?

Research

environment
(15%)

Impact
(20%)

Quality
profile

Weighted and aggregated across each submission

e.g.

Research income

PhD students

Staff development

Quality level

4*

3*

2*

1*

u/c

% research activity

21

38

37

4

0

Nossex
Business
School

Research outputs: 4 per person (65%)

Peer
review

Spot the

differences

Overview of REF assessment framework

7

Overall quality

Outputs

Maximum of 4
outputs per
researcher

Impact

Template and case
studies

Environment

Template and data

65%

20%

15%

Key changes since RAE 2008


Includes assessment of impact


Removed
esteem

as a distinct element


Structured approach to research environment


Fewer UOAs/panels operating more consistently


Only 4 main panels (main panel C for BMS)


No separate panel for Accounting & Finance


Strengthened equality and diversity measures


Revised eligibility criteria for staff


Some UOAs will make (limited) use of citation data


... But not in BMS (C19)


Increased ‘user’ input on panels; and an integrated role for
additional assessors


Publication of overall quality profiles in 1% steps

8

Panels and sub
-
panels

9

REF 2014: roles of panels


MAIN Panel C


Panel criteria & working methods


Oversight during assessment process


Consistent application of overall assessment standards


Sign off final profiles


SUB
-
PANEL C19: Business & management studies


Limited
, specific variations in criteria & working
methods


Conduct the assessment


Recommend final profiles


Add extra assessors & users during 2012/13

10

Structure of main panel C: REF 2014

11

Janet Finch

Chair

Manchester/Keele

Cara Aitcheson

Sports
-
related studies

Edinburgh

Hastings Donnon

Anthropology & dev studies

Queens Belfast

Gillian Douglas

Law

Cardiff

Colin Hay

Politics & int studies

Sheffield

Peter Neary

Economics etc

Oxford

Alan Penn

Architecture,

built env & planning

UCL

Mike Pidd

Business & mgt studies

Lancaster

Andrew Pollard

Education

Inst for Education

Keith Richards

Geog,

env studies & archaeology

Cambridge

John Scott

Sociology

Plymouth

Peter Taylor Gooby

Social work

& social policy

Kent

UK academic
members (sub
-
panel Chairs):

Structure of main panel C: REF 2014

12

Trevor Barnes

Geography

UBC

Frans Berkhout

Innovation & sustainability

VU Amsterdam

Francois Borguignon

Economics

Paris

School of Economics (prev
Chief Econ, World Bank)

Paul Finch

Architecture

& planning

Commission for the Built
Environment

Herbert Kritzer

Law & politics

Univ Minnesota

Jone Pearce

Business & mgt studies

UC Irvine

Mark

Robson

Various

Bank of England

Sue Rossiter

Education

NFER

Martin Walsh

Charity

sector

Oxfam

Paul Wiles

Social work & social policy

Oxford (formerly Home Office)

Sharon Witherspoon

Social policy

Nuffield Foundation

International and user members

Sub
-
panel C19: business & mgt studies

Mike Pidd
1

Lancaster

Chair & member of Main Panel C

John Arnold

Sheffield

Martin Laffin

Durham

Jan Bebbington

St Andrews

Alan Marsden

Ex
-
Arup

David

Blackaby
2

Swansea

Kathrin Moeslin

Erlangen
-
Nuremberg

Jane Broadbent
3

Roehampton

Peter Naude

Manchester

Robert Blackburn

Kingston

Andy Neely

Cambridge/Cranfield

Chris Brooks

Reading

Caroline Oades

ACCA

Colin Eden

Strathclyde

Richard Thorpe

Leeds

Paul Edwards

Birmingham

Ian Tonks

Bath

Guy Fitzgerald

Brunel

Caroline Tynan

Nottingham

Keith Glaister

Sheffield

Terry Williams

Hull

Mark Jenkins

Cranfield

Hugh Willmott

Cardiff

13

2: Joint with Economics sub
-
panel

3:
Deputy
Chair

Staff eligibility & other rules

14

Staff: Eligibility & requirements


Category A staff


Academic staff with a contract of at least 0.2 FTE,


on payroll of the HEI 31 Oct 2013


primary employment function of either ‘research only’ or ‘teaching
and research’


Category C staff:


Employed by an organisation other than an HEI


Have contract or job role including research,


Research primarily focused in the submitting unit on 31 Oct 2013


Will contribute to the quality profile but not volume for funding


Research assistants are only eligible by exception


Institutions MUST produce an Equality & Diversity Code


Four outputs per person unless special circumstances


15





Staff: Individual circumstances

Clearly defined (rules)


Qualified as ECR


Part
-
timers


Maternity, paternity or
adoption leave


Secondments of career breaks
outside HE

Complex (EDAP)


Disability


Ill health or injury


Mental health conditions


Constraints relating to
pregnancy or maternity in
addition to maternity leave


Childcare or other caring
responsibilities


Gender reassignment


Other circumstances relating
to characteristics protected by
the Equality Act 2010


Reduced # outputs allowed

16

Early career researchers

Date at which the individual first met the definition of an
ECR:

Outputs
may be
reduced


by up to:

Between 1 August 2009 and 31 July 2010

1

Between 1 August 2010 and 31 July 2011

2

After 1 August 2011

3







Part
-
timers and career breaks

Equiv months absent
from
work (1
Jan 2008 to
31 Oct
2013):

For part
-
time
staff, average
FTE
worked (1
Jan 2008 to
31 Oct
2013):

Outputs
may be
reduced

by
up to:

14


27.99

0.601


0.8

1

28


48.99

0.301


0.6

2

49 or more

0.3 or less

3

Output tariffs: individual circumstances

17

NOTE: % availability need not be same as % used in Volume calculation

Other individual circumstances


Maternity, paternity & adoption


Outputs may be reduced by 1 for each period of
statutory leave over the REF period.


Also by 1 for each extra period of such leave lasting 4
months or more.


Complex circumstances


REF EDAP (Equality & Diversity Advisory Panel)


Recommend any reductions to main panel Chair, who will
decide on any reductions


Sub
-
panels


Must abide by main panel Chair’s decision


Will not be told of the circumstances

18

Outputs

19

Eligibility of outputs


Must be a product of research (see REF defn)


All forms of output are welcome and will be
treated equally


Into public domain between 1/1/08
and
31/12/13


BUT may include
pre
-
published

work (e.g. online
first or working paper if (and only if)


Pre
-
published in 2007 but in print form after 1/1/08


Not included in RAE 2008.

20

Co
-
authored outputs


Recognised to be increasing as collaboration grows in Main
Panel C disciplines


Where outputs are co
-
authored/produced, submitting
author expected to have made a substantial contribution


Sub
-
panels wish to receive the fullest picture of a
submitting HEI’s research, hence


Co
-
authored/produced outputs may be submitted twice in the
same UOA by the same HEI


MUST

be accompanied by a statement explaining the substantial
and distinctive contribution of each of the submitting authors


Still unclear about pre
-
published, co
-
authored work if not used by
submitting author/institution in RAE 2008.

21

Double weighted outputs


May be requested for outputs
of extended scale & scope


MUST have supporting
statement


Sub
-
panels will assess the
request separately from its
quality


If accepted: counts for 2 items


1 reserve allowed for each
double weighing request

22

Environment

23


Vitality
: reflects the existence of a thriving, dynamic and
fully participatory research culture based on a clearly
articulated research strategy, displayed both within the
submitting unit and in its wider contributions, and in terms
appropriate to the scale and diversity of the research
activity that it supports



Sustainability
: understood in terms of the extent to
which it is capable

in the future

of continuing to support
and develop such research activity as defined in the quality
levels, both within the submitted unit and the discipline
more generally

Environment assessment criteria

24

Research environment: 2 aspects

1.
Narrative (template) with 5 sections

i.
Overview

ii.
Research strategy

iii.
Staffing strategy & development, including research
students

iv.
Income, infrastructure & facilities

v.
Collaboration & contribution to the discipline

2.
Quantitative data: based on HESA stats: e.g.
doctoral degrees, income ...

25

Impact

26

Impact defined


An effect on, change or benefit to the economy, society,
culture, public policy or services, health, the environment
or quality of life, beyond academia


Includes

an effect, change or benefit to:


The activity, attitude, awareness, behaviour, capacity, opportunity,
performance, policy, practice, process or understanding


Of an audience, beneficiary, community, constituency, organisation
or individuals


In any geographic location whether locally, regionally, nationally or
internationally


Excludes

impacts on research or the advancement of
academic knowledge within HE; and impacts on teaching or
other activities
within the submitting HEI


27

Examples of impact: main panel C

28

Impacts on:

Examples:

Creativity, culture and

society

Enhancements to heritage preservation,

conservation and
presentation

Shaping

or informing public attitudes and values

Economy,

commerce
or organisations

Improved products, processes or

workplace practices

Enhanced corporate social responsibility policies

Environment

Changes in public awareness or behaviour

Business operations have been changed

to achieve
environmental objectives

Health

and welfare

Development

of policy or practice with regard to health
services or social care provision

Practitioners

and
professional services

Influence

on professional standards, guidelines or training

Practitioner debate has been stimulated by research findings

Public

policy, law and
services

Legislative change or effect on legal practice

Influence on policy (by government, NGO or private
organisation)

Impact

on democratic participation or access to justice

Research impact submission & criteria

29

Impact template

How is impact facilitated? Context, approach(es) to
impact, strategies & plans, relation to case studies

Impact
case 2

Impact
case 1

Impact
case 3

Impact
case 4

Impact
case
n

CRITERIA

Reach
: understood in terms of the extent and diversity of the
communities, environments, individuals, organisations or any other
beneficiaries that have benefitted from or been affected
(not geographic,
but relative to potential domain
)

Significance
: understood in terms of the degree to which the impact
has enriched, influenced, informed or changed policies, opportunities,
perspectives or practices of communities, individuals or organisations


Maximum of 4 pages per case.


Must demonstrate that the unit’s research made a
distinctive contribution to the impact claimed
during the
REF period 1/1/08 to 31/12/13


Evidence for the impact claimed must be cited


May include qualitative, quantitative or material


Should be independently verifiable wherever possible


Case studies should not describe activity alone, but should
make clear links between activity and impact claimed


Research on which claimed impact is based


Must have taken place at the submitting HEI


Should meet the definition of 2* (internationally recognised)


Underpinning research published since January 1993



Impact cases

30

REF 2014 Timetable

31

2011


Panels appointed
(Feb)


Guidance on
submissions (Jul)


Draft panel criteria
for consultation
(Jul)

2012


Final panel criteria
and methods (Jan)


HEIs submit codes of
practice (Jul)


Pilot of submission
system (Sep)


Requests for multiple
submissions (by Dec)


Survey of submission
intentions (Dec)

2013


Launch REF
submissions system


Recruit additional
assessors


Staff census date (31
Oct)


Submissions
deadline (29 Nov)

2014


Panels assess
submissions


Publish outcomes
(Dec)

REF 2014: important dates

Date

Significance

31/12/13

End of REF period

for outputs, impact template
& cases and research underpinning impact

29/11/13

Submissions due

31/10/13

Census

date for staff

31/7/13

End of REF period for research

students,
research income & environment

01/01/08


Start of REF period

for outputs, environment &
impact template & cases

01/01/93

Start of period for research

underpinning
impact

32

Questions & discussion

Likely sub
-
panel workloads (based on 2008)

Sub
-
panel

FTE staff/member

16

Architecture

& planning

45

17

Geography

& architecture

67

18

Economics

& econometrics

47

19

Business

& management studies

159

20

Law

100

21

Politics

62

22

Social work & social policy

50

23

Sociology

47

24

Anthropology & development

studies

31

25

Education

81

26

Sports
-
related

studies

38

34

Source: REF team

HEFCE QR Funding weighting

4*

3*

2*

1*

0

09/10 based on RAE 2008

7

3

1

0

0

11/12 based on RAE 2008

9

3

0.294

0

0

Based on REF 2014

?

?

?

0

0

35

Distinct possibility that QR weighting for 2* will be zero after 2014

36

Main panel I: Chair David Otley

34: Economics & econometrics

(David Greenaway, Nottingham)

35: Accounting & finance

(Andy Stark, MBS)

36: Business & management

(Mike Pidd, Lancaster)

37: Library and information mgt

(John Feather, Loughborough)

Ray Paul

David

Blackaby

Jane

Broadbent

Structure of main panel I: RAE 2008

Sub
-
panels did the work

Main panel had
oversight