Microsoft PowerPoint - NCRM EPrints Repository

heehawultraΜηχανική

22 Φεβ 2014 (πριν από 3 χρόνια και 5 μήνες)

62 εμφανίσεις

(
1
)





Systematic Reviews:
The Potential of
Different Methods

Steve Higgins

Henry Potts

James Thomas

Geoff Wong


ESRC Research Methods Festival

Oxford, 5 July 2012


(
2
)

Introduction


Aims


To outline major approaches to systematic
reviewing and research synthesis


To demystify what on first glance looks like a
bewildering set of overlapping approaches


To examine some of the assumptions that
underpin these approaches


To discuss how these approaches might help in
my (your!) situation


(
3
)

Introduction


Session structure:


20 Minute presentation + 10 minutes discussion
/ clarification of four approaches to synthesis


Coffee break after the 3
rd

/ 4
th

presentation


Discussion in pairs


Questions + panel discussion

(
4
)

What is a systematic review?


Systematic
: ‘done or acting according to a
fixed plan or system; methodical’



Review
: ‘a critical appraisal of a book, play,
or other work’


(OED)

(
5
)

What is systematic research
synthesis?



Synthesis
:
‘The process or result of building up
separate elements, especially ideas, into a connected
whole, especially a theory or system’ (OED)



Not just a report of the findings of the individual studies in
a review




Involves a transformation of the data from primary studies
in some way




Synthesis of the findings of all the included studies in order
to answer the review question



(
6
)

Aims of synthesis reviews


Answer questions: what do we want to know
and how can we know it?


Bring together and ‘pool’ the findings of
primary research (i.e. clarify what we know)


Any question so potentially any type of
evidence


Driven by review users to answer relevant
questions in relevant ways


(
7
)

Introduction


We will look at four main approaches to
reviewing:


Meta
-
analysis


The synthesis of ‘qualitative’ research


Realist review


Meta narrative review


Each have different disciplinary
backgrounds and underlying assumptions


They all bring together research using two
main logics
-

(
8
)

Aggregation in reviews





Aggregation refers to
‘adding up’ (aggregating)
findings from primary
studies to answer a
review question




… to indicate the
direction or size of effect

… and our degree of
confidence in that finding


(
9
)

Configuration in reviews





Configuration involves
the arrangement
(configuration) of the
findings of primary
studies to answer the
review question….


… to offer a meaningful
picture of what research
is telling us

... across a potentially
wide area of research




(
10
)

Most reviews contain elements of both,
but some patterns are common

Aggregation

Configuration

Approach to theory

Test

Generate

Research question

Closed

Open

Concepts

Secure

prior to review

Emergent in review

Timing of methods
decisions

A priori

Iterative / emergent

Search for studies

Representative /
unbiased / exhaustive

Sufficient / purposive

Quality appraisal

Avoiding bias

Richness of data

Explore

(
11
)





The EPPI
-
Centre is part of the Social Science Research Unit at the Institute of
Education, University of London,
UK
1

Methods for the synthesis
of qualitative research

James Thomas
1


ESRC Research Methods Festival

Oxford, 5 July 2012


(
12
)

Synthesis of qualitative research:
outline


What can qualitative research tell us?



Current developments/approaches in
qualitative review methods




Examples of syntheses of qualitative
research


(
13
)

What can qualitative research tell us?

(
14
)

Qualitative
research…


aims to provide an in
-
depth understanding of
people’s experiences, perspectives and histories
in the
context

of their personal circumstances or
settings
…*


with the use of unstructured methods which are
sensitive to the social
context

of the
study*


the capture of data which are detailed, rich and
complex
…*


is
generalisable

on the basis of
theory

not
statistical probability **

* Spencer
et al
(2003
); **
Popay

(2006)

(
15
)

Methods used include*…


exploratory interviews


focus groups


observation


conversation


discourse and narrative analysis


documentary and video analysis

*Spencer
et al
(
2003)

(
16
)

Is it
simply the analysis of
textual
data?

“Increasingly, the terms ‘quantitative research’
and ‘qualitative research’ came to signify
much more than ways of gathering data;
they came to denote divergent assumptions
about the nature and purposes of research
in the social sciences”



Bryman

(1988)
Quantity and Quality in Social Research
. London:
Unwin

Hyman. p.3.

(
17
)

Qualitative research can*


help to
define

policy questions


look in detail at
how

a programme or trial
was actually implemented


help to determine
appropriate

outcome
measures by looking at ‘subjective’
outcomes


help to clarify
what counts
as effective or
successful

* Davies
(2000) ‘Contributions from qualitative research’ in Davies
et
al
(
eds
)
What works? Evidence
-
based policy and practice in public
services
. Bristol: Policy Press. Cited in *Spencer
et al
(2003). p.36

(
18
)

…continued


identify and explore
unintended

consequences


contribute to service delivery and policy
development by describing
processes

and
contexts


inform and
illuminate

quantitative studies, e.g. by
contributing to the design of structured
instruments, assessing the fairness of
comparisons in experimental studies, or
unpacking variation within aggregated data

Davies (2000) ‘Contributions from qualitative research’ in Davies
et
al
(eds)
What works? Evidence
-
based policy and practice in public
services
. Bristol: Policy Press. Cited in *Spencer
et al
(2003). p.36

(
19
)

Current developments in
synthesising qualitative research


Growing recognition of the value of
qualitative research to inform policy &
practice


Methods developing to facilitate this

(
20
)


Try: http://www.wordle.net/

(
21
)

Methodological study


We searched for methodological papers
concerning the synthesis of concepts or
theories


Purposive, rather than systematic search


Reference ‘chasing’


Google scholar search


Handsearching key journals


203 papers retrieved


9 distinct methods for synthesis

(
22
)

Methods for the synthesis of
qualitative research identified


Meta narrative synthesis


critical interpretive synthesis


meta
-
study


meta
-
ethnography


grounded theory


thematic synthesis


framework synthesis


ecological triangulation

(
23
)

Many methods: similar or different?


Examined the methods across different
dimensions:


Epistemology


Approach to quality assessment


Attitudes towards
problematizing

the literature


Use of review question


How similar / different the included studies were


Characteristics of the synthetic product


Found they fell into two broad camps: ‘idealist’
and ‘realist’

(
24
)

‘Realist’ approaches

Purpose


To answer a policy relevant question

Methods


Qualitative/ quantitative data analysed with

qualitative/ quantitative methods


Searching linear or iterative


Quality assessment of study methods

Product


Directly applicable to policy and practice
decisions

(
25
)

‘Idealist’ approaches

Purpose


To explore and construct concepts from the data


For generating theory

Methods


Qualitative data analysed with qualitative methods


Searching iterative


Qualitative assessment of study
content
>

method

Product


Complex, requiring further interpretation before
being used for policy or practice

(
26
)

Conclusions


Operationally, many methods are very
similar


Underlying principles differ


Product differs in terms of the amount of
additional interpretation required. This may
reflect…



(Barnett
-
Page and Thomas (2009))

(
27
)

Perspectives on how evidence
informs the policy process

Idealist

Realist

Knowledge
-
driven model;
Problem
-
solving (policy
driven)

model*


(either researcher
disseminates research or
policy
-
maker commissions /
requests specific research)

Less suited

More suited

Interactive model;
Enlightenment model*


(ongoing interactions

between researcher and
policymaker; ‘gradual
sedimentation of insights…’)

More suited

Less suited

*Weiss (1979)

(
28
)

Two examples


All this is a bit abstract… what do
systematic reviews of qualitative research
look like?


Two examples:


The ‘fine grain’ of a synthesis


‘Mixed methods’ synthesis

(
29
)

Example 1: Informing policy about
the ‘obesity epidemic’

What are children’s
views about:


the meanings of obesity
or body size, shape or
weight (including their
perceptions of their own
body size)?


influences on body size?


changes that may help
them to achieve or
maintain a healthy
weight?


Views = attitudes,
opinions, beliefs,
understanding or
experiences

As distinct from
health/weight
status, behaviour,
factual knowledge


Rees et al 2009

(
30
)

‘Thematic synthesis’


Similar to other methods of synthesising
qualitative research (e.g. ‘meta
-
ethnography’)



Source data = text (documents)


Source material = conceptual


Key method = translation


Final product = interpretation



(Thomas and Harden 2008)

(
31
)

Stages of thematic synthesis


Stages one and two: coding text and
developing descriptive themes


Identifying the ‘findings’


Line
-
by
-
line coding


Developing descriptive themes


Stage three: generating analytical themes


In the light of the review question

(
32
)

Screenshot


line
-
by
-
line coding


(
33
)

Screenshot


descriptive codes
diagram


(
34
)

Example 2. adding insights to other
existing reviews


How might a systematic review of qualitative
research support or add insight to a meta
-
analysis?


Product of qualitative synthesis can form the
conceptual framework within which heterogeneity
can be explored in a meta
-
analysis

Searching,
screening mapping and user
involvement

e.g. Statistical
meta
-
analysis of trials

a)
Data
extraction

b)
Quality assessment

c)
Effect sizes pooled

d)
Narrative
synthesis


Addresses sub RQ e.g. ‘which interventions are
effective?’

e.g. Thematic
synthesis of qualitative studies

a)
Data extraction

b)
Quality assessment

c)
Descriptive themes

d)
Analytical
themes


Addresses
sub RQ e.g. ‘what are people’s perspectives
and experiences ?


Synthesis
1

Synthesis 2

Synthesis
3

Driven by overall review question


Integration of
separate syntheses e.g.

a)
Matches, mis
-
matches
and gaps

b)
Hypotheses generated in synthesis 2 tested
amongst trials in synthesis 1

Review Question

e.g. ‘What is known about the barriers to, and facilitators of, outcome X (e.g. physical
activity) amongst population A (e.g. young people

(
36
)

Cross study synthesis: Findings

Children’s views

Outcome evaluations

Recommendation for
interventions

Good quality

Other

Do not promote fruit and
vegetables in the same way

No soundly
evaluated
interventions

No other
interventions
identified

Brand fruit and vegetables
as an ‘exciting’ or child
-
relevant product, as well as
a ‘tasty’ one

5 soundly
evaluated
interventions
identified


5 other interventions

Reduce health emphasis in
messages to promote fruit
and vegetables particularly
those which concern future
health

5 soundly
evaluated
interventions
identified


6 other interventions
identified

(
37
)

Cross study synthesis: an example of
sub
-
group analysis

Increase (standardised portions per day) in vegetable intake
across trials

0
0.1
0.2
0.3
0.4
0.5
0.6
Wardle
Liquori
Henry
Anderson
Reynolds
Auld
Auld (b)
Baranowski
Perry
Study
Portions
Little or no
emphasis on
health messages

(
38
)

This method of synthesis across study
types:


preserves the integrity of the findings of the
different types of studies


allows us to integrate ‘quantitative’
estimates of benefit and harm with
‘qualitative’ understanding from people’s
lives


allows the exploration of heterogeneity in
ways in which it would be difficult to
imagine in advance


protects against ‘data dredging’

(
39
)

Assumptions / theoretical models


Mixed methods synthesis can draw on thinking
relating to mixed methods in primary research


In terms of a frequently used taxonomy of paradigm
stances (
Creswell 2011)
e.g.


Incommensurability (cannot be mixed)


A
-
paradigmatic (can be mixed and matched in different
ways)


Complementary strengths (not incompatible, but are
different and should be kept separate)


Dialectic (paradigms are important in different ways
leading to useful tensions & insights)


Alternative paradigm (‘mixed methods’ paradigm;
foundation in e.g. pragmatism)

(
40
)

Conclusions


Qualitative research offers valuable insights
that can inform policymaking


Methods for synthesising qualitative
research are still being developed


Selection of a particular method depends on
the type of answer required (and the means
by which the review will inform policy)

(
41
)

References


Barnett
-
Page E, Thomas J (2009) Methods for the synthesis of qualitative
research: a critical review.
BMC Medical Research Methodology
, 9:59.
doi:10.1186/1471
-
2288
-
9
-
59. (
http://www.biomedcentral.com/1471
-
2288/9/59
)


Creswell J (2011) Mapping the Developing Landscape of Mixed Methods
Research in
Teddlie

C,
Tashakkori

A: SAGE Handbook of Mixed Methods
in Social &
Behavioral

Research. New York: Sage.


Gough D, Oliver S, Thomas J (2012)
An Introduction to Systematic
Reviews
. London: Sage


Thomas J, Harden A (2008) Methods for the thematic synthesis of
qualitative research in systematic reviews.
BMC Medical Research
Methodology
, 8:45 doi:10.1186/1471
-
2288
-
8
-
45
(
http://www.biomedcentral.com/1471
-
2288/8/45
)


Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G,
Kavanagh J (2004) Integrating qualitative research with trials in systematic
reviews: an example from public health.
British Medical Journal
328
:
1010
-
1012. (
http://www.bmj.com/cgi/content/full/328/7446/1010
)

(
42
)





Systematic Reviews:
The Potential of
Different Methods

Steve Higgins

Henry Potts

James Thomas

Geoff Wong


ESRC Research Methods Festival

Oxford, 5 July 2012


(
43
)

Summary

Heterogeneity

Aggregation /
configuration

Meta
-
analysis

A

problem in fixed effect

Allowance made for in
random effects

For ‘exploration’

䙩硥搠☠牡湤潭

敦晥f瑳
浯摥ms⁡牥 灵牥p慧杲敧慴a潮

卵p
-
杲潵瀠☠&整e
-
牥杲敳si潮⁢潴栠慧杲敧慴攠☠
c潮晩杵牥

卹湴桥nis 潦o煵慬i瑡瑩v攠
牥獥慲rh

b硰散瑥t⸠h敹 c桡ll敮来 is
j畤杩湧 w桥渠c潮瑥硴t ⼠
c潮c数瑳⁡牥 瑯t 摩ssimil慲a
瑯t扥⁣潭灡牥r.

卩浩l慲

c潮o数瑳⁡牥
‘aggregated’; most synthesis
i湶潬v敳 c潮晩杵牡瑩gn

剥慬is琠牥癩敷

乥c敳s慲a 瑯t潢瑡tn

摩f晥牥湴n
c潮瑥硴t 整e

䅧杲敧慴敳 wi瑨t渠䍍n

c潮晩杵牡瑩g湳㬠;潮晩杵牥猠
扥瑷敥渠瑨敭

䵥瑡j湡牲慴av攠牥癩敷

乥c敳s慲a 瑯t潢瑡t渠niv敲ei瑹
in paradigms

Aggregation

within
paradigms; configuration
between them

(
44
)

Your turn

Research
question

Heterogen
-
eity
?

Meta
-
analysis

Qualitative
synthesis

Realist
review

Meta

narrative
review

Research
question 1

Research
question 2