Answers to Frequently Asked Questions

clearsleepingbagSecurity

Nov 30, 2013 (3 years and 7 months ago)

62 views

Meta
-
Analysis of Clinical Data for Regulated
Biopharmaceutical Products:


Answers to Frequently Asked Questions




Brenda Crowe, Research Advisor, Eli Lilly and Company


With special thanks to Jesse Berlin




Midwest Biopharmaceutical statistics workshop

May 21, 2013

Disclaimer


The views expressed herein represent those of the
presenter and do not necessarily represent the views
or practices of the presenter’s employer or any other
party


MBSW May 21, 2013

2

Acknowledgements

Jesse Berlin

Amy Xia

Juergen

Kuebler

Carol Koro

Ed Whalen

MBSW May 21, 2013

3

Agenda


Background


The 6 questions


What studies should be pooled/combined?


Method of ascertainment?


Individual patient data (
vs
. aggregate patient data)?


Multiple looks and/or multiple endpoints?


Heterogeneity of design and results?


Fixed
-
effect models or random
-
effects models?


Concluding remarks


MBSW May 21, 2013

4

Background


During drug development, sponsors need to recognize
safety signals early and adjust the development program
accordingly


Crowe
et al
. (SPERT): overview of the framework and
planning of MA in drug development but did not provide
details regarding practical issues arising during
implementation.


Focus here on common analytical topics (6 questions)


Emphasis on situations that arise in drug development,
mostly premarketing

SPERT = Safety Planning, evaluation and Reporting Team

MBSW May 21, 2013

5

A little vocabulary (in today’s
context)


POOL (noun): a grouping of studies used to address
a specific research question


Swimming in data (avoid drowning)


MBSW May 21, 2013

6

Q1: WHAT STUDIES SHOULD
BE POOLED IN THE META
-
ANALYSIS?

MBSW May 21, 2013

7

Existing Guidance

FDA guidance on
premarketing risk
assessment

MBSW May 21, 2013

8

Existing Guidance

International
Conference on
Harmonization
(ICH) M4E

MBSW May 21, 2013

9

Existing Guidance

Council for
International
Organizations of
Medical Sciences
VI (CIOMS VI)
report

MBSW May 21, 2013

10

What to pool?


Decisions on what to combine depend on the
specific questions to be answered (duh)


Often there are several questions and these might
require different subsets of studies or subjects

MBSW May 21, 2013

11

Pools may be based on


Type of control: placebo vs. active


Dose route or regimen


Concomitant (background) therapy


Methods of eliciting adverse events (e.g., active vs.
passive).


Disease state


Duration of treatment (and follow
-
up?)


Subgroups of patients based on age groups,
geographies, ethnicity groups, or severity of disease,
etc.

MBSW May 21, 2013

12

Table to help pick the right studies

MBSW May 21, 2013

13

Usually exclude


Phase 1 pharmacokinetic and
pharmacodynamic

studies (because short duration, healthy subjects or
patients with incurable end
-
stage disease).


Studies that cannot / will not provide individual
patient level data if required for analysis.


Considerations for inclusion in a pool

MBSW May 21, 2013

14


It is generally most appropriate to combine data from
studies that are similar.


Strong

similarity is not required for pooling, if the
effects of treatment don’t depend on the trial
characteristics being considered.

Considerations for inclusion in a pool

MBSW May 21, 2013

15


Suppose some studies (or arms) were conducted
at a higher dose than the sponsor is proposing
for the marketing label. Would you
exclude

those
arms from the analysis?


Yes, if the goal for those analyses is to
characterize adverse events from proposed
indications at the proposed doses.


However, one might choose to combine the high
-
dose
studies or arms in a different pool to help assess what
could happen in an overdose situation.


For example . . .

MBSW May 21, 2013

16

Studies (or arms) at a higher or lower
dose than proposed for marketing?


In general, exclude dose arms that are lower
than the proposed dose for marketing, as these
may dilute the effects seen at the higher
marketed dose


However, events may occur in the lower dose studies
that should not be ignored


Including low
-
dose and high
-
dose studies may help
understand the dose
-
response relationship

MBSW May 21, 2013

17

AEs in all those who took the drug?


Can analyze ALL who took drug as a single cohort
without a comparator group: useful for accounting for all
events and estimating event rates for infrequent events


Can then be compared to external reference population
rates


However, external population rates limited by the
availability of event rates for a specific subset of the
population that is comparable to the trial population


If the underlying disease increases the risk of a particular event,
comparisons with an external reference could be biased against the study
drug.


Conversely, if enrollment criteria are such that high
-
risk patients are
excluded from trials, the on
-
study rates could appear to be artificially low.

MBSW May 21, 2013

18


What if a safety signal was detected in Phase 2
that resulted in a change in ascertainment of an
AE in Phase 3 (e.g., an adjudication process,
special case report form)?


Create a grouping of Phase 3 studies designed
for that particular event


Advantages


Studies with consistent ascertainment analyzed together


Excludes studies that generated the hypothesis being
tested


Hypothesis generating studies?

MBSW May 21, 2013

19


Previous addresses type I error but


sacrifices statistical power


discards data from what may be studies in a closely monitored
population, which may also be at differential risk due to exposure to
the compound


And it can raise all kinds of red flags (so
transparency is key


do the analysis with and
without those studies)


Hypothesis generating studies (cont.)

MBSW May 21, 2013

20


Do not do a crude
unstratified

analysis that
combines studies with a comparator and
studies without a comparator.


Results can be very misleading. See Lièvre
2002, Chuang
-
Stein 2010 for further information
on dangers of not stratifying.

Caveats

MBSW May 21, 2013

21

Q2: HOW DOES THE METHOD
OF ASCERTAINMENT IMPACT
THE QUALITY OF THE META
-
ANALYSIS?

MBSW May 21, 2013

22

Ascertainment method


Can affect observed event rates, e.g., actively
solicited events will have higher reporting rates than
passively collected events


E.g., for drugs that cross the blood

brain barrier, use
prospective tool to assess suicidal ideation and
behavior (vs.
post hoc

adjudication)


MBSW May 21, 2013

23

Retrospective adjudication

Even with strict criteria using previously collected data,
bias could be introduced by retrospective
adjudication


Important detailed clinical information may be missing


If
post hoc

adjudication is necessary, use an
external, independent adjudication committee that


Is masked to treatment assignment AND


Adjudicates events across the entire development program

MBSW May 21, 2013

24

Q3: WHAT ARE THE
ADVANTAGES OF USING
INDIVIDUAL PATIENT DATA
(VS. AGGREGATE
SUMMARIES)?

MBSW May 21, 2013

25

Individual or aggregate
-
level data?


For many questions get same answer with IPD as with
APD


For analyses that do not
require
patient
-
level data,
including
all

relevant studies improves precision


May also reduce bias that could be introduced by limiting
the analysis to those where patient
-
level data are
available


However, there can be advantages to IPD


Much easier to detect interactions between treatment and
patient
-
level characteristic with IPD than with APD


MBSW May 21, 2013

26

Advantages of patient
-
level data


Allows mapping all data to a common version of
MedDRA (or other) increasing consistency of
terminology across trials


Generally permits creation of common variables
across trials


E.g., age categories may have been defined using different
category boundaries


Different threshold hemoglobin values may have been used to
define ‘anemia’

MBSW May 21, 2013

27

More advantages of IPD


Allows specification of a common set of patient
-
level
covariates so subgroup analyses across trials can be
performed


Can define outcomes based on combinations of
variables defining specific events but that may
indicate a common mechanism, e.g., a combination
of weight loss or appetite reduction

MBSW May 21, 2013

28

And still more advantages of IPD


Post hoc

analyses of outcomes that require
adjudication can sometimes be derived, as in the
case of suicide event grading according to Columbia
Classification Algorithm of Suicide Assessment (C
-
CASA criteria)


Creation of time
-
to
-
event variables (may not be
available in publications)


Flexibility in defining time periods of interest for analyses, e.g.,
events occurring during “short
-
term” follow
-
up


MBSW May 21, 2013

29

Why not always use IPD?


Integration required to provide the database is labor
intensive, especially if done in retrospect


Sometimes summary statistics may be the only
information available for some studies of interest,
e.g.,


studies of a new therapeutic approach done by an academic group
that does not share patient
-
level data, or


the drug of interest may have been included as an active control by
another sponsor

MBSW May 21, 2013

30

Q4: SHOULD WE ADJUST FOR
MULTIPLE LOOKS AND/OR
MULTIPLE ENDPOINTS IN THE
CONTEXT OF META
-
ANALYSIS?

MBSW May 21, 2013

31

Q4: Multiple comparisons


Complicated by having multiple looks over time and
multiple (and an unknown number of) endpoints


Safety Planning, Evaluation, and Reporting Team
(SPERT) defined “Tier 1 events” as those for which a
prespecified hypothesis has been defined

MBSW May 21, 2013

32

Tier 1 Events


E.g., to rule out an effect of a certain magnitude for
assessing a particular risk (a noninferiority test


as
for diabetes drugs)


Generally, should consider performing formal
adjustment for multiple looks for Tier 1 events and
for multiple endpoints for other events

MBSW May 21, 2013

33

Diabetes drugs


Need to rule out a relative risk of 1.8 (for CV events) for
conditional approval, and 1.3 for final approval


Confidence level for that
specific
outcome may need to
be adjusted for multiple looks, which can be considered
separately from non
-
Tier 1 events because it needs to be
met for the drug to move forward


An event of interest: important regardless of the specific
side effect profile and


Analogous to a primary analysis in the efficacy setting


MBSW May 21, 2013

34

Multiplicity is a complicated issue in
the safety context


Often have low power, lack of
a priori

definitions, and
extraneous variability


Value in trying not to miss a safety signal, but
remember that initial detection is not the same as
proving that a given AE is definitively related to a
given drug


Worry about reducing false negative findings in drug
safety given the known limitations of our tools

MBSW May 21, 2013

35

Q5: WHAT IS HETEROGENEITY
AND WHAT ARE SOURCES OF
HETEROGENEITY?

MBSW May 21, 2013

36


Heterogeneity refers to differences among studies
and/or study results.


Can be classified in 3 ways: clinical, methodological
and statistical.

MBSW May 21, 2013

37

Differences among trials in their


Patient selection (e.g., disease conditions under
investigation, eligibility criteria, patient
characteristics, or geographic differences)







Clinical Heterogeneity

MBSW May 21, 2013

38

Differences among trials in their


Interventions (e.g., duration, dosing, nature of
the control)





Outcomes (e.g., definitions of endpoints, follow
-
up duration, cut
-
off points for scales)

Clinical Heterogeneity

MBSW May 21, 2013

39

Differences in


Study design (e.g., the mechanism of randomization).


Study conduct (e.g., allocation concealment, blinding,
extent and handling of withdrawals and loss to follow up,
or analysis methods).

Decisions about what constitutes clinical
heterogeneity and methodological heterogeneity do
not involve any calculation and are based on
judgment.

Methodological Heterogeneity

MBSW May 21, 2013

40


Numerical variability in results, beyond expected
by sampling variability

May be caused by


Known (or unknown) clinical and methodological
differences among trials


Chance

Statistical heterogeneity

MBSW May 21, 2013

41

Hypothetical example

MBSW May 21, 2013

42


Clinical heterogeneity may not always result in
statistical heterogeneity.


If there is clinical heterogeneity but little variation in
study results, may represent robust, generalizable
treatment effects.



MBSW May 21, 2013

43

Beware of Q

(unless you are James Bond)


Cochran’s Q is a global test of heterogeneity


I
2
is a measure of global heterogeneity


KEY POINT
: They are informative, but rely on neither of
these statistics


Apparent lack of
overall

heterogeneity does not rule out a
specific source of heterogeneity


Conversely, large studies with clinically small variability
can yield spuriously high
statistical
heterogeneity

MBSW May 21, 2013

44

Q6: IS IT SUFFICIENT TO USE
FIXED
-
EFFECTS MODELS WHEN
COMBINING STUDIES OR DO WE
NEED TO CONSIDER RANDOM
-
EFFECTS MODELS?

MBSW May 21, 2013

45

Fixed
-
effect vs. random
-
effects


Fixed = common effect across all studies


Inference is to the studies at hand


Reasonable to expect (?) when designs and populations
are similar across studies


Random
-
effects models: true underlying
population effects differ from study to study and
that the true individual study effects follow a
statistical distribution


The analytic goal is then to estimate the overall mean
and variance of the distribution of true study effects

MBSW May 21, 2013

46

More on FE vs. RE


In some situations, it may not be appropriate to
produce a single overall treatment
-
effect estimate


Goal should sometimes (often) be to model and
understand sources of heterogeneity



MBSW May 21, 2013

47

More points on FE vs. RE


Risk differences more heterogeneous than odds ratios
(OR) or relative risks (RR, a point that is also made in an
FDA’s draft guidance for industry on noninferiority trials)


Can model on OR scale then convert to RD or RR to help
with clinical interpretability


Constant OR implies effect size
must

vary for RD, so
-

must decide whether to estimate the baseline (control)
event rate from the external data or from the data
included in the actual meta
-
analysis (implications for
variance estimation)

MBSW May 21, 2013

48

How to decide on FE or RE?


Do you expect a common effect or not?


Single indication, similar protocols, same data collection
methods, definitions, etc., FE likely to be appropriate.


Different populations, etc., use RE but ALSO explore
sources of heterogeneity


Enough data?


Sparse data, few studies, may not permit RE estimation


Small studies may get “up
-
weighted” with RE:
are small study results systematically different?

MBSW May 21, 2013

49

Once you go Bayesian, you’ll
never go back


Specify a prior probability distribution


Today’s posterior becomes tomorrow’s prior


Flexibility to deal with heterogeneity through complex
modeling


Available under both FE and RE (use Deviance
Information Criterion to decide?)


Bayesian inferences are based on the full ‘exact’
posterior distributions (so useful for small numbers of
events)

MBSW May 21, 2013

50

For more details …

MBSW May 21, 2013

51


Meta
-
analysis increasingly used to address safety
concerns in drug development.


Up
-
front thought allows teams to improve planning
and enhance data capture, and enhances
transparency and interpretation of the results.

Concluding Remarks

MBSW May 21, 2013

52


Christy Chuang
-
Stein, and Mohan
Beltangady
.
Reporting
cumulative proportion of subjects with an adverse event based
on data from multiple studies.

Pharmaceut
. Statist. 2010


Crowe, Xia, Berlin et al.
Recommendations for safety planning,
data collection, evaluation and reporting during drug, biologic
and vaccine development: a report of the safety planning,
evaluation, and reporting team.
Clin

Trials 2009; 6 430
-
440


Lièvre,
Cucherat

and
Leizorovicz
.
Pooling, meta
-
analysis, and the
evaluation of drug safety.
Current Controlled Trials in
Cardiovascular Medicine
2002


Olkin

I, Sampson A.
Comparison of meta
-
analysis versus
analysis of variance of individual patient data.

Biometrics.
Mar
1998;54(1):317
-
322.



Additional References

MBSW May 21, 2013

53