Early Childhood Activities - Institute for Policy Research

bricklayerbelchedInternet και Εφαρμογές Web

5 Φεβ 2013 (πριν από 4 χρόνια και 6 μήνες)

145 εμφανίσεις

Where are we going?


What to do if no RCT, RD, ITS or sophisticated
matching is possible?


We describe and analyze principle of pattern
matching to improve the basic workhorse design by a
feature other than high quality matching


Illustrate what you might do if there is no pretest at
all and so not even a work
-
horse design is possible



Before that, a bit of a summary

Yesterday 1


Do not match from extremes unless forced to


Then match using a reliable set of measures,
moving to propensity score framework, latent
variables, or statistical reliability
-
adjustments
to handle unreliability


But you will still have problem of possible
specification error!

Yesterday 2


In selecting non
-
equivalent control group:


Use local focal matching to reduce the degree of
initial non
-
comparability though you cannot expect
total non
-
comparability


Sometimes the control group so formed will not
differ from what would have been achieved with
random assignment, at least on observables


At other times, the initial group non
-
equivalence will
be reduced for when you come to do statistical
analysis to “control” initial differences

Yesterday 3


When there is initial non
-
equivalence on observables:


Theory and empiricism about the constructs (covariates)
in the “true”selection model helps, as does careful
measurement thereof


Measures in multiple other domains helps too,


Unclear whether ANCOVA or propensity scores do better
-
-
Shadish et al, Glazerman et al.
--
though demographic
variables alone and Heckman IV models do not fare well.
Propensity scores preferred on theory alone.


You can never be sure of the final causal conclusion,
though


Implication


In designing research you do well to avoid the
workhorse if you can, though it is modal in current
educational practice


Can you add prior pretest waves?


Can you add any of the other design elements, some
mentioned in our ITS discussion and others we
discuss today


How can you design your way out of reliance on a
simple design with non
-
equivalent groups and
pretest and posttest?

When there is no Pretest on the same
scale as Outcome


Do randomized experiment
--
Abcedarian, Perry
-
Preschool, Head Start, Early Head Start, CLIO, Even
Start and Sesame Street


If not possible, do everything possible to make
control group focally local


Add design elements to rule out alternative
explanations of a possible causal relationship


Here’s an example

Minton’s Dissertation


Object: Evaluate Sesame St in 1st year


Problem 1: Program already launched


Problem 2: No pretest possible


Problem 3: No money for original data collection


Setting: One kindergarten in NJ that built SS into its
day and that has records on children and their
families plus annual PPV assessment



Question 1: What control group is
possible?


What control group to find, given program was very
popular in its first year.


Why is popularity a problem?


Neighborhood kids who did not view


Next
-
door kids of same age who not view?


Older siblings in general


Older sibs attending same kindergarten within last N
years


Older sibs attending same kindergarten last 2 years


Older
control
siblings

Younger
Sesame Street
siblings

Achievemen
t scores

Minton (1975) Sesame Street Study
-

1

Older
control
siblings

Younger
Sesame Street
siblings

Achievemen
t scores

Minton (1975) Sesame Street Study
-

2

Letter
skills

Non
-
letter
PPVT skills

Older
control
siblings

Younger
Sesame Street
siblings

Achievemen
t scores

Minton (1975) Sesame Street Study
-

3

Letter skills


high viewers

Non
-
letter skills

low viewers

Non
-
letter skills

high viewers

Letter skills


low viewers

What has happened here?


Single causal hypothesis of SS effective made
to have multiple data implications


These are meant to rule out alternative
hypotheses and not to recreate same bias


These implications here in the form of a
difference in difference in differences


Collect data and test hypothesis

Another Example




How the Introduction of TV affected Library
Circulation

19
45

1975

Library
circulation per
capita

Parker et al. (1966) Effects of TV
-

1

19
49

19
53

Short interrupted time series

Fiction book
circulation

19
45

1975

Library
circulation per
capita

Parker et al. (1966) Effects of TV
-

2

19
49

19
53

Short interrupted time series with control

Early TV communities

Late TV communities

19
45

1975

Library
circulation per
capita

Parker et al. (1966) Effects of TV
-

3

19
49

19
53

ITS with switching replication

1949 interruption

1953 interruption

19
45

1975

Library
circulation per
capita

Parker et al. (1966) Effects of TV
-

4

19
49

19
53

ITS with switching replication and control

Fiction

Fact

Fact

Fiction

What has happened here?


Combine an ITS with non
-
equivalent DVs and
switching replications


What alternative interpretations can you come up
with?


How plausible are these?


Have we seen this before with RD and ITS?


One general causal hypothesis has multiple
implications in the data


Predicted hypothesis as multiple differences of
differences; as higher order interactions

Reynolds and West’s (1987) “Ask for the
Sale” Experiment

From all stores selling lottery tickets, some stores
volunteered (or not) to post a sign reading “Did we
ask you if you want a Lottery ticket? If not, you get
one free”. So this is a basic nonequivalent control
group design, with the control matched on zip
code, store chain, and pretest ticket sales.

NR


O1 X O2

-----------------------------

NR


O1 O2

The Outcome of the Basic Design


But there might be many reasons besides
treatment that caused treatment group sales
to rise.

Adding a Nonequivalent DV


They added three
nonequivalent dependent
variables
, showing that the intervention
increased ticket sales but not sales of gas,
cigarettes, or grocery items.

Adding Multiple Baselines
-

recasting
as ITS Design


They located some stores in which the
treatment was initiated later than in other
stores
,
or initiated and then removed,
and
found that the outcome tracked the
introduction of treatment over time while
sales in the matched controls remained
unchanged


Adding Multiple Pretests and Posttests


They added
multiple pretests and posttests

by examining mean weekly ticket sales for
four weeks before and four weeks after the
treatment started.

The Point is:


To use the choice of additional design
elements to rule out more alternative
interpretations, hopefully all that can be
currently identified


The goal is ruling out plausible alternative
interpretations, and it can also be reached via
keeping the pattern of results constant but
varying the number and type of comparisons
involved

Main NAEP 4
th

grade math scores

by year and proficiency standards

D & J Results: 4
th

Grade Math

Main

NAEP 4
th

grade math scores by year:

Public and Catholic schools

Main

NAEP 4
th

grade math scores by year:

Public and Other Private schools

Trend NAEP 4
th

grade math scores by year:

Public and Catholic schools

Student Enrollment

Catholic

Other Private

Public

1994

5.73

4.72

89.55

1996

5.67

4.74

89.60

1998

5.58

4.87

89.56

2000

5.38

4.81

89.81

2002

5.26

5.13

89.61

2004

4.88

4.93

90.18

2006

4.56

5.07

90.37

Warning!


This pattern matching strategy requires:


Clear causal hypothesis
-

relevance to
discontinuity


Careful measurement
-

reliability and ceiling
or floor effects


Large samples (or large effects) because
hypothesis is of a complex statistical
interaction


How lucky Minton was!

Examples from you of the Basic Work
-
Horse Design


Let us take some from you and see if they can
be improved by adding design elements.

Design Elements to be combined:
Assignment


Random Assignment


Cutoff
-
Based Assignment


Researcher
-
controlled Matching
--

of many
kinds in econometric literature

Design Elements to be
combined:Treatments


Switching Replications


Reversed Treatments


Removed Treatments


Repeated Treatments

Design Elements to be combined:
Measurement


Single Pretest


Pretest Time Series


Proxy Pretests


Retrospective Pretests


Moderators with predicted Interactions


Measuring Threats to Validity

Design Elements to be combined:
Comparison Groups


Single Non
-
Equivalent Groups


Multiple Non
-
Equivalent Groups


Twins/Siblings


Cohorts


Other Focal, Local Comparison Groups


Golden Rules (1)


You can’t put right through statistics what you
have done wrong by design


Statistical adjustments work better the less
non
-
equivalence there is to adjust away in the
first place


Since the work horse is so prevalent but so
problematic, how can we complexify the
design through adding design elements

Golden Rules (2)


First, Do an experiment; if not


Do Regression
-
discontinuity study. If not,


Do ITS with some sort of a comparison series.
If not


Do study combining multiple design element,
preferably with focal local intact controls, case
matching on many covariates, reintroduction
of treatment at new time, non
-
equivalent DVs,
etc.

Golden Rules (3)


Don’t be bamboozled by fancy models in
Greek clothing. Always translate them into
structural design elements before evaluating
their likely validity. That will reveal what you
have got


Remember you only control for the reliably
measured part of any construct, not the
construct itself.



Evaluation, formative

On a scale from 1 to 6, with 6 being high, please rate the
following and then indicate how you would improve what we
did.

Contact with Valerie about the workshop

Accommodations

Food

Curriculum Content

Curriculum Relevance to your current or anticipated work

Quality of Instruction

Any other Suggestions for Improvements?



Grant Opportunities

at the Institute of Education Sciences

Allen Ruby, Ph.D.

Associate Commissioner

Policy & Systems Division

National Center for Education Research

Take Away


IES has a small number of education grant programs


Matrix of topics/goals within grant programs


Range of work: exploration, development, evaluation


Focus is on improving student outcomes


Competitions are held twice a year (6/24/10 and 9/16/10)


Competitive peer review using an absolute standard



Take advantage of available information


http://ies.ed.gov (Request For Applications, abstracts of
projects, webinars, resources for researchers webpage)


Program officer
(listed in RFA)


IES Structure

Office of the
Director

National Board
for Education
Sciences

National
Center for
Education
Research

National
Center for
Education
Evaluation

National
Center for
Education
Statistics

National
Center for
Special Ed
Research

Standards &
Review

Overall Research Objectives


Develop or identify education interventions
(practices, programs, policies and approaches)
that enhance academic achievement and that
can be widely deployed



Identify what does
not

work and thereby
encourage innovation and further research



Understand the processes that underlie
variations in the effectiveness of education
interventions


Final Outcomes of Interest are for Students

Preschool


School readiness


Developmental outcomes for infants and toddlers with
disabilities

Kindergarten through Grade 12


Academic outcomes in reading, writing, math and science


Behaviors, interactions, and social skills that support learning in
school and successful transitions to post
-
school opportunities


High school graduation


Functional skills for independent living of students with
disabilities


Postsecondary:
enrollment, persistence, and completion

Adult Education
: reading, writing, and math (basic, secondary,
and EL)

Key Dates

Application
Deadline

Letter of Intent

iesreview.ed.gov

Application
Package

www.grants.gov

Start
Dates

6/24/10

4/29/10

4/29/10

3/1/11

to
9/1/11

9/16/10

7/19/10

7/19/10

7/1/11

to


9/1/11

Research and Research Training Grant
Programs


Education Research Grant Programs


Special Education Research Grant Programs



Statistical and Research Methodology in Education


Evaluation of State and Local Education Programs
and Policies


Postdoctoral Research Training Grant Programs


National Research and Development Centers


Education Research Grants Programs (84.305A)


Reading and Writing


Mathematics and Science Education


Cognition and Student Learning


Teacher Quality (Reading & Writing; Math & Science)


Social and Behavioral Context for Academic Learning


Education Leadership


Education Policy, Finance, and Systems


Postsecondary Education


English Learners


Early Learning Programs and Policies


Education Technology


Adult Education


Organization and Management of Schools and Districts


Analysis of Longitudinal Data to Support State & Local



Education Reform


Special Education Research Programs (84.324A)


Early Intervention and Early Childhood Special
Education


Reading, Writing, and Language Development


Mathematics and Science Education


Social and Behavioral Outcomes to Support Learning


Cognition and Student Learning in Special Education


Professional Development for Teachers and Related
Service Providers


Special Education Policy, Finance, and Systems


Transition Outcomes for Special Education
Secondary Students


Autism Spectrum Disorders

5 Research Goals

1.
Exploration: examine relationship between
malleable factors and education outcomes

2.
Development and Innovation:

i
teratively develop
new or improved education interventions

3.
Efficacy & Replication: evaluate the efficacy and
feasibility of an intervention


Efficacy Follow
-
up Studies

4.
Scale
-
up Evaluation: evaluate the impact &
feasibility of interventions at scale


Scale
-
up Follow
-
up Studies

5.
Measurement: develop and validate assessments

Exploration


Explore malleable factors that are associated
with better student learning and education
outcomes


Malleable factor: something that can changed by
the education system
-

student, teacher, or school
characteristics, or an education program or policy


Explore factors that mediate or moderate the
relationship between malleable factors and
student outcomes


Small primary data studies, secondary


analyses, and meta
-
analyses



Development and Innovation


Develop new interventions (e.g., instructional
practices, curricula, teacher professional
development, policies)


Demonstrate the feasibility of the intervention for
implementation in an authentic education delivery
setting


Collect pilot data on promise of intervention to
achieve intended outcomes

Efficacy and Replication


Test whether or not fully developed
interventions are effective under specified
conditions and with specific types of students


Take place under supportive conditions , e.g.
homogenous sample, high assistance


Studies using random assignment to
intervention and comparison conditions are
preferred where feasible


New this year: Efficacy follow
-
up studies

Scale
-
up Evaluation


Test whether interventions are effective when
implemented
under typical conditions
.


As implemented by practitioners and with
sufficiently diverse samples to support
generalizability.


Studies using randomized assignment to
treatment and comparison conditions are
preferred whenever they are feasible.


Added this year: Scale
-
up follow
-
up studies

Measurement


Develop and validate assessments or other
measurement tools


Typically to be used by practitioners (e.g.,
screening, progress monitoring, and outcome
assessment) but can also be for researcher use


Validation of non
-
student measures must involve
student outcomes (e.g. Teacher Quality)


Program specific, e.g., cost
-
accounting under
Education Policy, Finance, and Systems


Not for evaluating an assessment used as an
intervention


The measure is the primary product

Maximum Grant Duration and Typical Size


Grants include both direct and indirect costs



Exploration: 2 years, $100,000 to $350,000 per yr 10%


Up to 4 years & $400,000 for primary data collection


Development: 3 years, $150,000 to $500,000 per yr 50%


Efficacy: 4 years, $250,000 to $750,000 per yr


26%


Follow
-
up: 3 years, $150,000 to $400,000 per yr


Scale
-
up: 5 years, $500,000 to 1.2 million per yr 2%


Follow
-
up: 3 years, $250,000 to $600,000 per yr


Measurement: 4 years, $150,000 to $400,000 per yr 12%

Information Sources


IES website: http://ies.ed.gov


Obtain IES Request for Applications


Describes type of research that can be done and substance
of what to include in application


http://ies.ed.gov/funding


IES Grants.Gov Application Submission Guide


Step by step instructions on how to apply


http://ies.ed.gov/funding


Obtain Application from and Submit Application to


www.grants.gov


84.305A (regular education) & 84.324A (special education)


Initial IES Contact (feel free to contact)


Allen.Ruby@ed.gov


Handout includes information covered in this presentation