Do cross-sectional student assessment

hardtofindcurtainUrban and Civil

Nov 16, 2013 (3 years and 4 months ago)

54 views

Do cross
-
sectional student assessment
data make a reasonable proxy for
longitudinal data?

Elena Zaitseva and Mantz Yorke

Liverpool John Moores University


HEIR Conference, University of Liverpool

2nd year underperformance


2
nd

year students are characterised as a cohort where ‘
many suffer
from reduced motivation or apathy, declining grade point averages,
or a letdown from their first year
’ (Pattengale & Schreiner, 2000)




Underperformance of 2
nd

year students is well researched by
North American scholars
; the topic receives less attention in the
UK and Europe



‘The Forgotten Year: Tackling the ‘Sophomore Slump’ is a HEA
funded institutional research project

http://secondyearexperience.ljmu.ac.uk/






Project:





Examines patterns in student performance across the levels of
study
and longitudinal dynamics of slump (quantitative research)





Investigates underlying factors behind the 2
nd

year students’
underperformance and diminished engagement (qualitative
research )




Develops and evaluates interventions
to enhance experience of
the 2
nd

year cohort








S
lump is a dynamic phenomenon

Longitudinal
performance of
programmes

with 20+ students
















2008
-
09

2009
-
10

2010
-
11

N

Consistent

non
-
slumpers

no slump

no slump

no slump

14

Consistent

slumpers

slump

slump

slump

8

‘Recovered’

slump

slump

no slump

2

‘In and out’

no slump

slump

no slump

4

‘Plunged into
recession’

no slump

slump

slump

6

New
slumpers

no slump

no slump

slump

2

Instead of waiting for a programme’s longitudinal data,

can we use cross
-
sectional student performance data

to predict slump?

Data from 08
-
09 and 09
-
10

Data from 09
-
10 and 10
-
11

Number of valid records

(56 programmes, N=20+ students)


08
-
09 Level 1 3006

08
-
09 Level 2 2519

09
-
10 Level 1 2975

Number of valid records

(49 programmes, N=20+ students)


09
-
10 Level 1 2807

09
-
10 Level 2 2385

10
-
11 Level 1 2984

Valid records:


Students who were awarded marks in respect of 120 credits at the Level stated


Exception to the above:

Social Work where 36 credits are awarded on pass/fail basis, i.e. 84 credits with marks

The data used

Starters

AY2008
-
09

Starters

AY2009
-
10

Starters

AY2010
-
11

Year

1

Year

2

Year

1

Year

1

Year

2

Longit

Longit

X
-
sect

X
-
sect

AY2008
-
09

AY2009
-
10

AY2010
-
11

B

C

A

True

cohort

True

cohort

Quasi
-


cohort

Quasi
-


cohort

The analytical frame


Cross
-
sectional



No slump

Slump

Longitudinal

No slump

22

11

Slump

4

12

Match

69.4 per cent

Calculating the match


Analysis A

Starters

AY2008
-
09

Starters

AY2009
-
10

Starters

AY2010
-
11

Year

1

Year

2

Year

1

Year

1

Year

2

Longit

Longit

X
-
sect

X
-
sect

AY2008
-
09

AY2009
-
10

AY2010
-
11

69.6%

match

69.4%

match

69.4%

match

The outcomes

Starters

AY2008
-
09

Starters

AY2009
-
10

Starters

AY2010
-
11

Year

1

Year

2

Year

1

Year

1

Year

2

Longit

Longit

X
-
sect

X
-
sect

AY2008
-
09

AY2009
-
10

AY2010
-
11

+ 1.53%

+ 1.49%

+ 1.40%

A complexity

Starters

AY2008
-
09

Starters

AY2009
-
10

Starters

AY2010
-
11

Year

1

Year

2

Year

1

Year

1

Year

2

Longit

Longit

X
-
sect

X
-
sect

Acad Year

2008
-
09

Acad Year

2009
-
10

Acad Year

2010
-
11

73.2%

match

81.6%

match

67.3%

match

Revised outcomes

Academic Year

2008
-
09

Academic Year

2009
-
10

Year 1 mean

Year 2 mean

Year 2 mean

Cross
-
sectional slump

Mark

Year 1 mean

A paradox
(not the only one)

What proportion of students enrolled on the programme

are ‘captured’ by the most popular 120 credits?

The question of homogeneity

Module

Max N
students

Enrolled on
module

Fraction of Max N


'Top 10' fraction
sum

Homogeneity
Index

xx01

78

78

1





xx02

78

78

1





xx03

78

78

1





xx04

78

78

1





xx16

78

50

0.64





xx16

78

50

0.64





xx13

78

42

0.54





xx13

78

42

0.54





xx12

78

37

0.47





xx14

78

35

0.45

7.28

0.73

xx14

78

35

0.45





xx07

78

29

0.37





xx15

78

29

0.37





xx15

78

29

0.37





xx06

78

26

0.33





xx05

78

23

0.29





xx08

78

17

0.22





xx10

78

12

0.15





xx09

78

10

0.13





xx11

78

2

0.03





A worked example

Programme

N with 120
credits Yr1+2

Homog

Yr1

Homog

Yr2

Applied Social Science 1

82

1.00

0.87

Applied Social Science 2

97

1.00

0.92

Applied Social Science 3

74

1.00

0.80

Applied Social Science 4

149

1.00

0.86

Arts 1

45

1.00

0.62

Arts 2

55

0.99

0.71

Education 1

21

1.00

1.00

Education 2

78

1.00

0.73

Humanities & Social Science 1

26

0.99

1.00

Humanities 1

47

1.00

0.92

Humanities 2

66

1.00

0.75

Science 1

27

0.98

0.94

Science 2

145

1.00

0.99

Science 3

26

1.00

0.80

Technology 1

63

1.00

1.00

Selected results from homogeneity analysis

Programme

N with 120
credits Yr1+2

Homog

Yr1

Homog

Yr2

Applied Social Science 1

82

1.00

0.87

Applied Social Science 2

97

1.00

0.92

Applied Social Science 3

74

1.00

0.80

Applied Social Science 4

149

1.00

0.86

Arts 1

45

1.00

0.62

Arts 2

55

0.99

0.71

Education 1

21

1.00

1.00

Education 2

78

1.00

0.73

Humanities & Social Science 1

26

0.99

1.00

Humanities 1

47

1.00

0.92

Humanities 2

66

1.00

0.75

Science 1

27

0.98

0.94

Science 2

145

1.00

0.99

Science 3

26

1.00

0.80

Technology 1

63

1.00

1.00

Selected results from homogeneity analysis