Phase 2 Final Report - Jisc

grapeafterthoughtSoftware and s/w Development

Dec 14, 2013 (4 years and 18 days ago)

115 views

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

1

UCAS
-
Learning Record
s

Service
Phase 2
Pilot

Case Study Report:
August
201
1

Executive Summary

This report describes
the second phase of
one of a series of pilot projects commissioned by
the
Higher Education Advisory G
roup for the
Learning Records Service (LRS)
. These
aimed

to

assess
the benefits of adoption by the HE sector of the Unique Learner Number

(ULN)
:

both in its own right, and as the key to

LRS
Personal Learning Records of HE applicants and
students.

T
h
e

UCAS
Phase 1 and 2
pilot
s
w
ere

designed

as part of the series,

to develop
electronic
links to the LRS in order to evaluate the benefits for UCAS and its key stakeholders of
sharing data on UCAS applicants.
Phase 1
demonstrated a
potential increase in accu
racy
of
GCSE qualification data if obtained from the LRS, compared with direct learner entry.

The

aim of the
Phase 2
pilot
was to actually acquire such data on real learners via a live link, and
to investigate the results. This involved the
population of a

trial version of UCAS Apply

with
v
erified GCSE results

belonging to
volunteer Year 12 students,
obtained directly from the
LRS

via a newly developed web service interface
;

followed by
evaluation of the process by

the learners themselves.

Of the 77
student

participants
from 6 schools, most
were intending UCAS applicants
and

just
under half had started
their
application
. They showed l
ittle
previous
knowledge of the LRS or
ULN
. The pilot

overcame

some significant technical challenges,
but

was
ultimately
highly
successful in using the
new
web service
to obtain
participant
data from
the
LRS
, and
in
using the data to populate a trial version of Apply.
Problems experienced by a few
participants
resulted from incomplete LRS records because of too recent regist
ration
.

The
GCSE results
entered into Apply automatically from the LRS
were
perceived

by
student
participants
as
over

95% complete and 94% correct
. They were generally successfully
flagged as ‘Verified’, reflecting a more authoritative

source than learners

themselves.
Students

were
, however,
able to amend results manually, with
consequent
loss of the
‘Verified’ flag.

Although

results were

overwhelmingly correct,
around 1/3
o
f
students

reported some results
as being listed in the trial Apply
under
the
wrong

heading
. This is
as a result of known
mapping issues
, which are recommended for further investigation
.

Student and staff p
articipants
greatly preferred
automatic capture of verified results

from the
LRS
to the current system of manual entry, but m
ost wan
ted the opportunity to amend

these
results
, with loss of verification
.

The v
ast majority (>97%)
of
student
participants
and all of
the school staff involved
found
the trial Apply
system useful or
very
useful
,

and hence
provided a m
ajor endorsement of

the work of the pilot.

Based on these results, recommendations for future work include

further investigation of the
data mapping issues; convergence in data standards development between UCAS and the
LRS;
extension of the pilot Apply system to allow retr
ieval from the LRS of verified ULNs;
and promotion by
UCAS amongst schools and universities of the potential benefits of
adoption of the ULN as a key to student records and
a
facilitator of UCAS application
.


grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

2

Contents

1.

Introduction

................................
................................
................................
.........

3

2.

Acknowledgements

................................
................................
.............................

3

3.

Background

................................
................................
................................
.........

4

4.

Objectives

................................
................................
................................
...........

5

5.

The challenge

................................
................................
................................
......

5

6.

Established practice

................................
................................
............................

5

6.1.

Current use of UCAS Apply to enter ULN and demographic data

........................

5

6.2.

Current use of Apply to ent
er GCSE qualification data

................................
........

6

7.

Changes evaluated by the pilot

................................
................................
...........

7

8.

Schools and students involved in the evaluation

................................
..................

9

9.

Results of
the student evaluation

................................
................................
.......

10

9.1.

HE Application experience/intentions of participants

................................
.........

10

9.2.

Knowledge of the Learning Records Service and ULN

................................
......

11

9.3.

Ability of the trial Apply system to retrieve data from the LRS

............................

12

9.4.

Completeness and correctness of classification of the results obtained

.............

13

9.5.

Verification of results and their accuracy

................................
...........................

14

9.6.

Effect of manual modifica
tion of qualification results

................................
..........

15

9.7.

Participants’ evaluation of the trial Apply system

................................
...............

16

10.

Results of the staff evaluation

................................
................................
............

17

10.1.

Assistance given by staff to students in confirming Level 2 qualifications

..........

17

10.2.

Staff knowledge of the Learning
Records Service and ULN

..............................

18

10.3.

Use of ULN by schools

................................
................................
......................

19

10.4.

Staff evaluation of the trial Apply system

................................
...........................

20

11.

Conclusions

................................
................................
................................
.......

21

12.

Issues for UCAS

................................
................................
................................

21

13.

Recommendations

................................
................................
............................

23

Appendix 1. Technical Report: UCAS Apply
-

Integration and use of the LRS Web Service
24






grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

3

1.

Introduction

This report describes
the second phase of
one of a series of pilot projects
commissioned by
the Learning Records Service (LRS


previously MIAP) Higher Education Advisory

Group to
evaluate the benefits of adoption by the HE sector of the Unique Learner Number, both in its
own right, and as the key to LRS Personal Learning Records of HE applicants and students.
Matched funding has been provided by HEFCE and administered by
JISC for the pilot, which
ran

from January to
July

201
1
. The UCAS
-
LRS pilot
s

ha
ve

been regarded as of particular
significance
for engagement of the sector,
because

of the pivotal role of UCAS in HE
admissions. The report describes the results obtained from the pilot and some of the
challenges faced; and makes recommendations for system development.

2.

Acknowledgements

The valuable contribution of a number of colleagues

is acknowledged in the planning and
execution of this pilot:

Nick Pendrigh

UCAS User Interface Developer

Development

Andrew Ching

UCAS User Interface Resource Pool Team
Leader

Planning and development

Bethanie Williams

UCAS Deputy Director of Policy
and
Research

Progress chasing

Helen Thorne

UCAS Director of Policy and Research

Executive support

Matt Rodda

UCAS Interim Policy Manager

Editorial comments

Dr Mike Coulson

LRS Business Development Manager

LRS liaison, advice &

support,
participation in evaluation
sessions

Jennifer Powles

LRS Senior Account Manager

LRS liaison, (main contact),
participation in evaluation
sessions

Ann Watson

LRS Deployment Manager

LRS identity verification support;
participation in evaluation
sessions

Carole Morley

LRS Deployment & Support Manager

Data Sharing Agreement,
participation in evaluation
sessions

Mario Ferelli

HEFCE; Chair of Pilot Projects Steering
Group

Project extension & finance

Ruth Drysdale

JISC e
-
Learning Programme Manager

Planning and advice

School and College
Staff involved in
organising and
supervising
evaluation
sessions

Cheltenham Bournside School & Sixth
Form Centre

Mr. Mike Williams

Chase Terrace Technology College,
Burntwood, Staffordshire

Mr. Tim Dowling

Ms.
Kim Woolley

Castle Sixth Form, Kenilworth School and
Sports College, Warwickshire

Mr. Nick Mummery

Ms. Liz Hadley

Stoke Park School &

Community Technology College, Coventry

Ms. Sandra Arch

Ms. Noreen Woodall

The Cotswold School, Bourton
-
on
-
the
-
Water,

Gloucestershire

Mr. Will Morgan

Ms. Jane Tanner

Pate's Grammar School, Cheltenham

Ms. Helen Hooper

Ms. Ruth Lucas

Mr. Chris Beal

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

4




3.

Background

Each year UCAS
typically
processes applications from over 6
5
0,000 applicants to full
-
time
undergraduate courses in the UK,
of which well over 4
7
0,000 are successful. Over 99% of
such applications, to over 3
0
0
1

separate HE institutions

(HEIs)
, are carried out via the UCAS
on
-
line application system, Apply.
Informa
tion on applicants, collected via Apply, is
transmitted to the relevant HEIs electronically via UCAS’ link products; while decisions and
feedback on applications are returned via the same electronic links, and may be viewed by
applicants in the on
-
line UCA
S Track service.

UCAS allocates a unique identification number to each applicant, which is passed to the HEI
with the application
, but has no automatic currency outside UCAS
. The Apply interface
allows applicants to provide demographic data needed for an
application
;

as well as
results
data on qualifications taken, a personal statement, and linked references (which applicants
can view but not change).

Results of qualifications entered by applicants are unverified and are transmitted
as such
to
HEIs, howev
er a more rigorous route is followed for the main UK qualifications at National
Qualification Framework
Level 3. In th
e
s
e

case
s

UCAS has an electronic Awarding Body
Linkage (ABL) process set up with the main awarding organisations, for direct transfer of
v
erified results up to
seven

days in advance of their release to applicants. A process of
‘fuzzy matching’ is applied well in advance to ensure that results are matched to the right
individual.

The Unique Learner Number (ULN) differs from the UCAS unique
identification number in
that it is currently allocated to all
14+ learners in English state schools
, is planned to be
rolled out to all 14+ learners in England, Wales and Northern Ireland, and is allocated to all
Skills Funding Agency

funded

learners in F
urther Education. It thus has the potential to act
as a key to a record of life
-
long learning, and to facilitate investigation of widening access to
education and employment. It is also therefore of interest as an identifier in HE student
record systems,
and a potential key to the Higher Education Achievement Report, evaluation
of which is included in the

overall

pilot programme

of the LRS Higher Education Advisory
Group
.

As part of its active and long
-
term collaboration with the Managing Information Acros
s
Partners (MIAP) programme, now the Learning Records Service (LRS), UCAS has collected
the Unique Learner Number (ULN) within Apply since the start of the 2009 application cycle.
The entry of this field by applicants is optional. At the point of entry
UCA
S

check
s

that the
number follows the correct validation for ULNs.
UCAS

do
es

not however currently integrate
its

processes with the LRS Personal Learning Record (PLR) system or verify that the
number belongs to the applicant.
Th
is

pilot provided the opportunity to investigate the
benefits of more closely integrating these processes
, in order to obtain valuable verified data
from the LRS, for supply to our member institutions
.




1

This total currently includes 154 HE

only institutions and 149 Further Educ
ation institutions offering
Higher Education. The former group includes universities, and schools or colleges with university
status; the latter includes general FE colleges and colleges offering technology, land
-
based, art,
design or other specialist prov
ision.

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

5

4.

Objectives

Th
e

pilot was designed to develop

live
links to the
LRS
in order to evaluate
acquisition by

UCAS
Apply
of

verified results for currently unverified qualifications entered by applicants
,
using GCSE results as an example
.

The following HE sector business needs were addressed by the
p
ilot:



Improv
ed verification of learner identity



Availability of verified learner achievement data (for example Level 2 results including
GCSE results)



Development of effective interfaces between UCAS and
LRS

for
efficient and secure
transfer of learner data, based on
appropriate technology (for example, web
-
services)



A more efficient and accurate UCAS Apply system



Overall increased efficiency of application, admissions and student record processes

W
ork
was carried out
with some volunteer schools

and colleges
, to provi
de feedback on the
user experience
of a trial version of Apply

and to highlight any issues that may be
encountered using a dynamic approach to data acquisition

by Apply

from the PLR.

Following hands
-
on evaluation sessions in the schools and colleges, stud
ents and staff were
surve
yed on their experiences.

5.

The challenge

The

challenge for UCAS, in its approach to the adoption of the ULN and the PLR linkage, is
to maximise the benefits to applicants and HEI members, without risk to the quality,
efficiency an
d cost
-
effectiveness of its current well
-
established systems.
This depends on
the ability to:



identify learners sufficiently rigorously to allow data retrieval from the PLR;



transmit data to and from the PLR reliably, securely and quickly, using web servi
ces
;



locate relevant good quality applicant data in the PLR and be assured of its accuracy
and currency
;




utilise PLR data to
populate Apply in real time, allowing applicants to accept or reject
the results;

The
Phase 2
pilot
was able to
evaluat
e
the
magnitude of this challenge
,
pointing the way to
possible
solutions.

6.

Established practice

6.1.

Current use of UCAS Apply to enter ULN and demographic data

Applicant personal data, including key demographic items of Given Name, Family Name,
Date of Birth, Gende
r and Postcode, are entered by applicants directly into the Apply on
-
line interface. The ULN is also collected in the same way
.


Entry of the ULN is not mandatory, and only a
relatively
small proportion of applicants
actually enter
ed
the number

for 201
1

e
ntry
, as shown in

Figure
1
.
T
he maximum
reached of 9.4% for English
-
domiciled applicants in January compares with an
equivalent value of 6.6%
in the same month
for 20
10 entry
, suggesting some progress
.

Numbers
we
re insignificant for domiciles outside England an
d Wales, as currently
expected from the scope of ULN allocation.
Approximately 71% and
4
% respectively of
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

6

201
1

applicants came from these two domiciles.
Th
e resu
lts

suggest that, for the
majority of UCAS applicants with ULNs, reliable
and

widespread use of the ULN will
depend on obtaining the value by a link to the PLR.


Figure
1

Percentage of applicants entering ULN




6.2.

Current use of
Apply to enter GCSE qualification data

Apply is also currently used to enter GCSE and similar Level 2 results, as well as results of
any other qualifications not currently covered by the ABL process. As shown
in
Figure
2

by
the arrows adjacent to the relevant fields, and illustrated by the list of grades, entry of
subject, date, awarding body and grade is generally by selection from a drop
-
down list. T
his
helps to minimise errors inherent in free
-
text entry. However, because of the large range of
potential subjects and the difficulty of ensuring comprehensive coverage, the option remains
of free
-
text entry for ‘other’ subjects.



0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
% of 2011 applicants supplying their ULN

England
Wales
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

7

Figure
2

Existing Apply page for entry of GSCE data


7.

Changes evaluated by the pilot

In Phase 1 of the pilot, i
n preparation for the transmission of real applicant data from the
LRS via web services,
a link
had been

established to the LRS test site
allowing

recei
pt of

test data. This was used to develop a test Apply interface, showing how in principle
Apply could be pre
-
populated using PLR data.
In Phase 2, f
urther development of the
test interface

was
carried out,

to incorporate PLR data on real ap
plicants,
and the
changed were evaluated by v
olunteer

Year 1
2

students from six schools, according to
the following steps:

7.1

All participants were logged onto a special test site using a dedicated URL, user
name and password. This gave access to a trial ver
sion of Apply, based on the
2010 Apply package.

7.2

Each volunteer was supplied with his or her ULN and entered this
ULN, with
their
Given Name

and
Family Name
, into the registration section of the trial Apply.

7.3

The trial Apply system used a web service to obta
in the gender, date of birth,
last
known postcode of

the participant from the Learning Records Service,
and
displayed these within the trial Apply page.


7.4

Following the entry of a user
-
defined password and answers to security questions,
each participant wa
s given a trial Apply user name and
was
logged in to the trial
Apply service.

7.5

The trial Apply system used
the

web service to obtain
qualifications data
held on
the
participant from the Learning Records Service
.
Data were predominantly on Level 2
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

8

qualifications (e.g. GCSE).
When
the participant clicked on the ‘Education’ link,
t
hese were

displayed within the
appropriate
trial Apply page

and flagged as
’Verified’
.

7.6

Participants were asked to critically examine the qualification results displayed and

to note any errors in: the school; qualification heading; qualification title;
qualification grade; and awarding body.

7.7

All were encouraged to correct any errors by manual input and/or to try editing
results, in order to note the effect on the ‘Verified’
flag.

7.8

Participants were then directed to an on
-
line survey in order to evaluate the success
of the trial Apply service. From this, it was possible to obtain
information
which
included:

7.8.1

How familiar participants had been with the ULN and the Learning rec
ords
Service before the trial
.

7.8.2

How successfully the LRS could identify the learner and provide demographic
and qualification data
.

7.8.3

Accuracy of the demographic and qualification
data obtained, as perceived by
the learner.

7.8.4

Success of the process which
flagged data obtained from the LRS as

V
erified’.

7.8.5

Ability of the learner to edit the qualification data displayed.

7.8.6

Success of the removal of the ‘
V
erified’ flag where such editing had been
carried out.

7.8.7

Any problems experienced by the participants in using

the trial service.

7.8.8

Preferences of participants in how qualifications data were acquired by

or
entered into
Apply
.

7.8.9

Participants’ views of the utility of the trial Apply system, with its links to the
Learning Records Service.

7.9

School staff involved were als
o asked to complete a short survey about their
familiarity with the ULN; its use in the school; and their views on the value of the
LRS
-
trial Apply linkage.



grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

9

8.

Schools and students involved in the evaluation

The schools and their students which were involv
ed in the evaluation are as shown in
Table
1
.

Table
1

Schools involved in the evaluation

Type of School

School Name

Staff Contact
(s)

Number of

students

Comprehensive

Cheltenham Bournside School
& Sixth Form Centre

Mr.
Mike Williams

11

Chase Terrace Technology
College, Burntwood,
Staffordshire

Mr.
Tim Dowling

Ms. Kim Woolley

14

Castle Sixth Form
,

Kenilworth
School and Sports College,
Warwickshir
e

Mr.
Nick Mummery

Ms.
Liz Hadley

10

Stoke Park School &

Community Technology
College, Coventry

Ms.
Sandra Arch

Ms. Noreen Woodall

17

Academy

The Cotswold School, Bourton
-
on
-
the
-
Water, Gloucestershire

Mr.
Will Morgan

Ms. J
ane Tanner

8

Pate's Grammar School,
Cheltenham

Ms. Helen Hooper

Ms. Ruth Lucas

Mr
.
Chris Beal

17


Although a range of schools were covered, the 77 students involved cannot be regarded for
statistical purposes as a representative sample of the whole cohort for England; far less for
the UK. Results cannot therefore be analysed for statistical significan
ce and should be
regarded as indicative only. This applies more strongly to the staff evaluation which was
carried out by eight of the staff concerned.



grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

10

9.

Results of the
student
evaluation


9.1.

HE Application

experience/intentions of participants

Almost half

the participants had started their UCAS application for 2012 (
Figure
3
)

and
the vast majority definitely or possibly intended to apply through UCAS (
Figure
4
).

Figure
3

Experience of using UCAS Apply


Figure
4

Intention to apply through UCAS





44%

56%

Have you used UCAS Apply before?

Yes
No
85%

1%

14%

Are you intending to apply through UCAS for entry
to Higher Education?

Yes
No
Possibly
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

11

9.2.

Knowledge of

the

Learning Record
s

Service and ULN

O
nly 23
%

of participants had heard of the ULN before the pilot

(
Figure
5
), but none of
them could remember their own ULN (
Figure
6
)
.
This is consistent with data from
UCAS 20
11 entry, which shows that only around 6% of those expected to possess
ULNs were able to enter them into UCAS Apply.

Figure
5

Knowledge of the ULN



Figure
6

Memory of the ULN





23%

77%

Had you heard of the Unique Learner Number (ULN)
before this trial?

Yes
No
100.0%

Before the trial, had you memorized your ULN?

Yes
No
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

12

9.3.

Ability
of the trial Apply system
to retrieve data from the LRS

In well over 90% of cases, the LRS was able to identify learners in the register and
provide their gender, date of birth and postcode (
Figure
7
).
Resulting data were
in
accurate in a few cases, mainly as a result change of address, hence of postcode.
Moving between schools appeared to account for most of the cases where school or
college was inaccurately identifie
d (
Figure
8
).

A
round 6% of participants had ULNs, but no qualification results in the LRS (
Figure
9
). This

is consistent with an LRS estimate of 5
-
10% of
state funded students in
England who are not automatically registered with ULNs when they reach age 14
.
These participants were registered subsequently by the school but

would hav
e
missed the transfer of relevant data from the National Pupil Database.


Figure
7

Success in identifying each learner in the LRS



Figure
8

Correct identification of school or college



Figure
9

Success in
obtaining qualification results




92%

8%

Was the system able to find your gender, date
of birth and postcode?

Yes
No
80%

20%

Did the trial Apply identify your school or
college from the LRS Personal Learning
Record?

Yes
No
94%

6%

Did the trial Apply pick up any of your GCSE
results from
the PLR?

Yes
No
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

13

9.4.

Completeness and
correctness of
classification
of the results
obtained

Survey q
uestions were based on GCSE results, as they were by far the most frequent
examples of Level 2 data

for t
he cohort involved
. Where
ver

results had been
successfully obtained from the LRS, these were perceived as complete in the vast
majority of cases (
Figure
10
). However t
he classification of results under qualification
type headings was less successful (
Figure
11
). This was the result of a known flaw in
the mapping of codes for qualif
ication type, used in the LRS web service, against
UCAS qualification headings.
Non
-
GCSE results, such as for BTEC qualifications,
also tended incorrectly to appear under the ‘GCE A level’ heading. This issue is
discussed more fully in
Section 11 (Issues for UCAS).

Figure
10

Completeness of the results, as perceived by the learner



Figure
11

Accuracy of
classification of
the results, as perceived by the
learner




95%

3%

2%

Were these GCSE results complete?

Yes
No
I'm not sure
68%

32%

Under what heading or headings did the GCSE
results appear?

All GCSE under
'GCSE' heading
Some GCSE under
'GCSE' heading;
some under other
headings
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

14

9.5.

Verification of
results and their accuracy

Where results were obtained from the LRS, they were, with very few exceptions,
flagged by the system as ‘Verified’, as intended (
Figure
12
).

It is possible that the few
exceptions reported were based in a misunderstanding by participants, as it should not
be technic
ally possible for this to occur;

however this needs to be checked by closer
analysis. The 6% of results perceived to be inaccurate

(
Figure
13
) appeared, however
,

to reflect a genuine disagreement with what the LRS had recorded. Participants
involved were invited to correct manually any perceived errors in the trial Apply, to
produce an unverified result. They were also reminded of the facility to challenge the
ori
ginal LRS data via their schools.

Figure
12

Flagging of results as ‘Verified’


Figure
13

Perceived accuracy of results





97%

3%

Were the GCSE results that were displayed
marked as verified?

Yes - all displayed
as verified
Some displayed as
verified and some
not.
94%

6%

Were all GSCE results from the LRS Personal
Learning Record correct, as far as you know?

Yes
No
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

15

9.6.

Effect of manual modification of qualification results

Although the majority of
the results obtained form the LRS were perceived as
accurate, even those participants without any perceived errors were encouraged to try
out the system of manual modification
, with emphasis that this was for the purposes of
the pilot only. Just under 2/3

chose to do so (
Figure
14
), though a few may have run
out of time as a result of teething problems in the trial interface. Grade change was (as
expected) the most po
pular modification (
Figure
15
). Results produced were then no
longer ‘Verified’ (
Figure
16
), though again a very few ‘rogue’ results need to be
followed up.

Figure
14

Proportion making manual modification of trial Apply results



Figure
15

Types of manual changes made



Figure
16

Effect on ‘Verified’ flag


61%

39%

Did you change, add or delete any GCSE results
manually using the trial Apply entry screen?

Yes
No
0
20
40
3

3

8

37

What were the changes or additions?

Subjects added
Subjects removed
Subject title
changes
Subject grade
changes
3%

97%

Did changed results then show with or without the ‘Verified’
comment?

Changed results were
still shown with the
'Verified' comment
Changed results were
shown without the
'Verified' comment
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

16

9.7.

Participants’ evaluation of the trial Apply system

When asked how they would prefer to fill the Apply electronic form with GCSE and
similar results, a large majority of participants chose the option o
f receiving verified
results automatically from the LRS, but with the
possibility

of manual amendment
(
Figure
17
). A significant minority were prepared to accept LRS
results without the
manual amendment option, and only a small minority preferred the current manual
system. Consistently with this, over 97% of participants rated the automatic receipt of
verified results with optional amendment
(
Figure
18
)
as either useful (42%) or very
useful (55%)
:

a strong endorsement of the
LRS

linked trial Apply system developed
by the pilot.


Figure
17

Participants’
preferences for entry of qualification results


Figure
18

Overall evaluation of trial Apply system






12%

81%

7%

Which of the following would you prefer?

Automatic verified results
with no manual
amendment allowed
Automatic verified results
with manual amendment
allowed, to give
unverified results
All results manually
entered as unverified (as
in the current Apply
system)
55%

43%

0%

3%

Please rate the automatic entry of your verified GCSE
results into Apply, with manual changes allowed

Very useful


could
save a lot of time
and errors

Useful - could save
some time and
errors
Not very useful


probably would not
use it

Not at all useful


would definitely not
use it.

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

17

10.


Results of the
staff
evaluation

10.1.

Assistance given by staff to students in confirming Level 2
qualifications

A large majority
of staff surveyed provided some assistance to their students in
confirming Level 2 qualifications for UCAS entry (
Figure
19
): most found this at least
slightly challen
ging (
Figure
20
), and spent between 10 and 20 minutes on the task
(
Figure
21
).

Figure
19

Assistance given or not given



Figure
20

Challenge faced by staff in providing assistance



Figure
21

Time taken in providing assistance



88%

13%

Do you normally assist your UCAS applicants in confirming
GCSE or other Level 2 qualification results for entry to UCAS
Apply?

Yes
No
33%

50%

17%

How challenging is this?

Not at all
Slightly
Very
17%

66%

17%

How much time does it take per student?

<10 minutes
10-20 mins
21-30 mins
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

18

10.2.

Staff k
nowledge of

the

Learning Record
s

Service and ULN

Although less than half the staff had heard of the Learning Records Service (
Figure
22
), over 85% had heard of the ULN, compared with 77% of students (
Figure
23
). In a
few
cases, staff wer
e able to use the

LRS

Organisation Portal to obtain ULNs

(
Figure
24
).

Figure
22

Knowledge of the Learning Records Service


Figure
23

Knowledge of the Unique Learner Number



Figure
24

Use of Organisation Portal


43%

57%

Had you heard before this trial of the Learning Records
Service (previously known as Managing Information Across
Partners
-

MIAP)?

Yes
No
86%

14%

Had you heard of the Unique Learner Number (ULN) before
this trial?

Yes
No
14%

86%

Did your or a colleague use the LRS Organisation Portal to
obtain ULNs for participants in the pilot?

Yes
No
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

19

10.3.

Use of ULN by schools

About half the staff surveyed provided students with their ULNs (
Figure
25
), but in only
about 1/3 of cases was the ULN actually used as a key in the student record system
(
Figure
26
). In the schools involved, there was no explicit discussion of the sharing of
student data in the LRS (
Figure
27
), but this is regarded as implied by the genera
l
agreement between the school and its students.

Figure
25

Provision of ULN to students


Figure
26

Use of ULN as a key to student records


Figure
27

Discussion of LRS data sharin
g



50%

50%

Does your school/college normally provide your students
with their ULNs?

Yes
No
67%

33%

Does your school/college use the ULN as a key in its student
record system?

Yes
No
100%

Does your school/college normally discuss sharing of data
in LRS Personal Learning Records with your students?

No
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

20

10.4.

Staff evaluation of the trial Apply system

All staff surveyed
preferred the
chose the option of receiving verified results
automatically from the LRS but with the possibility of manual amendment
, over a
totally automatic or totally manual system (
Figure
28
). All staff similarly found this
preferred system as useful or very useful, of which around 70% chose
‘very useful’

(
Figure
29
). This represents an even stronger endorsement by staff in the schools
surveyed of the LRS

linked trial Apply system, than by their students.

Figure
28

Staff preferen
ces for entry of qualification results


Figure
29

Staff overall evaluation of trial Apply system




100%

Which of the following would you prefer?

Automatic filling in of verified
GCSE results to Apply from the
LRS with no manual amendment
allowed
Automatic filling in of verified
GCSE results to Apply from the
LRS with manual amendment
allowed, to give unverified
results
All GCSE results entered by
applicants directly as unverified
(as in the current Apply system)
71%

29%

How would you rate the automatic completion of verified GCSE
results into Apply, with manual changes allowed?

Very useful


could save a lot of
time and errors

Useful - could save some time
and errors
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

21

11.

Conclusions

11.1

Most
student
participants were intending UCAS applicants


just under half had
started
their
application
.

11.2

Participants showed l
ittle
previous
knowledge of the LRS or ULN (as expected)
.

11.3

Good success (>90%)
was achieved
in using
the
web service to obtain
participant
data from
the
LRS
.

11.4

A few participants had p
roblems
,

mainly as a result of
their
educational histo
ry


LRS records
were
incomplete

because
registration was recent.

11.5

GCSE results obtained
were
perceived

by participants
as > 95% complete and
94% correct
.

11.6

Results obtained were, with very few exceptions, successfully flagged as
‘Verified’

11.7

Participants were
able to amend results manually, with loss of the ‘Verified’ flag.


11.8

However around 1/3
o
f participants
reported some results under wrong heading
,
as a result of known mapping issues.

11.9

Participants provided a m
ajor endorsement of automatic capture of verifi
ed
results
.

11.10

Most wanted the opportunity to amend

these results
, with loss of verification
.

11.11

The v
ast majority (>97%)
of participants
found
the
system useful or
very
useful

11.12

Most school staff surveyed spent 10
-
20 minutes per student in helping confirm
Level 2

qualifications for UCAS entry.

11.13

Although around half of the school staff provided ULNs to their students, in only
about 1/3 of cases was the ULN actually used as a key in the school student
record system.

11.14

School staff surveyed also provided a clear

endorse
ment of automatic capture of
verified results
, with an option for students to amend results, losing verification.

11.15

All school staff surveyed evaluated the pilot system as useful, with a large
majority
finding it very useful.


12.

Issues for UCAS

12.1

Ambiguities re
main in mapping LRS web service to UCAS Apply
:

12.1.1

This m
ay interfere with correct identification of Qualification Type,
leading to occasional

listing of

qualifications under wrong headings

12.1.2

Further work is required to compile a definitive list of LRS web service
Qualification Type codes and their meanings in relation to UCAS
qualification type headings.

12.2

The need for verification of identity at Level 1 currently restricts the number of
LRS re
cords available
:


12.2.1

Current proposals for a ‘
Learner Passport


may help

by allowing a larger
proportion of LRS records to satisfy the Level 1 requirement.

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

22

12.2.2

An alternative for the LRS is to amend the current criterion for identity
verification, so that ‘Relati
onship with school’ is regarded as meeting the
necessary requirement.

12.3

There are k
nown gaps in allocation of ULN to HE applicants in the UK

12.3.1

The Scottish administration remains resistant to adoption of the ULN

12.3.2

The Welsh administration is, in contrast, commi
tted to its adoption,
partly as a facilitator for the Welsh Baccalaureate.

12.3.3

Adoption by the private school sector is patchy.

12.3.4

EU and overseas applicants through UCAS will not possess ULNs or
Personal Learning Records at the time of application.

12.4

Even in Eng
land there is currently poor knowledge or
uptake in
state
schools of
the Learning Record
s

Service and ULN
.

12.4.1

Applicants cannot be expected to know their ULNs.

12.4.2

Schools can provide these to applicants, but many are not currently in a
position to do so.

12.4.3

UCAS
can play a part in raising awareness amongst schools and
universities of the potential benefits of adoption of the ULN as a key to
student records and facilitator of UCAS application.

12.4.4

Alternatively or additionally, the trial Apply system developed for the

pilot could be extended to include the ‘Find by Demographics’ web
service call. This would use Given Name, Family Name, Date of Birth,
Gender and Last Known Postcode to locate a learner in the Learner
Register, and return to UCAS a verified ULN, in additi
on to qualifications
data.



grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

23

13.

Recommendations

13.1.

In order to tackle the qualification heading issue, UCAS and the LRS should
collaborate in order to compile a definitive list of LRS web service
Qualification Type codes and their meanings in relation to UCA
S qualification
type headings. This may take place as part of the ongoing development of
data standard
s

in the two organisations.

13.2.

In developing data standards, UCAS and the LRS should also aim at
convergence in the definitions of Given Name and in the
validation of
Postcode. This recommendation r
em
ains from the Phase 1 pilot
, but its
implementation could

not

be

included in the scope of Phase 2.

13.3.

In order to overcome lack of knowledge by learners of their ULNs, t
he trial
Apply system developed for the pil
ot
sh
ould be extended to include the ‘Find
by Demographics’ web service call. This would use Given Name, Family
Name, Date of Birth, Gender and Last Known Postcode to locate a learner in
the Learner Register, and return to UCAS a verified ULN, in addition
to
verified
qualifications dat
a. Verification of both types of data is of great value
to HEIs and could pave the way to the use of the ULN as a single student
identifier in HE.

13.4.

Following these
developments, it would be beneficial to carry out a
larger
-
scale evaluation of a prototype LRS

linked UCAS Apply system with a larger
and more fully representative sample of school and FE college students.

13.5.

This could form part of a programme by
UCAS
to

rais
e
awareness amongst
schools and universities of th
e potential benefits of adoption of the ULN as a
key to student records and facilitator of UCAS application
.


Geoff Ramshaw

UCAS Policy Executive

August 201
1



grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

24

Appendix 1
.
Technical Report
:
UCAS Apply
-

Integration and use of the LRS
Web Service

UCAS Ap
ply
-

Integration and use of the
Learning Records Service (
LRS
)

Web
Service


1.

Overview

The U
C
AS Applicant Apply suite of programs was modified to access the secure LRS web
service in order to extract applicant qualification records held by the Learner Record Service
during the applicant registration process. The applicant’s Unique Learner Number (
ULN) and
basic demographic data (name and date of birth) were used to authenticate the request to
the LRS database and qualification records were imported into the UCAS qualification
database tables with suitable mapping of LRS data elements to UCAS data e
lements. The
mapping was primarily concerned with identifying GCSE results and creating valid UCAS
records to store them, although other qualification types were processed and stored. Only
students with a suitable verification status were able to extract q
ualification records.

As a result, after successfully registering, the applicant was able to sign in to Apply and view
his pre
-
loaded qualifications listed by educational establishment. Each qualification record
loaded from LRS was marked as “verified”. T
he student was able to edit LRS qualifications
where there was an inaccuracy or add missing qualifications, although edited or added
qualifications did not carry the “verified” status.

2.

Technical Implementation

LRS provides a WDSL (Web Service Definition
Language)

document and associated XSDs
(XML schema definitions) of the service. These define the network and methods provided by
LRS in order to locate and extract qualifications. Using these definitions, a client web service
was built using the Java 6 lan
guage and its associated XML Web Service protocols. The
web client was embedded in a prototype version of Applicant Registration and was used to
invoke and communicate with the LRS web service hosted remotely. The LRS web service is
a secure HTTPS service
and both UCAS client and remote server security certification was
provided by LRS and installed on the UCAS web service client server using Java Weblogic
HTTPS protocol.

The UCAS pilot web service was ultimately deployed on an Oracle Web Logic server

runni
ng
Java 1.6.with default server and database settings.

In order for students to access the pilot service in their schools, a temporary external service
was provided so they could communicate with the server which resides behind the UCAS
firewall.

3.

Problems

Encountered

3.1

Security certificate issues and other technical barriers to the web service link

The implementation of SSL certification proved to be problematic and consumed
unscheduled effort in the project. The implementation of SSL Java keystores in
JDeveloper OC4J 10.1.2 using the proprietary Java 1.6 Keytool utility to generate the
SSL keystore
s proved impossible and a decision was made to migrate the Apply pilot
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

25

software from 10.1.2 to JDeveloper OC4J 10.1.3 where the SSL keystores were
successfully deployed.

However, it was then discovered that Information Services has no externally facing
se
rver running OC4j 10.1.3 (which was required to publish an externally accessible web
application) and that there were no plans to do so. As a result, a decision was made to
migrate the Pilot Apply application once again, this time to Oracle 11g IDE and the

Oracle Weblogic server running under Java 1.6.2. Weblogic is the strategic server for
UCAS so this was an opportunity to migrate a key service to the new server, albeit in
pilot form, and much was learned from this exercise. Considerable effort was requir
ed to
get Apply Pilot and the associated Web Services and SSL keystores running correctly in
this environment (which uses a different version of the J2EE framework) both in terms of
software re
-
engineering and server configuration.

3.2


Mapping from LRS web se
rvice codes to UCAS variables (especially
qualification type codes, but also any others).

This proved problematic because there was no natural mapping from LRS classification
data to required UCAS data elements; indeed some LRS data elements had no place i
n
the UCAS qualification record structure. A substantial amount of time was spent
performing analysis of the LRS data model and its relationship to mandatory UCAS
fields. This activity required the assistance of LRS staff as well as the UCAS project
team.
A number of data elements were analysed and finally mapped successfully to
provide a UCAS record that was both internally correct and also contained the imported
LRS qualification information.

3.3

The problems encountered in making this available externally

T
he final challenge was to deploy the web application to the external students
participating in the pilot. This problem was solved by a special secure log in page
provided by ICT that schools could access and then successfully browse to the Web
Application.

3.4

Impact of Project

All the difficulties described above consumed additional unforeseen developer time.
However, the technical knowledge gained from the secure web service implementation
and the Weblogic server migration is of great value to the department.

4.

Performance

of Pilot

Initially, some problems were encountered running the Registration service and then logging
into the main Applicant service. This was due to a problem with an insufficient number of
database connections available for the multiple data
base accesses required by the student
group which numbered about 20.The problem was rectified by increasing the number of
database connections or threads and subsequent student sessions were almost problem
free.

grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

26

5.

Glossary of terms

5.1

Web Service Definitio
n/Description Language (WSDL)

A
XML
-
based language (see below) that is used for describing the functionality offered
by a w
eb service
. A WSDL description of a web service provides a machine
-
readable
description of how the service can be called, what parame
ters it expects and what data
structures it returns.

5.2 XML


Extensible Markup Language (XML) is a set of rules for encoding documents in
machine
-
readable form, widely used for the representation of arbitrary data structures,
for example in web services.
XML
-
based formats have become the default for most
office
-
productivity tools, including Microsoft Office
.

5.3

XSD (XML Schema Definition)

XSD can be used to express a set of rules to which an XML document must conform in
order to be considered 'valid' accordin
g to that schema.

5.4

HTTPS

Hypertext Transfer Protocol Secure

(
HTTPS
)

is a combination of the
Hypertext Transfer
Protocol

(HTTP) with the
SSL

protocol (see below) to provide encrypted communication
and secure identification of a network
web server

5.5

Oracle
WebLogic server


Proprietary Oracle software (‘Middleware’) managing links between database & web
applications.

5.6

SSL certification

Secure Sockets Layer

(
SSL
),

is a
cryptographic protocol

that provides communication
security

over the
Internet
, based on a
digital certificate which allows access to an
encryption key.

5.7

IDE
(
I
n
tegrated
D
evelopment
E
nvironment)

IDE
is a software application that provides comprehensive facilities to computer
programmers for software development

5.8

JDeveloper

JDeveloper is a freeware
IDE
(see above)
supplied by Oracle Corporation which covers
the full development lifecycle from design through coding, debugging, optimi
s
ation and
profiling
,

to deploying.

5.9

Web Services

The term
Web services

describes a standardized way of integrating Web
-
b
ased
applications using the XML, SOAP, WSDL and UDDI open standards over an Internet
protocol backbone. XML is used to tag the data, SOAP is used to transfer the data,
WSDL is used for describing the services available and UDDI is used for listing what
ser
vices are available. Used primarily as a means for businesses to communicate with
grapeafterthought_eb04419f
-
06d1
-
4fb3
-
b2ea
-
3d277377d88f.docx

27

each other and with clients, Web services allow organizations to communicate data
without intimate knowledge of each other's IT systems behind the firewall


Nicholas Pendrigh

Application Support and Maintenance Programmer

August 2011