Project Document Cover Sheet

bewgrosseteteΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 6 μήνες)

64 εμφανίσεις

Project Acronym: ITS4SEA

Version: 1

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009

PAGE
1

of
19


Document title: JISC Final Report


Project Document Cover Sheet


Project Information

Project Acronym

ITS 4 SEA (I
ntegrating

T
hin

client

S
ystems
for S
ecure

E
-
A
ssessment

Project Title

Integrating thin client systems
and smart card technology to provide
integrated, flexible, accessible and secure E
-
assessment

Start Date

October 2007

End Date

March 2009

Lead Institution

University of Bradford

Project Director

Prof Peter Hartley

Project Manager &
contact details

Mrs Sara Eyre,
s.j.eyre@bradford.ac.uk

01274 233335

Partner Institutions

None

Project Web URL

http://www.brad.ac.uk/lss/jisc/

Programme Name (and
number)

JISC

Institutional Exemplars

Circular 1/07

Programme Manager

Lawrie Phipps


Document Name

Document Title

Final report

Reporting Period

N/A

Author(s) & project role

Sara Eyre, project manager

Date


Filename


URL

if document is posted on project web site

Access




General dissemination


Document History

Version

Date

Comments

1

25 February 2009

First draft for approval by project board

2

1 June
2009

Final version for submission to JISC




Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
2

of
19

JISC Final Report

Table of Contents
JISC Final Report

................................
................................
................................
................................
....

2

Table of Contents
................................
................................
................................
.............................

2

Acknowledgem
ents

................................
................................
................................
..........................

2

Executive Summary

................................
................................
................................
.........................

3

Background

................................
................................
................................
................................
......

4

Aims and Objectives

................................
................................
................................
........................

4

Methodology

................................
................................
................................
................................
....

5

Implementation

................................
................................
................................
................................

5

Thin client activities

5

Word processed exams

6

Smart ca
rd technology

7

Examination Administration

7

Estates and E
-
coversity

9

Quality assurance and project management

10

Outputs and Results

................................
................................
................................
......................

10

Outcomes

................................
................................
................................
................................
.......

14

Major improvements to E
-
administration

15

Sustainable developments

16

Major Deliverables

16

Conclusions

................................
................................
................................
................................
...

17

Implications

................................
................................
................................
................................
....

17

Technology

17

E
-
assessments

17

Exam administration

18

Recommen
dations

................................
................................
................................
.........................

18

Project management

18

For HEIs developing E
-
assessment facilities

18

For HEIs seeking to improve their E
-
assessment
processes

19

For HEIs planning to improve their examination timetabling

19

Consider the technology options

19

Appendixes

................................
................................
................................
................................
....

19

Appendix A.

Designing a new facility for Online Assessments

19

Appendix B.

ExamAdmin Draft Functional Design Specification

19

Appendix C.

Student exam evaluation

19

Acknowledgements

Gratefully acknowledgements are made to JISC for the funding
of the development of this project
under the Institutional Exemplars programme and for the support provided by the programme
manager, Andy Dyson. Special thanks are also due to the members of the project board for their
unfailing support and helpful advice
, especially our two external evaluators Prof Mark Stiles from
Staffordshire University and David Lewis from Glamorgan. Very special thanks to all the University of
Bradford staff who worked on the project teams for their patience, particularly when thing
s
have not
always gone as planned. Special thanks to John Dermo who has taken on the bulk of the
dissemination work for the project at a wide range of conferences and been the main motivating force
behind the increased uptake of online assessment at the Un
iversity of Bradford.


Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
3

of
19

Also much appreciated has been the work of the external companies who worked so hard to ensure
that the new E
-
assessment facility
,

which has been created at the University
,

was finished in time to
be used for the January 2009 examina
tions, which were the only exams
that

could be scheduled
during the life of this project. These include:
-



Farrell and Clark, Architects
,

for the overall design based on a random list of general ideas



PSK Tilbrook, project managers
,

for ensuring the building work completed on time



Andy Hague, University project manager



Wood Mitchell, main contractors, for completing the work to a very tight timescale.

Our thanks also need extending to S
un

Microsystems for their matching grant programme which
allowed the Sun Ray workstations to be obtained at the most competitive pricing.

Executive Summary

This project has demonstrated that thin client technology, allied to systematic institutional procedur
es
and processes, can deliver effective and efficient summative e
-
assessment to learners in a flexible,
secure, and accessible way. This technology has enabled major improvements to this University’s
delivery of e
-
assessment and offers a number of advantag
es over ‘traditional’ PC clusters. Benefits to
the sector include ‘proof of concept’ in the application of this technology to e
-
assessment and guides
and exemplars to assist institutions who wish to expand or improve their provision in e
-
assessment
and exa
mination timetabling.


Major learning points for the HE community lie in the development of the infrastructure to support
large numbers of thin client workstations using VDI technology and in the secure delivery of online
assessments using Questionmark Per
ception, with Appsense used to control the facilities that
students have access to during the examination. During the design phase of the thin client
assessment facility, research was also conducted into best practice for layout, facilities required and
ot
her related features; a report detailing the findings was produced and forms part of this final report.


The project has made a significant impact on our institutional use of e
-
assessment, with a
major

increase in uptake of online assessments following th
e creation of a new 100 seat thin client cluster
and the demonstration that it can be used to deliver assessments in a secure manner. It has
highlighted the need for high quality data on both student enrolments and module assessment
recording.


The studen
t exam experience has been improved by the provision of personalised examination
timetables, delivered by email, and the automatic allocation of facilities for students with special
needs.
The
design specification for a new examination administration syste
m

forms part of this final
report.
The majority of students also found doing their exams online a major improvement over using
Optical Mark Recognition forms; they were also considerably quicker for administrators to process the
results

from.


The final le
arning outcome from this project relates to the environmental credentials of using thin
client technology. The project has worked closely with the JISC funded SusteIT
1

project to try and
evaluate whether thin client technology provides a more environmental
ly friendly alternative to the
standard PC.

The tools developed as part of this collaboration will assist other Universities and
Colleges to assess whether the total cost of ownership of thin clients will benefit their organisation in
the long term
.







1

http://www.jisc.ac.uk/public
ations/publications/sustainableictfinalreport.aspx

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
4

of
19

Ba
ckground

The project board believed at the outset that using thin client (Sun Ray) technology linked with smart
card identification would allow a flexible, personalisable and scalable system for summative e
-
assessment. Thin client technology has already be
en successfully piloted at the University in other
contexts and this project has built on work we have conducted with Sun Microsystems to evaluate and
introduce their Sun Ray thin client technology as an environmentally friendly, secure and sustainable
alt
ernative to the PC.



Some of the issues addressed by this project were highlighted in the 2007 JISC report ‘Effective
Practice with e
-
Assessment’
2
??LQ?SDUWLFXODU?WKH?LVVXHV?DURXQG?µVHWWLQJ?XS?YLDEOH?SK\VLFDO?DQG?,7?
infrastructure to support e
-
assessment t
o the required scale’ and ‘cheating by digital means during
examinations


for example, by accessing the internet...’.


The project also aimed to improve a number of processes around scheduling of examinations and
integrating the procedures with key Unive
rsity systems, such as student records, disability information
and module registrations.


The project was designed to demonstrate that thin client technology can be used to provide an easy
-
to
-
manage, secure and reliable classroom resource for
computer aid
ed
and computer
-
based
assessment. To add further value to the project, we also proposed testing whether this system could
provide easier to support technology than standalone PCs for use within lecture rooms.


The use of online assessment
at the University of Bradford
ha
d

seen an exponential growth in the 12
months before this p
roject began
3

and
there was

already
a strong

commit
men
t to the use of
Questionmark Perception as
the

assessment tool of choice
4
. The difficulties encountered running

large scale assessments on standard PC clusters were deterring some staff from attempting to make
use of online assessments, so a large number of modules were still being assessed using paper
based Optical Mark Recognition forms, despite the time taken to

check each form by eye and then
scan them in
; a figure of 10 hours work to vet 200 OMR forms has been quoted.


This project also formed a natural follow on to the work that was done on the JISC and Higher
Education Academy Pathfinder program, which
investigated embedding support processes for e
-
assessment to ensure the reliable and secure large
-
scale implementation of CAA
5
. The work done
under this programme had already streamlined the way for increasing the number of summative
assessments conducted
on line.

Aims and Objectives

At the start of the project the aims were to address the following:
-



The need for secure and reliable systems to deliver computer aided assessments to large
classes in a manageable and sustainable environment



The need for flexi
ble systems which can deliver a wide range of assessment types and
which support automated delivery of anonymised scripts (as recommended in current QAA
guidance) for marking and for checking (e.g. testing for plagiarism through Turnitin etc.)



Full integra
tion of Computer Aided Assessment (CAA) processes and marks with the
student record system



Streamlining of examinations management by integrating exam scheduling software with
student record systems



The need for effective strategies to manage and improve t
he learning and assessment
process for inclusion, widening participation and accessibility.



Systems to allow use of IT facilities as part of the teaching process which are up to date,
easily managed and responsive to academic staff needs.




2

http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf

3

http://www.brad.ac.uk/elearning/pathfinder/documents/QMP_use_stats.pdf

4

http://www.questionmark.co.uk/uk/casestudies/bradford.aspx

5

http://www.brad.ac.uk/elearning/pathfinder/documents/Briefing_Papers.pdf

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
5

of
19



The development o
f sustainable and environmentally responsible approaches to our
development in learning, teaching and assessment, particularly in the deployment of
educational technologies.


The aims did not alter as the project progressed, but some proved impossible to h
andle within the
timescale of the project for a wide variety of reasons; these are explained in detail in the outcomes
section below.

Methodology

The various aspects of the project were split into separate work packages
at

the start of the
project;

these w
ere managed separately with the project teams coming together when the integration became
more important than the design processes. The Bradford project management methodology (based
on Prince2) was used throughout the project with regular project team mee
tings, and project board
meetings every 3 months where the risk and issue logs were considered and updated.


The work packages were

1.

Thin client related
activities

2.

S
mart card technology

3.

Integration of Examinations systems with Computer Aided Assessment and

student record
systems

4.

Estates and E
-
coversity


5.

Evaluation of impact of the project on the Institution

(see Output and Results section)

6.

Dissemination to interested external parties

(see project completion report)

7.

Quality assurance of the project and its
outcome

8.

Overall project management

Implementation

Thin client
activities

The
thin client related activities

commenced with a re
-
evaluation of whether Sun Rays were the best
choice of thin client technology for this project. Other manufacturers’ equipment w
as evaluated but the
Sun Rays provided more flexibility than other ‘terminals’ and the card reader that is incorporated into
the design was thought to provide additional benefits for online assessments and potentially for staff
use in teaching rooms. It wo
uld potentially allow a classroom session to be prepared in advance and
then simply transferred to the teaching room by placing the smart card in the card reader.


At the start of the project, Sun Rays were being slowly deployed into the
University’s
stud
ent
administration area known as the HUB. The aim was to allow staff to move rapidly from handling
student

queries at the front of house
desks to their own desk in the back office while retaining details
of the student’s account. Initially desktops were s
upplied using Windows
Terminal Services
but this
was causing increasing problems whereby a single failure with an application for one user had caused
all other Sun Ray sessions to hang and require restarting. Around this time, VMware introduced a
new syste
m for handling Virtual Desktops, called VDI. VDI was selected instead of
Terminal Services
due to the increased flexibility it could provide for future use with the Sun Ray cluster. VDI should
provide access to a wider range of applications software as wel
l as providing increased security; VDI
also builds more directly on existing expertise within the University which is based around managing
PC desktops rather than round Windows servers.


It wasn’t until the large scale Sun Ray cluster became available for

testing that problems integrating
the working of VDI and Sun Rays became evident!
We faced a number of issues surrounding storage,
the amount of time required to clone the virtual images and coping with the load from such a high
number of consecutive logi
ns. Low end NFS based storage, as is commonly recommended for VDI,
gave unsatisfactory clone times of up to a couple of days to provision all the work stations and proved
unable to cope with the login load bringing the system to a halt. After trying variou
s alternatives we
found that iSCSI arrays configured for high performance were able to handle the number of logins
required in the new facility. At the time of writing, there are still outstanding issues around clone times.
An alternative
hardware solution

was investigated
but
proved not to be

cost effective. A
s the
Virtualisation/VDI technology
develops,

more software solutions
from different suppliers
are becoming
Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
6

of
19

available
that claim to address the problems encountered during this project
and we
aim
to i
mplement
one
before the
2009
-
10

academic session
.


For normal student lab usage, the c
urrent
image is

25Gb

and

each image takes approximately 10
minutes to clone, for online assessments the image is down to 6Gb and can
be
clone
d

in about 3
minutes.

To overcome these problems for the January 2009 exams, a virtual desktop was assigned to
each work station rather than to individual users or to each type of smart card. Since all the exams
that were running at the same time were identical, this did not c
ause any problem for the exams but
would for teaching using the Sun Rays since a new class of students would need to login and be
assigned a new desk top within a short space of time.


Another aim was to trial use of Sun Rays in General Teaching Areas of t
he University in rooms that
do not have a fixed PC. Several staff have complained about having to carry laptops to these
locations and there are frequent problems with the PCs that are in GTA becoming out of date, trying
to install Windows updates at the s
tart of a class etc. Tests were conducted to ascertain if the
projectors fitted in classrooms were compatible with the screen resolution of Sun Rays and also with
some multimedia applications to see if Sun Rays were suited to all potential software applica
tions that
may be required while teaching. The projectors worked correctly but some software applications
struggle to display moving images and sound synchronised together which mean that, at the current
time, some teaching applications may not be suited t
o Sun Rays. However, Sun Microsystems are
aware of this shortcoming and
have

prioritis
ed

this for future firmware releases.


However, the project encountered difficulties trying to get the single data connection in lecture rooms
accessible to Sun Ray serv
ers as well the Univer
sity’s roaming network, RoamNet. These

have
proved insurmountable

due
in

part
to the ongoing
relocation of the main computer suite and network
equipment. It is not possible to simply reconnect the data sockets to a different network r
outer as they
still need to be used with staff’s own laptops in cases where a Sun Ray is unsuited to their teaching
needs. Thus, the functionality and suitability of Sun Rays in teaching area has still to be tested.

Word processed exams

Another aim of the
project was to
move

students who
currently
word process exams on PCs,
often
due to dyslexia,
and currently store their work on floppy disks, to use Sun Rays instead for increased
exam security.
In association with this, a trial was also planned to convert
a part written, part multiple
choice examination into a format where it could be taken entirely online using the Sun Ray facility.


Ideally, students should be able to use the word processing package with which they are most familiar
for exams. Some invest
igation was conducted into what word processing features were actually used
by the students who currently take their exams using a PC
;
this indicated that they were basically text
processing only. However, it is not possible to tell from the exam scripts w
hether the students were
also utilising features of Microsoft Word such as auto
-
correct and spell checking. In order to allow for
these options, work was conducted into ways of controlling what aspects of a software package can
be disabled to

prevent acces
s to the internet

(possibly

via the research and help options
)

and also
to
prevent access to files students had produced and stored previously either on USB devices or
university provide
d

files space. This investigation found a product called Appsense
6
, wh
ich allows
many applications to have their menus and functionality to be exactly tailored to meet particular
needs. The Appsense system was used for one of the trial examinations run on the new Sun Ray
cluster in January 2009 to provide access to a file in

Excel, which could not be saved and could not
be u
s
ed to access the internet.


After much discussion, there were too many unresolved issues round word processing exams and the
storing of files
which

need to be more fully investigated to risk running a su
mmative assessment yet.
The issues are detailed in the project Issue Log
7
.
Hopefully this investigation can be continued under
the proposed benefits realisation follow up project.



The trial examination
,

which would have allowed students to word process their exam answers
,

failed
at the last moment for two different reasons. The first was that the class size was so large it had to be



6

http://www.appsense.com/Files/Documents/University%20of%20Bradford%20(US).pdf

7

http://www.brad.ac.uk/lss/jisc/issuelog.pdf

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
7

of
19

split into three cohorts for the exam; this would mean that effectively
the last group would have well
over an hour to research the exam questions after

the f
i
r
st cohort had left the exam. This is obviously
unacceptable. The second reason was that the students were in Life Sciences and the exam
questions will often benefit fro
m diagrams being included in the response; this is not easily possible
with current word processing and drawing packages. The exam was therefore taken in two separate
parts; the first online using Questionmark Perception and the second in a large hall wher
e all the
students could hand write their responses in a single sitting.

Smart card

technology

The use of
smart cards

was another thorny issue that was not resolved in time for the January exams.
The plan was to link the image for a particular exam to a smart card; this would allow for multiple
exams to be held in a single location with student simply being told to use a

workstation with a red
triangle or blue square etc on the card.


The University has been issuing Mifare based smart cards to all staff and students for several years;
the chip was initially used for photocopying applications in the Library and more recen
tly also by the
access control system being introduced across the University. To find out more about smart cards
and their uses within the HE community, the project manager tried to join the mailing list
sma
rtcards@jiscmail.ac.uk
. However, the list was configured so that joiners had to be approved by
the list owner, who seemed to have gone inactive. After contacting the JISCmail list manager, the list
is now operative again and has increased from 12 to
54

mem
bers
, with the author as list owner
.


P
ostings to the
Smartcards list introduced the Higher Education Smart card Association (HESCA
8
).
This has proved to be a very useful forum for staff looking at a wide range of services that can be
provided with smart
cards; however, the technology required by the card readers in Sun Rays does
not appear to be in wide spread use for other applications, so deployment at Bradford will require dual
technology cards with both the Mifare close proximity chip and the JCOP jav
a enabled card
;

the new
JCOP31 chip is alleged to provide both sets of functionality
but
it has not been tested
as

the price of a
single JCOP31 card is higher than one wi
th both Mifare and JCOP10 chips
. Having discovered what
was needed for the January exa
ms and for staff who would potentially benefit from having both Mifare
and JCOP in a single card, an initial order was placed. The design of the cards and the positioning of
the chips on it to avoid areas that would need printing, took several weeks. The c
ards eventually
arrived a few days before Christmas and didn’t work with the Sun Rays! Subsequent investigations
showed that the JCOP chip had not been initialised and the cards have since been sent to the JCOP
manufacturing plant in Austria to have this d
one. Smart cards for use with the Sun Rays were
therefore unavailable for use during the January 2009 examinations.


However, dual technology chip smart cards have
now
been ordered and will become the standard for
new students arriving at the University fr
om summer 2009, so that this technology can be used in the
future. The ability to mix different exams within a single room may become important as the number of
modules assessed online increases further; linking a student’s own card to the appropriate exam

would allow this. Alternatively programming a smart card with a particular symbol on it to only link to a
particular exam could enable the exam schedulers to tell students to sit at a ‘red triangle’ or ‘blue
square’ workstation and distribute them appropr
iately round the available seats. Use of this smart
card technology will only work with thin clients which have a smart card reader, eg Sun Rays.

Examination Administration

The
integration of examinations

scheduling

with Computer Aided Assessment and stud
ent record
systems has proved to be problematical. At the start of the project, the University ha
d

recently
purchased a new module from the suppliers of the student record system, SITS
,

and a project board
had been set up to oversee its implementation. The

new module claimed to integrate information
about students’ disabilities with the exam schedules so that extra time and special facilities required
were taken into consideration; since this was one of the goals of the ITS4SEA project, it should have
made
the project very easy indeed. After attending training courses and trying to use the new module,
WASP, to schedule the May 2008 examinations, it was found that it was not fit for purpose. It did very
little with the disabilities data and could not split cl
asses into multiple cohorts so that they would fit into



8

http://www.hesca.com/

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
8

of
19

the size of rooms available. This was a particular issue for online assessments that would eventually
be held in the new Sun Ray facility as the class sizes were in excess of 300 in some instances. So
me
6 months into the project a decision was made to revert to the original exam scheduling programme
and develop the code necessary to integrate it with the module and student data information.

A report
on the deficiencies of the WASP exam scheduling softw
are has been produced but not made
generally available due to its commercial sensitivity


As a
result

of the unsuitability of the WASP scheduling system,
a system specification for
a

new Exam
Admin system
was drawn up and
finalised in August 2008. Unfortun
ately this did not allow sufficient
development and testing time for the new system before the January 2009 examinations had to be
scheduled. The decision was taken to timetable the January exams based on the processes in the
system design but to actually
implement the new system during March 2009 when the May 2009
exams would have to be finalised. One major improvement for the January exams was the production
of a personalised timetable for each student of the University which was sent out by email in the
last
week of term before Christmas.

This has been repeated for the May examinations.

The automatic
allocation of additional time and separate locations for students with specific exam needs has seen an
improved take up of special facilities since the
introduction of the new exam administration system.


The development of the system has proved demanding due to difficulties with the reliability of the data
that has to be processed. After two attempts to automate the generation of an examination timetable

during the course of the ITS4SEA project, the level of incorrect data related to type of assessment,
length of assessment and students enrolments on modules has meant that far too many alterations
are still required to be made manually. Correcting the dat
a will take some considerable time and effort

but tools have been developed to assist with this process and will be made available on the project
web site
.

It is noticeable that some areas of the University who engaged with the data cleansing
process had o
nly 3% of modules requiring manual intervention whereas other areas required over
60% of their exams to be handled manually.


As part of the investigation of examination processes, the possibility of linking the exam timetable
directly with the Questionmar
k Perception system was evaluated. “
Questionmark Web Integrated
Services Environment, QMWISe, is a comprehensive series of web services methods that act as an
Application Program Interfaces (APIs) that enable registration, management and reporting systems
to
tightly integrate with Questionmark Perception”
9
. While QMWise provided the tools to allow tests to be
scheduled for a particular time on a particular date and also to extract the results files in order to
process them directly
into the student record s
ystem, it actually proved impossible to automate the
online assessments in this way. T
he unreliability of the data of which student
s were enrolled on which
module,
which part of the assessment related to the online examination

and which students were
repea
ting the assessment proved to be insurmountable.


However, a major reduction in the administrative burden
of exams
has been achieved in the transfer
of some paper based multiple choice assessments into online assessments, where no post
-
exam
administration
is required to create the results file in a format that can be processed directly into the
student record system for the majority of students.

A simple expedient of modifying the results files
produced by Questionmark Perception to include the student numb
er as well as their computer user
account name has also saved tedious manual processes importing the results into the student record
system.


The earlier Pathfinder project had developed strong working links
between

staff in the central
examinations office

and IT Services
and streamlined processes around the booking of IT facilities
during the exam period. It had also developed improved regulations on the conduct of online exams
10
.
All these previous activities were essential to the success of the ITS4SEA pr
oject but that has
highlighted yet more work that needs to be done in these areas; these are highlighted in the Risk
Register and Issue Log for the project.




9

http://www.questionmark.co.uk/uk/perception/qmwise.aspx

10
http://www.brad.ac.uk/elearning/pathfinder/do
cuments/CAA_policy.pdf

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
9

of
19

Estates and E
-
coversity

The Estates issues have also proved interesting. There appeared to be little research into appropriate
layouts for computer aided assessments, so an email requesting advice and information was
circulated round various
email
lists. It generated a lot of int
erest and many useful replies with many
photographs and pointers to other resources. These have been summarised in a separate guide to
designing assessment facilities. The architects appointed to design the new assessment room took on
board the information

which had been received and
designed

a variety of different possible layouts
based round furniture that was commercially available. These designs were discussed by the project
board and agreement reached that having a room with the screens facing towards
each other would
minimise the possibility of viewing another student

s screen during exams. This philosophy led to the
eventual design that was constructed.




The room is split into two unequal sized spaces with a fully glazed divider between them. The
smaller
part can be used either for scheduling students who have been allocated extra time for exam
s

or for
small group teaching leaving the large part for ad
-
hoc use by students. The size and shape of the
room mean that standard projection facilities for
teaching are not appropriate, so screen management
software has been investigated though not yet deployed
; it is planned to make Net Support School
available for
teaching for the

2009
-
10 academic year
.


The
University of Bradford is engaged in a University

wide
E
-
coversity Programme
11

which
aims

to
embed the principles and practice of sustainable development across the entire institution by getting
people in involved, taking the lead on issues, and encouraging and making it easier for people to
adopt sustai
nable behaviours and lifestyles
’. It
has also hosted another JISC funded project
investigating
Sustainable

I
C
T
in Further and Higher Education
.

The two JISC projects have met
several times over the
project’s life

to try and analyse whether thin client syst
ems are more cost
effective and more environmentally friendly than standard PCs. As part of the SusteIT project, two



11

http://www.brad.ac.uk/admin/ecoversity/

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
10

of
19

tools
12

have been d
eveloped to test this hypothesi
s
, however,

the server and disk configuration to run
more than 100 fully configured
Sun Ra
y thin clients
h
as only just be
e
n

finalised,

so
the output from
these tools
h
a
s still to be fully evaluated.



Sun Microsystems

have
a
strong commitment to minimising the environmental impact of its
operations and

th
ey have published both a returns policy
to ensure that all equipment purchased from
Sun can be returned for recycling
13

and also a statement on the
European Union Directive on Waste
Electrical and Electronic Equipment (WEEE), which took effect on August 13, 2005
14
. It is notable
however that Sun
Ray workstations do not have a power switch included in the design to allow them
to be easily switched off when not in use. This is a part of the design to provide longevity of the
equipment as it is often a manually operated switch which cause
s

equipment
to fail; Sun claim a life
time in excess of 10 years for their thin client workstations. The decision was taken however that 100
Sun Rays left on 24 hours a day for 365 days a year would be an unacceptable increase in power
consumption so, in an attempt to

encourage student users to turn off the Sun Ray workstations when
not
in use
, the power cables to each unit

s transformer have been modified to include a power switch;
as Sun have surmised, these have not proved very reliable to date.

Quality assurance an
d
project management

The project board met 6 times during the course of the project, with two external members for the final
5 of these. Our grateful thanks go to Professor Mark Stiles from Staffordshire University and David
Lewis from the University of Gl
amorgan for their
enthusiasm and constructive criticism. The project
board also contained members of the Teaching Quality Enhancement Group and Academic
Standards Unit of the University to ensure that what was being developed fitted with University
policie
s and processes.


There were two project teams who met regularly during the project; one concentrating on exam
timetabling and integration of administrative systems and the other on the technical development of
the Sun Ray cluster and infrastructure. The b
uilding work was project managed by the University
Estates department and their external project management partners, PSK Prout Tilbrook.

Outputs and Results

The following quotes exemplify the major achievement of this project.


I’ve already told the DVC t
hat we will need another cluster like this within 2 years”,
Dr
Nigel
Lindsey, Assoc
iate

Dean
,

Life Sciences and lecturer in immunology


“This is just what I always wanted but we couldn’t do on the PCs!”,

Dr. Bob Lomas, Lecturer in
Technological Management


They relate directly to the successful creation of a new 99 seat assessment facility utilising Sun Ray
thin client technology and, in the second one, the use of Appsense to control the software which was
made available for the examinations.

The creation o
f the 100 seat assessment facility was major
achievement in itself; finding a large enough empty space and also persuading
Estates
sta
ff that an
assessment facility wa
s the best use for it was a major
achievement

and need
ed

support from
academic staff who
w
ould

benefit from the development at a very early stage.



Following the successful tria
ls involving over 400 students i
n the January 2009 examination period, a
further 9 new modules were timetabled to use online assessment for summative assessment for th
e
first time at the end of semester 2
,

covering over 900 student assessments.



Feedback from students taking their exams online has also been encouraging; in one group of 163
students, 128 agreed or strongly agreed with the statement ‘
Would you like to ha
ve other summa
tive
assessments in this format
’. However, it must
also
be noted that 14 students disagreed or disagreed
strongly. Comments ranged from ‘
For most modules, summative and if possible, real tests / exams



12

http://www.susteit.org.uk/files/index.php

13

http://www.sun.com/aboutsun/environment/returns
-
intro.jsp

14

http://www.sun.com/aboutsun/ehs/ewaste.jsp#weee

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
11

of
19

should be done on the computer, there's a

lot less pressure and it helps to relax and maybe even do
better
’ and ‘
This format was perfect for me, it was much less hassle compared to the usual hassle
during exam times’ through to ‘
I prefer it for shorter tests, however, for MCQs with over 50
or so
questions, I prefer paper’ and ‘
Your eyes start hurting staring at a screen, you can't think. Your wrist
starts to
hurt too whilst using the mouse’.
Analysis of a second set of exam candidates is attached as
Appendix C.


Several students mentioned
problems with the screen and the font size used. It is planned to make
footrests and monitor stands available for the 2009
-
10 academic year so that the impact of these can
be assessed. It is possible to change the contrast and brightness levels on the Sun
Ray monitors,
although how to do this is not obvious. The magnification of text on the screen can also be enlarged.
For the 2009
-
10 assessment periods, we will provide all exam candidates with an instruction card on
how to get comfortable

and

how to modify

the screen to suit their personal pr
e
ferences

to assess
whether this improves student feedback on taking exams online instead of on paper.


An unexpected outcome from the development of the new 100 seat facility and the way the
workstations have been arra
nged is a request to use the room for a new way of teaching. One
lecturer whose traditional lectures have been poorly attended and the content poorly understood is
planning to replace his lectures with small group working in the new facility; he will set e
xercises to be
done in groups and go to groups who are having difficulty with an exercise to coach them in the
problem. It will be interesting to see how this approach works. It could not have been accommodated
in any of the existing P
C

cluster
s

on campus
where the layout would not encourage this type of
working and the space between workstations would hinder staff moving between the groups.


There were several major benefits to the e
xamination processes
during the January 2009
period



There was

no need to c
heck and scan
190 O
ptical Mark
R
ecognition

sheets

thus saving
approximately 9 hours work
; the marks were available at the end of the afternoon

for import
into the student record system
.



Students with extra time allowed were scheduled into
the area adjacent

to the main
assessment room

and started at the same time as the bulk of the class who could leave
without disturbing them
.



Student
s

were enrolled faster against the appropriate modules in the student records system

although major issues remain with the re
liability of data for both module assessment modes
and student enrolments
.



There was an increased

take up of specialist facilities by students with disabilities

as they
were informed individually where and when their assessment would occur compared with th
e
previous system of having to look up each module and see where the assessment was
scheduled
.



Personalised examination timetables were emailed to all students
. This highlighted to every
individual whether they were entered for the correct assessments and
led to many changes to
modul
e

registrations at the last minute
.



Students requiring extra time, or their own separate room for exams, were scheduled within
Syllabus+ for the first time. Previously these students were scheduled manually.



ExamAdmin

provide
d

a flexible and easy to use interface that g
a
ve the
examination Office
staff control

over which examination records
we
re extracted from SAINT. Previously this was
hard coded in a PERL script, and not under the user’s control.



The automatic production of d
ata quality reports so that academic schools could validate the
accuracy of SAINT held examinat
ion data before any scheduling wa
s performed.


Work on the examinations processes during the ITS4S
EA

project also resulted in a r
eport on the
shortcomings of the

WASP exam scheduling product from SITS
. These findings were validated given
the increasing number of students with additional requirements and the proliferation of multi
-
session
PC based exams which WASP would not have been able to cope

with. An additiona
l benefit of the
exam administration work was the cleansing and enhancing of examination data in SAINT prior to
scheduling, based on the results of Data Quality reports

that were circulated to administrators in
Schools before the exam timetable was publish
ed
. However, there is a noticeable reluctance of
academic schools to adopt changes in procedures or engage with the data quality process; there was
a better response for the May exams than there was in January.


Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
12

of
19

A number of advances have also been made in
the running of o
nline assessments
:



The a
bility to improve
the
security of Q
uestion
m
ark
P
erception

assessments by controlling
the
IP addresses
which can be
used to access the server
. This was thought desirable

since
Q
uestionmark

does not
have an

IP address
checking facility

currently. Although this has
developed it has not so far been deployed due to last minute
networking
difficulties.

It will be
used to increase security for online assessments during 2009
-
10.



The ability to control what software and
services are made available to the exam candidates
by using Appsense has also encouraged some more sceptical academics to use computer
base
d

assessments for the first time in the
May
examination period.



The 10% failure proposed in the Bradford CAA exam gui
delines for PC clusters has not
proved necessary when using Sun Rays for online assessments. Two examinations were run
successfully for 94 students in the 99 exam seat facility in order to avoid running them over 2
sittings.



The reputation and benefits of
online assessment over
Optical M
ark
Recognition
based
examinations have been g
reatly enhanced
across the institution. Good news spreads quickly
amongst the academic community!



Student feedback

has also been encouraging but more assessment of their views an
d any
associated issues is required. Although at the design stage the potential need for footrests,
monitor stands etc was identified, these have not so far been made available or suggested to
exam

candidates.


Some potential difficulties with online asses
sments have also been identified:



In order to run
exams
using a word processor, if all students cannot take assessment in one
sitting then a bank of questions may be
need
ed so that students in cohort 3 cannot be told
what the possible questions are by coho
rt 1
. If only 2 sittings
are required then the second
cohort
could be
kept separate from the first by requiring them to be present at an alternative
location until after the first sitting has finished and then supervised while they move to the
exam locatio
n.

This is also the case if a multiple choice exam is run online in multiple sittings
but the questions cannot be randomised or chosen from a bank of questions of similar
difficulty. One such exam was run during the May exams and the second cohort were
req
uired to arrive at a separate location shortly before the first cohort were due to leave the
exam room; this worked successfully.



Some examinations may not be suited to w
ord processing,

for instance
if
they would benefit
from diagrams being included
.



The saving of word processed scripts and providing secure, anonymous delivery to the
marker
was not resolved. While it is administratively straight forward to ensure that the word
processor package is configured to save the files created by the exam candid
ates to a
location from where it can be automatically extracted (using Appsense), how to make it
available for marking is less obvious. Other issues were around how to separate different
answers to different exam questions; how to deliver different answers

to different academics
for marking; should marking be done online or on paper; should questions for answering be
supplied on screen as a part of the word processor file or delivered on an exam paper; should
students be allowed to opt to word process an ex
am

instead of hand writing their answers; do
word proces
sed exams disadvantage those who cannot type well; should extra time be
allocated for word processed examinations.


The e
nvironmental
aspects of the project have also been challenging. Although assist
ance has been
given to the
SusteI
T project to produce
their
carbon calculator and total cost of ownership calculator
for thin clients
, we have so far been unable to complete these to see the figures for the 100 seat Sun
Ray cluster due to the problems fina
lising the server and disk configuration.
It is planned to calculate
these figures prior to a UCISA conference to Advisory and IT Support staff in July 2009
15

where they
will be presented alongside other green IT topics and the use of thin clients for secur
e e
-
assessment.
An attempt has been made to save some power by including power switches in the mains leads for
the Sun Ray workstations so they can be turned off easily. There is the potential to save enormous
amounts of paper by converting
OMR exam papers

to online assessments
and
thus also saving on
the
results sheets
; though at the current time both are still being produced to cover any failure of the
IT systems




15

http://www.ucisa.ac.uk/groups/ssg/sdg/Events/2009/SDG2009.aspx

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
13

of
19


The t
echnical issues

involved in the project have been a challenge due to the use of new vir
tualisation
software. We have gained a g
reatly increased knowledge of Sun Ray server technology

and
increased experience with VMware Virtual Desktop Infrastructure

(VDI) during the project. Whether
the use of Sun Ra
ys and the

servers
they require will
p
r
ov
e to be financially advantageous
or more
eco
-
friendly than using standard PCs is unclear currently but use of Sun Microsystems own calculator
shows that c
omparing
t
raditional
PCs

with
Sun
’s

Virtual Desktop Solution for 100 Sun
Ray
clients
r
esults in
a

£183
,932
advantage
in total cost of ownership
for
the thin client solution over 3 years
.
Much of this is claimed in reduced support costs and less down time for users

rather than in hardware
or power consumption costs
.


The University of Bradford has also gain
ed a much b
etter understanding of creating virtual desktops
and assigning them to users, smart cards or workstations

during this project. While the use of smart
cards has not so far been successfully introduced into the examinations process, the
option

wil
l be
available for future years. The ability to use
Appsense to both control applications software packages
and to manage virtual desktops

has been an incredible benefit to both the security of the
examination
s

and to the delivery of the virtual desktop fo
r each exam. We are still investigating the
use of
NetSupport School to control screens on thin client workstations for teaching purposes

and

hope to deliver this for teaching in the 09/10 academic year.


The use of s
mart cards

within the University continues to evolve as it does at most other Higher
Education sites. The impact of the recently announced requirement to report on international students
to the UK Borders Agency has become highly relevant and the
smartcards@jiscmail.ac.uk

list has
been revi
talised

after being apparently dormant at the start of this project. It now has 54 members
compared to 12
initially
.
Dual technology
MiFare

and JCOP cards
have been
ordered to allow
staff
and students
to be save
Sun Ray sessions
and reactivate

them later,
as well as
for
access control,
printing and copying functionality
.



Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
14

of
19

Outcomes

The table below gives the Outcome
s

expected from the original project plan. The final column shows
what the project has or has not achieved towards the planned outcomes.


Outcome

Impact

Change

Project achievements

and progress

Improvements to
exam facilities
for disabled
students

Automatic
extension to
time allowed for
exams

Less stress
for students
and
invigilators



䑩s慢l敤⁳t畤e湴n⁢ in朠gorr散瑬y⁰ick敤
異⁦r潭⁓A䥎I⁡ 搠dll潣慴敤⁥ 瑲愠aim攠
慮搠de灡r慴攠牯潭⁡ 瑯t慴ac慬ly



䡡H攠eo琠td摲敳s敤 iss略s⁷ 瑨⁤ sa扬敤
s瑵t敮瑳 睯w搠
灲潣敳si湧

t桥ir⁥ 慭s



S瑵t敮瑳⁳im灬y⁲敱uirin朠g瑲愠aim攠
慬l潣慴a搠d摪oini湧⁳灡c攠e漠o瑨敲e
s瑵t敮瑳 i渠湥w 慳s敳sme湴ncl畳瑥t

A扩lity 瑯t
瑩m整e扬攠em慬l
cl慳s敳⁦潲o
sim畬瑡t敯畳
潮li湥
慳s敳sm敮t

䱥ss⁣潮flic琠tf
s灡c攠ee敤s⁢y
E慭s⁏晦ic攠

剥摵ce搠
c
r畳栠h渠
數慭
瑩m整e扬e



乯琠N敥d敤⁦潲⁊慮u慲y⁴ ial⁥ 慭s⁳i湣攠
瑨ts攠睥w攠e慲来⁣l慳s⁳izes



䱩k敬y⁴漠o攠ee敤e搠d潲⁁og畳琠牥
-
si琠
灥riod

A扩lity 瑯tm潲攠
敡sily⁣h慮g攠
煵敳瑩潮s⁴ 慮
潦f敲敤⁢y⁏䵒

P慳琠t慭
煵敳瑩潮s敥搠
湯琠t攠e数琠
s散r整

Ac慤敭ics
c慮

慬瑥爠
煵敳瑩潮s
敡sily⁦r潭
ye慲⁴漠y敡r



Eis瑩n朠g㔠位删煵敳ti潮s⁦潲o
䥭m畮潬潧y⁣潮v敲e敤⁴漠QmP



Or摥r ⁱ 敳瑩潮s⁲慮摯mis敤



䍯Cr散琠t湳w敲⁴漠o略s瑩潮s⁲慮摯mis敤

A扩lity 瑯t潦f敲e
摩ff敲敮琠eyp敳
煵敳瑩潮s 桥r
than ‘Select A to
E’

䵡y 湥敤⁴漠
e
湧a来⁲el畣瑡湴n
慣慤敭ics.

䵡y 湥敤
慤di瑩潮al
瑲慩ni湧 慮搠
s異灯rt

p瑵t敮瑳⁣慮
慴a敭灴p灡s琠
數慭⁰ 灥rs
f潲⁦潲m慴av攠
慳s敳sm敮t



Pr潶id敤 Ec敬⁦il攠e潲湥 數慭⁷i瑨潵琠
潰瑩潮s⁴ ⁳慶攠el瑥牡tio湳爠慣c敳s
i湴nrn整e



Tim攠e敥摥d⁴ 摥v敬潰 睩摥r⁲慮g攠

煵敳瑩潮s⁡湤ⁱ略sti潮 扡nk f潲敷
䍁A⁥ 慭s

A扩lity 瑯tm潶攠
瑲慤iti潮慬 wri瑴敮
數慭s⁴  lin攠
慳s敳sm敮t

Wo畬搠牥煵ir攠
慤di瑩潮al⁳瑡tf
瑲慩ni湧 慮搠
敮条来m敮琠
睩瑨t湥w
瑥t桮ol潧y

䱥ss⁴ m攠
c潮s畭i湧
m慲ki湧



S瑵t敮瑳ay 敤⁴漠o湣lu摥⁤i慧r慭s



T
桲敥⁥慭 si瑴tn杳敡渠ni湡l⁣潨潲琠will
k湯w wh慴a敳say 瑩o湳⁡r攠睩瑨⁥ 潵g栠
瑩m攠e漠牥ois攠eo灩cs



E慭⁴ 灩cs 敤⁴ 扥 v慲i敤 扥t睥敮
c潨潲瑳



䵵c栠h潲攠睯ok 敤敤 o渠nhis

E慳y
m慩湴n湡湣攠潦
摥m潮s瑲慴ars
com灵瑥爠tn
l散瑵牥⁲o潭s

䱥c瑵牥t⁨慳⁵

瑯t摡瑥ts潦tw慲攠
imm敤i慴aly
慶ail慢l攠e渠
慲aival⁡琠t散瑵牥t
r潯m

乯NWi湤潷s
異摡瑥tⰠ,湴n
-
vir畳⁵灤慴as
慮搠d潦tw慲攠
灡瑣桥s⁴漠
i湳瑡ll



啮慢l攠e漠oi湫⁤ 瑡ts潣k整ei渠n散t畲攠
r潯ms⁷i瑨tb潴o 副慭乥琠慮搠S畮⁒ y
s敲e敲e⁤ e⁴ 桡lf⁣潭灬整敤慣桩湥
r潯m⁡
湤 t睯wk⁲潵瑥爠m潶敳



䑵慬⁤ t愠a潣k整e 眠w瑡t摡r搠d渠
r敦畲扩s桭敮琠tf†瑥慣桩n朠慲敡aay
慬low⁤数loym敮琠i渠n〰9
-




S畮 剡ys 湯琠t畲ue湴ny⁳uit敤⁴ ⁲畮nin朠
慬l⁴ 慣hin朠gp灬ic慴i潮s⁥.朮gFl慳h⸠
Firmw慲攠異杲慤攠摵e⁳畭m敲′〰㤠9ay
慳sist

剥R畬瑳⁦
r潭
潮li湥
慳s敳sm敮瑳
s瑯牥搠t渠n瑵t敮琠
r散潲搠oys瑥t
慵瑯t慴ac慬ly

E慭⁲敳畬瑳
慲攠慣c畲慴攮

䱥ss⁴ m攠
湥敤敤 by
慤mi湩s瑲慴av攠
s瑡tf⁴ ⁰牯 敳s
r敳畬瑳

乯N
瑲慮scri扩n朠
潦⁲敳畬瑳
扥twe敮
摩ff敲敮琠
syst敭s⁡ 搠
fil攠e潲m慴a



S瑵t敮琠湵m扥r⁩湣l畤e搠dn
潵瑰畴ufil攠
from⁑mP⁡ ⁷敬l⁡ ⁳瑵te湴n湡m攠e漠
慬low⁥慳y 摡瑡⁴牡湳f敲⁩湴漠o瑵te湴n
r散潲搠oys瑥t



䑯湥慮畡lly⁦潲⁍oy‰㠠慮搠da湵慲y
〹⁥ 慭s



T潯 m慮y⁩湣潮sis瑥湣i敳 i渠nss敳sm敮琠
typ攠e潤敳⁴漠o畴um慴a⁴ is⸠䑡.a
Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
15

of
19

Outcome

Impact

Change

Project achievements

and progress

cleansing of module descriptors plann
ed
for summer 2009



䥭灯ssi扬攠e漠桡n摬e⁲数ea瑩湧⁳瑵te湴n
慵瑯t慴ac慬ly
.⁕ lik敬y 瑨is⁣慮⁢
慣桩eve搮

Pr潶isio渠nf⁡
l慲来a
慳s敳sm敮琠
cl畳瑥爠

Will⁡ low l慲a攠
杲潵灳⁴ ⁢
慳s敳s敤⁩渠愠
s桯r瑥爠瑩m攠
fram攠e湤⁵ 攠
l敳s⁩n摩vi摵al
sm慬l⁐C⁲潯ms

F敷敲e
i
nvigil慴ars
r敱畩red



S灡c攠e潲‱〰⁳敡瑳⁤ velo灥搠⡩湣⸠
l散瑵牥t⁳瑡tio温



䍯C灬整e搠慦瑥爠敮t 潦⁴ rm⁄散em扥r
㈰〸⁦潲⁵o攠wit栠h慭s⁦irs琠睥tk
㈰〹!!




䍬畳瑥爠tev敬潰m敮琠瑩m攠ec慬敳
灲敶e湴n搠dri慬s⁰ i潲⁴漠數am⁰ ri潤



E湣潵ra来d⁳瑡tf⁴ ⁴ i湫
慢潵琠摩ff敲敮琠
慳s敳sm敮琠ty灥s



E湣潵ra来d⁳瑡tf⁴ ⁡ s漠ohi湫⁡ 潵琠
慬瑥牮慴av攠ee慣桩n朠gtyl敳

E灡湳i潮 i渠
畳攠ef⁃ A

䵡y i湣re慳攠
r敱畩r敭敮瑳⁦潲o
l慲来aSu渠剡y
cl畳瑥牳

F敷敲e
scri灴p
瑯tm慲k⁢y
桡湤



㤠9e眠w潤畬敳⁴ m整ebl敤⁦潲湬in攠
慳s敳sm敮琠t渠nay ㈰0㤠9ams



Ov敲‹e〠0慭⁳敡瑳⁳c桥摵l敤⁦潲⁍oy
㈰〹


T桥

潲楧i湡l
灲潪散琠tl慮⁰ 潰潳敤⁴ a琠瑨攠e潬lowi湧 d敬iv敲慢l敳 w潵l搠扥慤攠慶ail慢l攠e琠瑨攠en搠
潦⁴ 攠er潪散琮t䵡ny ⁴ 敳攠eav攠扥e渠n摤
r敳s敤 i渠敡rli敲⁰慲瑳 潦⁴ is⁦i湡l⁲数潲琠扵o 愠a畭m慲y⁩s
made following each target to show what was and wasn’t achieved and the main reason behind this.


Major improvements to E
-
administration

Improved interfaces and processes

within the
Examinations

Office


The initial stages of implementation of the Exam
Admin system has achieved this aim

Improved delivery of individually tailored software
configuration
s for computer based assessment

This has been achieved using Appsense to
deliver different configurations for different
exams but
t
he process is still manually controlled

Controlled, reliable IT facilities for students with
disabilities to take examinations with automatic
extension to t
he time allo
wed for the online
examination

Students with disabilities are scheduled into
separate locations so they are not disturbed
when other students finish
. Exam length
controlled by invigilators.

Replacement of existing Optical Mark
Recognition base
d paper exami
nations with
online assessments

Several OMR exams have been replaced during
the January and May exam periods. We
plan for
this trend to continue

by encouraging lecturers to
think about alternate assessment methods.

Automatic delivery of appro
priate assessment to
each student if mixed cohorts of students are
locat
ed in the same examination room

This has not been required so far but should be
achievable if required

Automation of transfer of marks between testing
software and student record system

Processes not automated due to inconsistencies
in coding of assessment and problems with
students repeating assessments

Reduced support requirements for PCs in
examination

and teaching rooms.

Sun Rays have proved MUCH easier to control
and have lower failure rate during exams
.

Some examinations run successfully without
allowing 10% failure rate.

I
nvestigat
ion of

options for allowing anonymous
submission of word processed s
cripts to
examiners without resorting to printing

Not achieved due to
unresolved

problems
related to word processing exams
. Discussion
with other sites to plan ways forward is required
to take place at UCISA E
-
assessment
conference in November 2009
16




16

http://www.ucisa.ac.uk/groups/ssg/asg/Events/2009/eAssessment.a
spx

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
16

of
19

Sustainable developments

Measurement of the resources required to run a large
room of Sun Rays and compare it with the requirements
of similar facilities based round networked PCs.

The final server infrastructure to support 100
Sun rays was only finalised
as this report
was being produced.
It is planned to develop
t
his
aspect of the project by July 2009
.

Replacement of a large number of tests which currently
use paper based Optical Mark Recognition with online
assessment, thus removing the need for large
quantities
of paper and the laborious process of scanning it.

This has been achieved for an additional 10
modules covering over 1000 student exam
papers.

Major Deliverables

Comprehensive documentation on all
aspects of the new system which will
allow othe
r universities to explore
similar applications, e.g. full
documentations on how to develop
different server images for delivery to
Sun Ray terminals for computer based
assessments.

Following the successful use of the Sun Ray cluster during
two exam
periods, it is now vital that this work is documented
to allow for the sickness or resignation of the member of staff
who ran the processes and was indirectly funded by the
project.

This will be made available
by July 2009

at

http://www.brad.ac.uk/lss/jisc/


Scripts and APIs to allow other sites to
develop similar systems integration.

Scripts to improve exam scheduling and student selection
processes will be made available
by July 2009

at

http://www.brad.ac.uk/lss/jisc/


The scheduling of tests into Questionmark Perception and
the extraction of results back into the student record system
were not automated using QMWise due to the unreliability of
data related t
o students taking modules and the coding of the
assessment type
.

Evaluation of the new e
-
assessment
system against a range of criteria,
including educational (e.g. effectiveness
in delivery of a wide range of
assessment types with improved
feedback to stu
dents and staff),
administrative (e.g. effectiveness and
efficiency of procedures) and
sustainability (e.g. comparison of the
power consumption and management
issues relating to use of a large cluster
of thin client terminals against
comparable PC clusters
).

The educational benefits of the new assessment facility are
clear, with immediate results being available at the end of
each exam or test. Initial student feedback also rates the
benefits highly.


The benefits to exam admin staff have also been immediat
e
with no paper forms to
correct and
scan

and the results
available for immediate transfer into the student record
system.

A report on the exam scheduling process will be
available
by July 2009

at

http://www.
brad.ac.uk/lss/jisc/


The sustainability aspects of using Sun Rays vs. PCs
will be
reviewed and data made available by July 2009
.

Use cases documenting user and
stakeholder needs, perspectives and
evaluation feedback

To be disseminated at confererence

http://www.ucisa.ac.uk/groups/ssg/asg/Events/2009/eAssess
ment.aspx

QmP perspective

http://www.questionmark.co.uk/uk/casestudies/bradford.aspx

Appsense perspective

http://www.appsense.com/Files/Documents/University%20of
%20Bradford%20(US).pdf

Student evaluation in Appendix C.

Report on
handling the needs of
disabled
students as a part of large
scale Computer Aided Assessments

Students with special needs who were scheduled to take
online assessments were either timetabled automatically into
a s
eparate area of the new assessment facility if they simply
had extra time allowed or into individual rooms equipped with
a Sun Ray if other facilities were required.

Due to a wide variety of different problems around word
processing exam, work with student
s who word process their
exam scripts has not been achieved.

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
17

of
19

Integration of SMART Sympodium
tablets with Sun Ray terminal
technology.

This has not been achieved due to networking problems
related to single sockets in teaching rooms requiring access
to
bot
h the Sun Ray servers and the University roaming
network.

Evaluation of issues of using thin client
technology in teaching rooms instead of
networked PCs.

As above.

Report on special factors affecting the
design of a new cluster designed for
online summative assessment.

Appendix A to this report

Conclusions

The project team firmly believes that using Sun Ray thin client technology to support E
-
assessment
does provid
e a more secure and flexible solution that using standard PCs
.
We have recently received
news that the University of Hanoi has installed a 100 seat Sun Ray cluster for conducting online
assessments and plan to roll out duplicate facilities across Vietnam
.

They have obviously reached the
same conclusion
!

Although smart cards were not actually used for the first trial examinations, they
should also assist in providing a more flexible and accessible solution for future examination using the
new Sun Ray facilit
y.


Integrating the information stored about student disabilities in the Student Record System with the
exam scheduling systems improves
the
interface
s

and processes for staff in the Exams Office.
The
risk
s

and time
associated with manually linking results

of online assessments
to individual students or
double entry o
f

marks into
the student record system

have been minimised by post processing the
output file produced by Questionmark to include the student’s number as well as their login name
.
Reducing, or
better still removing, the use of paper based multiple choice examinations minimises the
work required of exam administrators in academic Schools whilst increasing the security of the
assessment since questions can both be randomised in the order of delive
ry and in the correct
response. However, these process
e
s

rely on the original data being used by the Exam Admin system
be
ing

correct. The issue log
for the project
highlights a large number of data
errors
that must be
resolved before the new system can fun
ction anywhere near 100% of its capabilities.


We faced a number of issues surrounding storage on the servers supporting the Sun
Ray
workstations, notably with the amount of time required to clone the virtual images and coping with the
load from such a hig
h number of consecutive logins.
H
igh performance
disk arrays are essential to
handle these problems in a realistic timescale.

Implications

There are a number of issues logged which still require resolution. Some have local work rounds but
others will need

University procedures adapting in due course.


When implementing any of the aspects of this project, sites should ask themselves the following
questions.

Technology



Desktop
virtuali
sat
i
on



what is the most appropriate system, what server infrastructure does
it require, is it cost effective in the long term, does it reduce staff support costs
?



Thin clients


which is the most appropriate
platform, do they reduce your carbon footprint, do
y
ou need smart cards for them to work most flexibly, what applications are they suited for,
what
software
may

not run appropriately
?



By changing the technology used to deliver online assessment
s
, can the
equipment f
ailure
rate allocated for exams be reduced
?


E
-
assessments



Does controlling the services that students have access to during online exams using
Appsense increase the opportunities for conducting an exam online instead of

on
paper
?

Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
18

of
19



Can the expense of building a large thin client cluster be
justified by the savings in staff time
running assessments online instead of on paper
?



Does doing assessments online
affect

the student experience
?

Exam administration



How can you ensure that data held about students
module enrolments are correct

to allow
for
exam timetable accuracy
?



How to ensure the correct allocation
of appropriate facilities for students with special needs
?



How can you ensure that data about module assessment are correct
?

L
ength of exam, time
of year of assessment, part time vs. full ti
me vs. overseas options all need consideration
.



Is it possible to transfer results of online assessments directly

into the stu
d
en
t record i
f

assessment types are coded consistently
?



How can you layout workstations to maximise security during exams
?



How bes
t to
disseminat
e

individual examination timetables, e.g.
personalised email, directly
into personal diary, student portal
,
VLE
, SMS?



Can the theories in the design of the
ExamAdmin

system be incorporated into the
examination processes

to provide a

flexible real
-
time reporting system
that can also be
used
to drive data cleansing?

Recommendations

Project management



Do not underestimate the amount of project management time that will be required for
projects with multiple goals to achieve. Two days a week were needed towards the January
2009 exam period to handle building work, Sun Ray developments, smart card issues
and
data quality problems.



Writing final and completion reports is very time consuming after 18 months of continuous
development and implementation work.

For HEIs developing E
-
assessment
facilities



Get a realistic assessment of how many exams can reasonabl
y be moved to an online format
and assess how many students would take each of the exams in order to size
the
optimum
number of workstations required in any new assessment facility.



Get the support of all the academic staff who would benefit from the intro
duction
of a new
assessment
facility to back a bid for space and budget
.



Do not start development of a new assessment facility with only 7 months to go until it is
needed! The Bradford University
faci
lity

was approved in mid

April and handed over in mid
-
December prior to exams starting on January 5
th
; this did not leave sufficient time for testing
and student familiarisation even though the builders did not run a single day late!



Make sure all students hav
e an opportunity to familiarise themselves with new examination
formats and locations to minimise any additional stress during summative assessments.



Give students a formative assessment before any high stakes exams and ask them for their
feedback on what
facilities would help them adapt to the new online format more comfortably.
Make sure you address what they tell you.



If existing PC facilities are being redeveloped, try to ensure there is room for invigilators to
move between workstation
s
; this may requi
re the number of workstations to be reduced.
Consider if using different shaped tables would
minimise

screen visibility between
neighbouring workstations; this could require physically different PC configuration such as
piggy
-
backed system units.



Project Acronym: ITS4SEA

Version: 2

Contact: s.j.eyre@bradford.ac.uk

Date: 1 June 2009


PAGE
19

of
19

For HEI
s seeking to improve their E
-
assessment processes



Make sure there are robust guidelines about the running o
f

PC based exams.



Ensure staff are properly trained in producing online assessments that are fair, secure,
appropriate and accessible.



Ensure you hav
e an appropriate assessment tool that is known to be reliable and flexible in its
question formats.



Decide in advance how to handle multiple exam sittings. This will depend on the number of
cohorts and whether the online assessment
ha
s

to be

identical for all candidates.

For HEIs planning to improve their examination timetabling



Engage examination administrators within academic schools with a data quality and cleansing
exercise before any steps are taken to automate examination processes cent
rally.



Before trying to automate scheduling of exams,

ensure that details of module assessment
type, length and date are correct


MAJOR task!

Tools which may assist with this for SITS
and Syllabus Plus customers will be available from
http://www.brad.ac.uk/lss/jisc/

by the end
of June 2009.



Before sending personalised exam timetables to students, ensure
that student enrolments
against modules are correct


another MAJOR task
.



Ensure students who are entitled to special arrangements for examinations have the correct
entitlement recorded and that the details
are

available electronically so examination locations
can be allocated to them automatically and appropriately.




Inform exa
mination administrators within academic schools that they cannot tell students
when their exams will be held before the exam timetable is produced.

Consider the t
echnology

options



Investigate the potential of using thin client workstations for examinations
. They are much
easier to re
-
image between exams than standard PCs so more exams can be schedule
d

into
a location within the day.



Do not assume that a
small,
trial thin client cluster will scale upwards seamlessly.



Consider the use of thin client
technology in teaching rooms as they should provide a more
reliable, easier to support technology than standard PCs

but may not
currently
be suited to all
applications
.



Use the tools developed by the SusteIT project to decide if the total cost of ownership

and
carbon emissions of thin clients
are

advantageous in your situation.



Investigate tools such as Appsense which could be used to make the online environment
much more
secure during exams t
han running standard web browsers and
applications
software.



If i
nterested in smart cards and their uses, join the smartcards@jiscmail.ac.uk mailing list and
the Higher Education Smart Card Association
17
.

Appendixes

Appendix A.

Designing a new facility for Online

Assessments
18

Appendix B.

ExamAdmin Draft Functional Design Specification
19

Appe
ndix C.

Student exam evaluation
20


Thanks to Dr Liz
Carpenter, FHEA, Lecturer in Medical Physiology, Clinical Sciences, School
of Life Sciences for use of her student evaluation





17
http://www.hesca.com/

18

http://www.brad.ac.uk/lss/jisc/layout.pdf

19

http://www.brad.ac.uk/lss/jisc/examadmin.pdf

20

http://www.brad.ac.uk/lss/jisc/evaluation.ppt