JISC Final Report

hotdogrelishnoseSoftware and s/w Development

Nov 4, 2013 (4 years and 1 month ago)

315 views

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date:29/10/2011


Page
1

of
20

Document title: JISC Final Report Template

Last updated : Feb 2011
-

v11.0



JISC Final Report


Project Information

Project Identifier

To be completed by JISC

Project Title

QTI Implementation, Profiling and Support (QTI
-
IPS)

Project Hashtag


Start Date

14/03/2011

End Date

23/09/2011

Lead Institution

University of Glasgow

Project Director

Niall Barr, University of Glasgow, Learning & Teaching Centre,
0141 330
8179 niall.barr@glasgow.ac.uk

Project Manager

Sue Milne, ELandWeb Ltd, E
-
Learning Services, 0141 943 1307,
sue.milne@e
-
learning
-
services.org.uk

Contact email

sue.
milne@e
-
learning
-
services.org.uk

Partner Institutions

University of Edinburgh, Kingston University

Project Web URL

http://qtisupport.blogspot.com/


Programme Name

E
-
Learning

Programme Manager

Sarah Davies


Document Information

Author(s)

Sue Milne, Niall Barr,
Graham Smith,

Project Role(s)

Project Manager
, Project Director, Consultant

Date

09/09/2011

Filename

QTI
-
IPSfinalreport.doc

URL


Access

This report is for general dissemination


Document
History

Version

Date

Comments

001

27/08/2011


002

09/09/2011

Additions by Graham Smith

005

16/10/2011

Additions by Sue
M
ilne and Graham Smith

006

23/10/2011

Additions and rearrangements by Sue Milne

007

25/10/2011

Additions by Sue Milne and Graham
Smith

008

29/10/2011

Additions and rearrangements by Sue Milne

009

01/11/2011

Changes by Sue Milne and Dick Bacon

010

10/11/2011

Changes by Sue Milne using Programme Manager’s comments

0
11

16/11
/2011

Changes

by Sue Milne

and Niall Barr

0
12

23
/1
1
/2011

Final adjustments

by Sue Milne

and Niall Barr following CETIS
comments


Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
2

of
20

Table of Contents


1

ACKNOWLEDGEMENTS

................................
................................
................................
............................

3

2

PROJECT SUMMARY

................................
................................
................................
................................
.

3

3

MAIN BODY
OF REPORT

................................
................................
................................
...........................

3

3.1

P
ROJECT
O
UTPUTS AND
O
UTCOMES

................................
................................
................................
................

3

3.2

H
OW DID YOU GO ABOUT
ACHIEVING YOUR OUTPU
TS
/

OUTCOMES
?

................................
................................
......

5

3.2.1

Profiling

................................
................................
................................
................................
...........

5

3.2.2

Implementation

................................
................................
................................
..............................

5

3.2.3

Support
................................
................................
................................
................................
............

7

3.2.4

Engagement with QTI User Community

................................
................................
..........................

8

3.3

R
EVIEW OF
O
BJECTIVES

................................
................................
................................
................................
.

9

3.3.1

Project Management

................................
................................
................................
......................

9

3.3.
2

Assist with Development of QTIv2.1 Main Profile

................................
................................
.........

10

3.3.3

Provide Online Support to QTIv2.1 Users

................................
................................
......................

11

3.3.4

Remaining Tasks

................................
................................
................................
...........................

11

3.4

W
HAT DID YOU LEARN
?

................................
................................
................................
..............................

12

3.4.1

Evaluation

................................
................................
................................
................................
.....

12

3.4.2

User Testing

................................
................................
......................

Error! Bookmark not defined.

3.4.3

Project Experiences

................................
................................
................................
.......................

13

3.
5

I
MMEDIATE
I
MPACT

................................
................................
................................
................................
...

13

3.6

F
UTURE
I
MPACT

................................
................................
................................
................................
........

14

4

CONCLUSIONS

................................
................................
................................
................................
........

14

5

RECOMMENDATI
ONS

................................
................................
................................
.............................

14

5.1

M
AINTENANCE OF THE
IMS

QTI

SPECIFICATION AND PR
OFILE

................................
................................
............

14

5.2

M
AINTENANCE OF
T
OOLS

................................
................................
................................
............................

15

5.3

C
OMMUNITY
B
UILDING

................................
................................
................................
...............................

15

5.3.1

Extending Interest in QTI

................................
................................
................................
...............

15

5.3.2

Training in QTI

................................
................................
................................
...............................

16

5.4

O
B
JECTIVES

................................
................................
................................
................................
..............

16

6

IMPLICATIONS FOR THE

FUTURE

................................
................................
................................
............

17

7

REFERENCES

................................
................................
................................
................................
...........

17

8

APPENDICES

................................
................................
................................
................................
...........

17

8.1

A
PPENDIX
1:

S
USTAINABILITY FOR
QTI
V
2.1

U
SAGE IN
UK

HE

................................
................................
............

17

8.1.1

Community Building

................................
................................
................................
......................

17

8.1.2

Expanding the Community of Specialists

................................
................................
......................

19

8.1.3

Community

................................
................................
................................
................................
....

20

8.1.4

Summary

................................
................................
................................
................................
.......

20


Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
3

of
20

1


Acknowledgements

This project belongs to the JISC E
-
Learning programme and has been funded by JISC. We are
grateful for the support we have had from JISC
-
CETIS and from our partner institutions, the University
of Edinburgh and
Kingston University.

2

Project Summary

1.

The Question and Test Interoperability specification (QTIv2.1)

http://www.imsglobal.org/question/

will shortly become a final specification
.

To enable this to happen, at least two profiles must be
specified, implemented and documented. This project has produced and implemented a profile for
the UK
HE
community and demonstrated interoperability in three different renderers. We have
also made th
ose renderers available as part of the provisions of a support site for users of
QTIv2.1, available on the Amazon Elastic Compute Cloud (EC2)

at
http://cloud.niallbarr.me.uk/



2.

Interoperability with the first,
simple Common Cartridge QTIv2.1 profile had already
been publicly
demonstrated with twelve different tools

during the
IMS quarterly held in Koblenz, Germany,
August 2010, and it is intended for inclusion in a future version of Common Cartridge. We have
specified, implemented and demonstrated a UK
-
HE

Profile
, which covers the functionality that is
used by a wide set of sta
keholders within the UK HE community, and incidentally across a much
wider international community. In the process, we have created and documented a demonstration
set of items and tests and used these to demonstrate that there is interoperability between
r
endering and authoring tools.

3.

The project team has, with the aid of JISC
-
CETIS, run two QTI Roadshows

http://www.jisc.ac.uk/events/2011/06/jisccetisroadshownorth.aspx


to prom
ote QTIv2.1 and
encourage colleagues to adopt the standard. The tools and support facilities were demonstrated
and participants were able to gain some hands
-
on experience of the tools. Feedback from these
events was generally encouraging,
and although

atte
ndance was lower than we had hoped
,
several delegates were enthusiastic about the possibilities for using QTI at their institutions
.
Follow
-
up contact has been made with some of these potential users and we hope to bring some
of them into the Assessment an
d Feedback projects as informal partners.

4.

We have created and deployed a QTI User Support Site

at
http://cloud.niallbarr.me.uk/

, to
facilitate the involvement of other communities in QTI development,
implementation and use, by
ensuring that expert advice and practical help is at hand. Users can test their own questions using
the renderers on the site, or try out examples from the collection of questions provided in the site.
We have also included on th
e support site links to the project blog

at
http://qtisupport.blogspot.com/

, several question and test editors, conversion tools
, the CETIS
QTI FAQs
htt
p://wiki.cetis.ac.uk/Assessment_FAQ

and the mailing lists and forums for QTI. This
site is also available as a Virtual Machine (VM), downloadable from the original site.
We have
assurance from CETIS that the Support Site will be hosted on the JISC CETIS Am
azon Cloud,
although the URL is not yet available.

3

Main Body of Report

3.1

Project Outputs and Outcomes

Output / Outcome Type

(e.g. report, publication,
software, knowledge
built)

Brief Description and URLs (where applicable)

Report

Project plan

(
on
JISC web
site)

Publications

Two Roadshows, one at the University of Strathclyde and the other as a
pre
-
conference meeting at CAA2011, were held in June and July

Publications

Conference papers:

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
4

of
20

CAA2011, Workshop: “Sharing and Repurposing in Standards
J
扡s敤
Ass敳sm敮琠


Tools for Converting Resources to Use QTIv2.1”

caaconference.co.uk/wp
-
content/uploads/Milne
-
CAA2011.pdf



E
-
Assessment Scotland 2011, Poster: “QTI
J
IPS”

A䱔
J
C 2011, Workshop: “
p瑡td慲摳
J
扡s敤 Ass敳sm敮琺⁃牥t瑩n朠
䥮fov慴iv攬ef湴nr敳ti湧Ⱐ䥮ter
潰敲慢l攠e敳潵rc敳⁕ ing⁑q䥶㈮2


altc2011.alt.ac.uk/talks/22296

Variety in Chemistry Education, short paper, “
p瑡湤慲ds
J
扡s敤
Ass敳sm敮琺⁃牥慴i湧 䥮fov慴iv攬ef湴nr敳瑩湧Ⱐ䥮f敲潰er慢l攠剥o潵rc敳
啳i湧⁑q䥶㈮1
”. York, September 2011

睷眮wsc⹯牧⽩m慧敳⽖䥃b_㈰ㄱ彰ro杲慭m敟瑣mㄸ
J
1㠶㠹㐮Q摦


Physics Higher Education Conference, short paper, “
p瑡t摡r摳
J
扡se搠
Ass敳sm敮琺⁃牥慴i湧 䥮fov慴iv攬ef湴nr敳瑩湧Ⱐ䥮f敲潰er慢l攠剥o潵rc敳
啳i
湧⁑q䥶㈮1
”. Lancaster, September 2011

桴h瀺pL睷眮桥慣a摥my.慣⹵.⽡Ls整e⽰L⽤Lc畭敮瑳⽥L敮瑳⼲LㄱL灨散_㈰
ㄱ⽤牡L瑟tro杲慭m攮e摦

䵯r攠er潭⁓䴠j湤⁎B

剥灯rt

ci湡l⁲数潲琠o瑨ts 摯c畭敮琩

p潦瑷慲攠

㌠牥湤敲敲s⁣慰慢l攠潦⁲敮摥rin朠gh攠兔䥶㈮2 啋eb mr潦il攺

䵡瑨Ass敳sb湧i湥ⰠIAssessⰠIq潍m⁉䤠
e慣栠i湣潲灯o慴an朠
愠aoll散瑩o渠nf
數慭灬攠e略sti潮s

p潦瑷慲t

p灥c瑡瑵t⁰牯vi摥s⁴ s琠敤i瑩湧⁦潲⁴桥onq䥶㈮2⁕ eb mr潦ile

p潦瑷慲攠

m慣k慧i湧 慰plica瑩o渠n潲⁴敳瑳…ⁱ 敳瑩o湳


摥laye搠by l慴攠ec桥ma

睩ll⁢攠e潭灬整
敤 畮d敲⁕ei煵r慴e

p潦瑷慲攠

nq䥣慴a灲潦ili湧⁴潯l⁦潲⁩瑥ms…⁴ s瑳

p潦瑷慲t

䍯Cv敲eio渠n慣ili瑩敳⁦潲

䵯潤l攠euizz敳
慲攠avail慢l攠睩瑨i渠pp散瑡t畳

p潦瑷慲攠

䍯Cv敲eio渠n慣ili瑩敳⁦潲⁂BLte扃qⁱ izz敳
㨠瑨is

will 扥com攠evail慢l攠
摵rin朠gh攠e桡湧e潶敲e
瑯tnqf
J
䑉a

p潦瑷慲t

䍯Cv敲eio渠n慣ili瑩敳⁦潲⁑q䥶ㄮ㈠2漠兔䥶2⸱.⁑q䤠fi杲g瑥tc慮 扥
摯wnl潡摥d⁦rom
桴瑰㨯⽣潤攮杯潧o攮e潭⽰⽱瑩Li杲慴楯港

p潦瑷慲t

A⁑q䥶㈮2⁉瑥t⁂慮k⁰ 灵l慴a搠睩t栠hx慭灬攠e略s瑩潮s⁦潲⁥oc桡湧in朠
慮搠dh慲楮朠gu敳瑩o湳⸠.桥⁢ 湫⁩si
湫敤⁦rom⁴ 攠eu灰潲琠pit攮

p潦瑷慲t

sir瑵慬 䵡chi湥⁩湣潲灯r慴i湧⁲敮摥r敲e⁡ 搠d異灯r琠ti瑥t


摯wnlo慤
from
桴瑰㨯⽣l潵搮di慬lb慲r⹭攮畫

p潦瑷慲攠

剥湤敲敲s⁤ ploye搠dn⁁mazo渠nl慳tic⁃潭灵瑥tCl潵搠⡅䌲) 慴a
桴瑰㨯⽣l潵搮diallb慲r⹭攮ek


p潦瑷慲t

p異灯r琠tit攠e渠n䌲C


瑨t
im慧攠es

慶ail慢l攠e潲⁊䥓䌠C漠oe灬oy 潮⁴桥ir
䍬潵搠d灡c攮

m畢lic慴ion

啋eb⁐r潦il攠


lis琠tf⁑q䤠fl敭敮瑳⁡ 搠慴arib畴us⁣潮t慩n敤 i渠n桥⁕heb

灲潦ile

m畢lic慴io渠

q敳琠t整e潦⁩瑥ts…⁴ s瑳⁦潲⁕䭈o⁐r潦il攠et
桴h瀺p⽣l潵搮di慬l扡rr⹭攮ek

m畢lic慴io渠

剥o敲敮e攠ex慭灬敳⁦潲⁲敮摥ri湧

慮搠敤i瑩湧
瑨敳攠數慭灬敳⁳敲e攠
扯瑨t灵rp潳敳)



lv敲e㌰M c潬l散
t敤

a湤 潮⁴桥⁳u灰or琠ti瑥t慴
桴瑰㨯⽣l潵搮diallb慲r⹭攮ek

⸠䍯.l散ti潮 睩ll
co湴n湵e⁢ yon搠dh攠e湤
瑨t⁰牯 散t


m畢lic慴io渠

䑯wnl潡摡bl攠瑲慩nin朠gli摥s…⁤ c畭敮瑳
slid敳⁦rom⁇唠灲潪散琠tn搠
r潡摳桯ws

慴a
桴h瀺p⽣l潵搮di慬l扡rr⹭攮ek

Ⱐ䍅q䥓⁑q䤠
q
r慩湩湧
duid攠
桴h瀺pL睩ki⹣整es⹡..畫⽑q䥟fr慩湩湧彇畩de
)

m畢lic慴io渠

A湮潴ot敤ⁱ略sti潮 C⁴敳琠摯wnl潡d

m畢lic慴io渠

䑯wnl潡摡bl攠im灬敭敮瑡ti潮⁳u灰潲琠摯c畭敮瑳
桴h瀺pL慱畲慴攮ui湧s瑯渮nc⹵kLfil敳⽴Lch
J
摯d
J
fi湡l⹰.f


m畢lic慴io渠

cAn⁰ 来⁦潲⁡畴桯ri湧⽣o湶敲ei潮


䍅q䥓 睩ki

慴a
桴h瀺pL睩ki⹣整es⹡..畫⽁ss敳sm敮瑟䙁n

ⰠI灤a瑥t by⁃bq䥓⁡ d⁑qf
J
䥐p
瑥慭

p潦瑷慲t

䍡灡扩lity⁴漠o瑯牥⁘j䰠i潲⁳異灯r琠t潮v敲e慴ao湳


瑨攠eAss敳s⁘䵌
q
敳琠ti瑥

s敲e敳⁴ is⁰ rpose

h湯睬e摧e⁴牡湳f敲

bx灥ri敮c攠e慩湥d

潦⁵ i湧⁁m慺潮⁃l潵搠d湤⁰慣k慧in朠gi瑥t⁦潲o
Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
5

of
20

deployment on the EC2

Knowledge transfer

Support on demand via forums and email


瑥tm m敭扥rs⁨ v攠
慮sw敲敤⁴ e⁳m慬l m扥r ⁳異灯r琠牥煵敳瑳 which⁡ p敡r敤⁤ ri湧
瑨t⁰牯 散琬t慮搠willⰠIf⁣潵rs攬e
c潮瑩湵e⁴ 摯⁳漮


3.2

How did you go about achieving your outputs / outcomes?

3.2.1

Profiling

5.

The team have collaborated with JISC
-
CETIS and other members of the QTI Working Group to
bring the QTIv2.1 specification to its final form. It is expected that the final

version will be released
by the end of 2011

or the first quarter of 2012
. We have been instrumental

clarifying some of the
descriptions of elements and attributes in the specification and

in specifying several additional
elements:



mathOperator: provides a
ccess to mathematical functions, e.g. trig functions, logs and
absolute value,



statsOperator: provides access to basic statistical functions, e.g. mean, variance, etc.,



mathConstant: enables authors to include the mathematical constants
e

and
π
into
proces
sing calculations,



lcm: calculate the lowest common multiple of a set of numbers either in a multiple or ordered
variable or as children of the operator,



gcd: calculate the greatest common divisor of a set of numbers either in a multiple or ordered
variabl
e or as children of the operator.

6.

We have also worked with JISC
-
CETIS and the IMS QTI Working Group to specify a Profile for
QTIv2.1 which covers the elements and attributes implemented by systems used in the UK HE
community (UKHE Profile). Team members
ha
ve prepared a

list of elements and attributes to be
included in the UKHE Profile.

7.

QTIcat (
http://www.niallbarr.me.uk/QTIcat/

) is a tool written by Niall Barr that allows collections of
QTI XML files to b
e used to build profile descriptions. It

has been

used to create the list of
elements and attributes for the UKHE profile and

is currently being used by the IMS QTI working
group as part of their profiling process
. The tool

has been enhanced to improve the

level of detail
in its output and to conform to the latest QTIv2.1 schema. It will be made available for cataloguing
the elements used in items and tests.
In this form it is useful for finding items with, for example,
particular interaction types.

8.

The ori
ginal QTItalk website was developed for CETIS by Niall Barr, and used by developers of
QTI 1.2 players including Graham Smith and Dick Bacon to improve the quality of documentation
available to developers of QTI 1.2 systems, and define clearer interoperabi
lity guidelines. A new
version of QTItalk for QTI 2.1 making use of Wiki technology is currently being used to build more
user
-
friendly documentation. The project team have begun work on this documentation, and the
site is open for editing by the entire QT
I community.

3.2.2

Implementation

3.2.2.1

Rendering

9.

The MathAssessEngine, JAssess and SToMPII renderers are included in the project’s Virtual
Machine (VM) deployed on the Amazon Elastic Compute Cloud (EC2) at
http://cloud.niallbarr.me.uk

. Users can use MathAssessEngine and SToMP II to display and
respond to example questions provided within the site, and to render items they themselves
upload. JAssess, as mounted on the VM, holds a collection of some 300 work
ing XML examples
Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
6

of
20

and Content Packages which have been tested on MathAssessEngine, SToMP and JAssess.
Many of the simpler examples are accompanied by notes, and this annotation will be extended.

10.

Both the MathAssessEngine and the JAssess renderers for QTIv2.
1 have

been updated to ensure
that all the UKHE Profile elements render correctly.

11.

The JAssess rendering/responding system is available to download as a desktop application and
is also available as an XML test site to which XML can be uploaded, producing

(with XML that
conforms to the specification) rendered versions of items and tests.

12.

The SToMPII assessment system is included in the VM deployed on EC2 as a demonstration
system with examples which can be rendered or downloaded and for which notes on que
stion
features are provided.

3.2.2.2

Authoring and Conversion

13.

Spectatus, which was developed during the FETLAR project to create tests using questions either
in XML form or as IMS content packages, has been refactored to conform to the new schema
.

14.

Steve Lay’s
open source
QTI Migration tool converts QTIv1.2 to QTIv2.1 in either single file or
batch mode for an entire folder.

15.

Facilities for conversion of
many
questions from Blackboard and WebCT
are

available
using Dick
Bacon’s

open source

converter
,

developed for

the Physical Sciences Question Bank
,

combined
with the QTI Migration tool
.

16.

Spectatus contains facilities for direct conversion of Moodle quizzes to QTIv2.1.
The QTI export
facility for Moodle is being upgraded to QTIv2.1 for use in Moodle 2.0 and later.

17.

Error reporting
facilities in Spectatus have been extended to provide
a more comprehensive
message when problems occur.

18.

An individual question can be previewed from a test in Spectatus
.

19.

Additional metadata fields have been provided, including the use of th
e JACS subject taxonomy
alongside another, optional tax
onomy
, such as the MathCentre taxonomy developed under
FETLAR. The capability to set default metadata has been added to make packaging more
efficient and to improve the likelihood of users setting meta
data on their resources.

20.

Example metadata has been produced in consultation with JISC
-
CETIS, with annotations
indicating what data is mandatory, which fields can be duplicated, and what is considered
appropriate for the UK HE and FE community in the contex
t of assessment.

3.2.2.3

Items and Tests

21.

A suite of items and tests has been assembled which can be used as a test set for checking the
implementations. These include the Common Cartridge set produced for the CC profile, together
with items and tests utilising the

additional elements and attributes identified for the UKHE Profile.
A QTIv2.1 Item Bank (a derivative of the HEA Physical Sciences Question Bank which is the
responsibility of one of the consultants) has been developed and is populated with example
questi
ons. This will be maintained and developed to provide a vehicle for the exchange and
sharing of questions.
The questions and tests themselves are annotated, and standard responses
and feedback are being documented to enable comparison of results.

A link is

provided to the
bank from the Support Site.

3.2.2.4

Testing

22.

Questions and tests have been validated against each of the supposedly final QTIv2.1 schemas
produced during the project, and then used with the tools to ensure that interoperability can be
demonstrated.

They have also been made available to the IMS Profiling Team for interoperability
testing.

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
7

of
20

23.

The
tools have been tested against the profile to ensure that their operation is as expected for all
the UKHE Profile elements and attributes.

24.

The
three renderers
have been used with the test suite of items and tests, to ensure that the
software behaves as expected, both in isolation and as deployed on EC2. Where items are
randomised, multiple renderings were carried out.

3.2.3

Support

3.2.3.1

Website

25.

A website containing the so
ftware tools has been set up, constructed as a virtual machine to
facilitate deployment on the Cloud.

26.

Software and content has been collected in the site deployed on EC2 at
http://cloud.niallbarr.me.uk
; it will

be packaged and transferred to JISC’s Cloud at a time to be
agreed with JISC.

3.2.3.2

Liaison with JISC
-
CETIS

27.

The team is working closely with JISC
-
CETIS personnel involved in the profiling activities; there
have been regular online and face to face meetings
throughout the project.

28.

The support site and other support provision was planned in consultation with JISC
-
CETIS.

29.


Staff at JISC
-
CETIS have assisted in assembling advice on the metadata aspect of packaging.

30.

JISC
-
CETIS have assisted the project team by id
entifying documentation suitable for use from the
support site, being are comprehensible to readers who are technically literate, but unfamiliar with
IMS QTI and IMS Content Packaging. JISC
-
CETIS have also ensured that the documents
collected and generated

by the project are also suitable for their intended audience.

3.2.3.3

QTI Support Website

31.

The QTI Support website provides a number of facilities to assist users in creating QTI questions
and tests, whether from scratch or by converting existing resources. Thes
e include:



The rendering facilities set up for the main profile, in two modes:

a.

As test facilities for authors to check that their resources behave as expected. The
uploaded XML and rendered HTML can be stored so that difficulties can be discussed
with memb
ers of the team. The renderers include the functionality of a production
implementation so that users are able to see a “student’s” view of the rendered
questions, including feedback.

b.


As a 'read
-
only' demonstration site to illustrate correct use of QTI X
ML, providing over
3
00 working examples of the (X)HTML produced.



The documentation wiki, QTITalk, describing QTI elements and attributes, including example
code,



The conversion tools mentioned above,



The Spectatus test construction tool, a Webstart applic
ation which downloads and installs
locally,



Downloadable example questions and tests, with documentation and sample input and
output,



Downloadable XML snippets, annotated and documented, including response processing
templates,



Downloadable training materi
als (see below), including slides and examples for use in
workshops,

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
8

of
20



FAQ collections covering the widely used elements and attributes and broader aspects such
as response processing,

3.2.3.4

Other Support Activities

32.

Training materials have been constructed, coveri
ng conversion and authoring of questions and
tests using the tools available on the site. These have been and are currently being used for:



Dissemination of the project outputs at conferences and meetings,



Training colleagues at the partner institutions an
d elsewhere,



Running two pilot “QTI Roadshows” at the University of Strathclyde in June 2011 and the
Computer Aided Assessment (CAA) conference in July 2011 under the auspices of JISC
-
CETIS.

33.

Requests for assistance with authoring submitted through the IMS
Question & Test
Interoperability mailing list
ims
-
qti@lists.ucles.org.uk

have been answered by team members.
Team members are available to answer such queries on a one
-
to
-
one basis if required, and are
redi
recting the more routine enquiries to the support website if appropriate.

3.2.3.5

Support for Implementation

34.

The developers of the renderers and authoring applications are available to provide support for
colleagues who are also developing software implementing QTI, via a forum such as the IMS QTI
Public Forum at
http://www.imsglobal.org/community/forum/categories.cfm?catid=52

or the IMS
Question & Test Interoperability list
ims
-
qti@lists.ucles.org.uk
. The small number of enquiries tha
t
have arisen during the project have been answered by team members.

3.2.4

Engagement with QTI User Community

35.

Team members at the University of Glasgow and at Kingston University are currently engaged in
assisting colleagues to deliver QTIv2.1 questions and test
s in Maths support, teacher education
and Engineering
;

t
his activity has
been
extended to Harper Adams University College (HAUC)
during the life of the project
.

C
olleagues at HAUC and at the University of Strathclyde are now
partners

in two new QTI
-
based projects funded by JISC: QTI Delivery Integration (QTIDI) and
Uniqurate
, led by the University of Glasgow and Kingston University respectively
.

36.

The team includes representatives of two companies who have commercial interest in using
QTIv2.1, Bildungsportal Sachsen GmbH and QuestionMark, who have provided assistance in the
following ways:



BPS have contributed questions and tests for the Test Set and the annotated examples,



BPS and QuestionMark have provided feedback on the profiling a
nd documentation activities
of the project
.

37.

Gary Wills and Lester Gilbert, representatives of the University of Southampton, which is well
-
known for QTIv2.1 development, have continued to support the project team, particularly by
providing the contacts and

organisation for the pre
-
CAA Roadshow.

38.

Although

the QTIv2.1 specification has not yet become final, the expected adoption of the
standard by commercial organisations has
begun with several publishing companies and
commercial tool vendors all over the wor
ld taking up QTI 2.1
, see
http://wiki.cetis.ac.uk/Assessment_tools,_projects_and_resources

.

We anticipate that the
QTIv2.1 specification will become final at the IMS Quarterl
y Meeting in February 2012.

T
he
team
will liaise with the
publishing community
in their

use

of

QTIv2.1, to provide support as needed.



Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
9

of
20

39.

IMS has set up a QTI/APIP alliance on the IMS website, that will be staffed and have
implementation activities after the

release of the specification. The team will

engage with the
QTI/APIP alliance as required.

40.

Team members have taken part in the dissemination activities listed in the table below. The
intention throughout the project has been to enthuse potential adopters
in HE and FE. We have
demonstrated the tools and discussed users’ ideas for adopting QTI in their institutions, and can
report that several
potential
new user institutions have been identified.

These will be contacted by
QTI
-
DI and Uniqurate with a view to

bringing them into the projects as informal partners.


Timing

Dissemination
Activity

Audience

Purpose

Key Message






23/06/2011

CETIS QTI
Day
(Strathclyde)

New users in
HE/FE

Introduced QTI to
colleagues who are
unaware of it, announced
support site
launch.

QTIv2.1 is ready for
new users

42/07/2011

CETIS QTI
Day (CAA,
Southampton)

New users in
HE/FE

Introduced QTI to
colleagues who are
unaware of it, announced
support site launch.

QTIv2.1 is ready for
new users

July 2011

CAA workshop

E
-
Assessment
co
mmunity

Demonstrated
interoperability and
conversion, introduced
support site

E
-
assessment users
can share resources

Sept 2011

VICE
presentation

Chemistry
community

Introduce QTI
-
IPS and
support site

Users are aware of the
project and its outputs.

Sept
2011

ALT
-
C
workshop

Education
technology
community

Demonstrated
interoperability, authoring,
conversion, tests in VLEs,
support site

Users can convert
basic quizzes into more
powerful QTI
assessments using the
project

s⁴ ols⁡ 搠
s異灯r琠t慣iliti敳

p数t′
Mㄱ

m䡅䌠
灲敳敮瑡tion

mhysics
comm畮ity

䥮fr潤畣e⁑qf
J
䥐p⁡ 搠
s異灯r琠tite

啳敲e⁡牥⁡w慲a ⁴ 攠
灲潪散琠t湤 i瑳畴u畴u.

3.3

Review of Objectives

3.3.1

Project Management

Objective

Due Date

Measure

Comments

Ensure that the project is managed
efficiently and that

all the project
outputs are delivered within the agreed
deadlines and budget

23/09/2011

All project outputs are
delivered within the
agreed deadlines and
budget.

Some outputs
have been
delayed
, but are
in preparation



see below

Ensure project progress
is visible via blog

23/09/2011

Project blog up to date

Several blog
entries have
been made

Ensure that information about the
project reaches potential users via at
23/09/2011

Presentations at 3
meetings or
See list of
conference
Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
10

of
20

least 3 meetings and conferences open
to the HE/FE community

conferences (see
D
issemination
P
lan)

papers in table
at 3.1 above

3.3.2

Assist with
D
evelopment of QTIv2.1 Main Profile

Objective

Due Date

Measure

Comments

Assist the QTIv2.1
Working
Group

to complete a profile of
QTIv2.1 which covers the
elements used by the more
advanced implementations This
is known as the Main Profile (MP)

03
/05/2011

List of MP elements
and attributes is
complete.

QTI W
orking
G
roup

moved away from Main
Profile;
QTI
-
IPS
returned
to the concept of a UKHE
profile and produced a
set of example items
which
,

used with QTIcat
,

produce a draft of this
profile

Collect and validate examples
which demonstrate the Main
Profile of QTIv2.1

24
/05/2011

The collection of
examples covers the
MP and
all examples
validate against the
schema

The examples cover the

draft
UKHE profile which
replaced the Main Profile
as an objective during
July2011

Ensure that bugs found in tools
since FETLAR ended are fixed

23/04/2011

Tools operate as
expected with
examples

Tools have been updated
to operate correctly with
the profile elements

Ensure that all tools
(MathAssessEngine (MAE),
JAssess (JA), ST
o
MP
II
, Spectatus,
QTICat
) operate appropriately
with all Main Profile elements
and attributes

31
/05/2011

Render example
questions using all MP
elements and
attributes in MAE, JA,
SToMPII
. Edit tests in
Spectatus using all MP
elements and
attributes for tests.

MAE and JA render whole
draft
UKHE profile;
SToMPII renders almost
all; Spectatus
was
seriously

delayed by

schema problems but
nonetheless
handles test
requirements.

Create a website containing tools
and examples for profiling, and
package as a Virtual Ma
chine
(VM) .

24
/05/2011

Websi
te VM is
available for
download.


Support site is up and
contains tools and
examples

Deploy a website containing
tools and examples for profiling
in EC2.

31
/05/2011

Website VM is
deployed and live on
EC2.

Support site is
deployed
on EC2


Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
11

of
20

3.3.3

Provide Online Support to QTIv2.1 Users

Objective

Due Date

Measure

Comments

Create, and publish online, tools
and support materials for
QTIv2.1 authors

23/09/2011

Tools, examples and
workshop materials
for authors available
from support
website

A large collection of tools
and resources is available
on the site

Collaborate

with
CETIS
on
briefing papers

23/09/2011

Briefing papers
on
support facilities and
QTI+Maths capability
available for download

from support site.

Expected by end
November 2011;CETIS to
promote these
.

Create training materials for QTI
trainers to present in live
workshops

23/09/2011

Training materials
available for download

Workshop

and
conference

slides
are
loaded
.

Create, if appropriate, and
publish online,
tools and support
materials for converting
resources to QTIv2.1

23/09/2011

Tools for conversion
available from support
website

QTIv1.2to QTIv2.1
conversion tool is
available

and handles
conversion of many
questions exported as
QTIv1.2 from Blackboard

and W
ebCT
; Moodle

conversion tool is being
updated
.

Create, and publish online,
support materials for
implementing QTIv2.1

23/09/2011

Support materials for
implementation
available from support
website

A link to the SourceForge
FETLAR download area is
provided
.

Deploy all support resources as a
VM

31/08/2011

VM version of support
website is ready for
use

This will be completed
during the transition to
QTI
-
DI

Deploy all support resources as a
site in EC2

23/09/2011

Support website is
available for use

Site

has been available
since July 2011.

3.3.4

Remaining Tasks

41.

These tasks, although delayed beyond the duration of the project, have been completed in the
intervening period; the delays were

caused by
:

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
12

of
20



The non
-
availability of a final Schema
; IMS have produced a
series of schemas, but we are
still awaiting corrections to the latest version (6.1) which appeared in mid August. A version of
the 6.1 schema edited by the QTI
-
IPS team is c
urrently the most stable schema available.



L
ack of developers


time
,

result
ing

partly from
the lack of a schema,

since

re
-
checking has
been necessary as different versions of the schema have emerged, and to some extent
from
additional

work to include other maths operators not envisaged in the original proposal
.


3.3.4.1

The drafting of a pr
ofile

42.

O
n the basis of the collection of examples we have so far, the publication of this profile
, with the

QTIcat list
and the definitive set of examples, and the testing of the renderers for compliance is
complete
.

3.3.4.2

The inclusion in the renderers and in S
pectatus of an agreed definitive
schema

43.

It is not envisaged that this will be a major task as we have all kept up with the various versions as
they have appeared
;

the QTI
-
IPS

version is

currently

the most reliable
.

3.3.4.3

The final mounting of the renderers on th
e cloud web site.

44.

The new version of MathAssessEngine needs to be incorporated in the site when the refactoring
of the test rendering is completed.

45.

Advantage has been taken of the delay in mounting JAssess to improve the XML Test site with a
version which

may avoid the need for a login.

46.

The new

version of SToMPII accepts uploaded XML.

3.3.4.4

Promotion of QTIv2.1 and QTI
-
IPS outputs

47.

There would be a good opportunity for promoting QTIv2.1 the work of this project when the
briefing papers are in place on the Suppo
rt Site. If CETIS could mention the facilities available in a
blog and/or newsletter, it would help to raise awareness of our work and could also be used as
advance notice of QTI
-
related events early in 2012.

3.4

What did you learn?

3.4.1

Evaluation

48.

,
.

The draft UK
HE profile of QTI has been completed only at the end of the project, and the
support site has not been fully operational to enable us to investigate the user experience
, so it
has

not been possible to perform an evaluation of the project within the timesca
le available
. It
would also be useful to evaluate the impact of QTIv2.1 on the sector, but this should wait at least
until the Assessment and Feedback projects, QTI
-
DI and Uniqurate, are in their final stages.
Thereafter,

in perhaps 3 years’ time,

JISC
-
CET
IS
might wish to

review the impact of
both
the
specification

itself

and the resources available from the support site and elsewhere
.


3.4.2

Deployment and Sharing on the Amazon Cloud

49.

The project made use of the Amazon EC2 cloud as a platform for running a demonstration server
that could then be reused by other users following the end of the QTI
-
IPS project. The major
advantage of the Amazon cloud is ease of deployment


while similar f
unctionality exists with
other Virtual Machines, Amazon provides a particularly easy to use and easy to share solution.
The QTI
-
IPS VM is able to run as a free
-
tier server for the first year of a user’s use of the Amazon
cloud, so the initial cost of deplo
yment can be very low. An image of the Machine
(
Amazon public
Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
13

of
20

machine image identifier

ami
-
45a09d31
) is available for other Amazon users to deploy on the EU
West (Ireland) section of
the Amazon cloud.

3.4.3

User Testing

50.

A simple form of t
he
s
upport
s
ite
became

available
in

July and
this has been developed so that it
now
constitutes the most comprehensive collection of resources for QTI users to date
. We have
used the opportunities provided b
y dissemination at conferences to trial the site with users.

51.

The feedback has been positive, particularly with regard to the many examples now available.
Many users are happier about editing an existing example than starting to author a question from
scra
tch
; the support site contains a collection of example questions which can be used for this
purpose
.

3.4.4

Project Experiences

52.

Delays in receiving a final version of the QTIv2.1 schema have resulted in



Multiple attempts at validating the collected and newly
-
aut
hored examples,



In the course of collecting XML examples from a number of authors, unexpected nuances in
their interpretation of the IMS QTI specifications have called for modifications in the renderers
JAssess, MathAssessEngine and SToMP II
.

The use of u
nreliable schemas has compounded
this as it has often been necessary to spend some time determining whether an apparent
failure of a renderer has not in fact been caused by the schema. This has delayed the
deployment of the renderers to EC2
,



Delays in crea
ting a finalised QTIv2.1 version of the QTIcat tool for profiling; the current
version uses an amended version of the schema edited by hand,



Serious delays in updating the Spectatus test authoring tool because its Java classes are
generated from the schema

using JAXB and the new schema is unsuitable for this approach;
this was discovered in July and the missing classes have since been inserted by hand.

53.

The schema is still not finalised at the time of writing, but is expected in the next few weeks.

54.

The lack

of easily
-
accessible editors for the more sophisticated QTI question types appears to be
affecting uptake of the standard.

There are simple editors available which will suffice to excite
interest among the more technically minded academics who may be expe
cted to be the targets of
the initial phases of the dissemination proposed under section 5 below
. However,

t
he newly
-
funded Uniqurate project aims to provide an editor
which will handle more interaction types and
flexible, targeted feedback capabilities, w
hich

will
motivate

the wider adoption of QTIv2.1.

3.5

Immediate Impact

55.

The following factors are regarded as being indicators of the success of the project:



The UKHE
Draft
Profile for QTIv2.1 is specified and tested,

This contributes to the

finalisation
proces
s for QTIv2.1



Online support facilities for authoring QTIv2.1, converting resources to QTIv2.1 and
implementing QTIv2.1 are deployed and hosted
on the cloud; CETIS are ready to provide a
sustainable location for the site when the content from this project
is in place
,



The user community for QTIv2.1
in the UK has

increased in size and in the expertise of its
members
, although QTI usage is not as widespread as we had hoped
.



The
support site provides access to

standard rendering implementations and a
comprehen
sive set of example item and test XML, together with a site at which authors’ XML
can be tested and the results stored for comment form valuable resources for beginners.



Those who have participated in the project have seen their own expertise extended
,

ad
d
ing

to
the growing pool of expertise in QTI. This, together with the resources described in the
previous paragraphs, provides a foundation which is now available upon which to construct
more formal training schemes.

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
14

of
20

3.6

Future Impact

56.

The project is likely to
have impact on the following groups:



E
-
learning practitioners have a one
-
stop centre from which they can obtain tools, examples
and support, and to which they can direct colleagues who adopt QTI,



Academic and teaching support staff, as advantage starts to

be taken of the advanced
facilities available to produce assessments for learning. . . . .

57.

Over the course of the recently
-
started QTI
-
D
I

and Un
i
qurate projects, we expect to see
more
interest in QTI from the community.
There is already some evidence of
potential uptake by new
users.

4

Conclusions

58.

The major objective of the QTI
-
IPS project has been to secure the future of QTI. As the report
shows, several resources have been created to this end, and access to others has been
facilitated via the Support Webs
ite. The time
-
consuming nature of the work and the need to
coordinate with the activities of other projects has, inevitably, resulted in these resources reaching
a useable stage only at the end of the project. We believe that support for the activities des
cribed
in section 5 below, while much less expensive than the project itself, is important to getting proper
value from what has been done
. In the appendix below (Paragraph 8.1 et. seq.) we suggest some
means by which these activities might be supported
.

59.

T
he QTI
-
IPS Support Site
is

a resource which provides users of QTIv2.1 with a first stop for
locating tools and examples and for communicating with other users and, in particular, with the
experienced practitioners
. It is particularly relevant to the client

partners of the newly
-
started QTI
-
DI and Uniqurate projects, who need examples and guidance as they trial the editing and delivery
tools being produced and introduce colleagues to QTI.

60.

The results of a number of JISC projects are available either directly

from the Support Site or via
links on the front page, and it is anticipated that the products of QTI
-
DI and Uniqurate will also be
made available in this way. This enables users to see these products “under one roof” and select
the artefacts they need.
Th
ere is a need to secure the future of this collection in a centrally
-
hosted
site, preferably in a form which permits new resources to be submitted as they become available.


5

Recommendations

61.

We have included an appendix (Paragraph 8.1 et seq.) which suggest
s how the recommendations
we now describe might be brought about
.

5.1

Maintenance of the IMS QTI specification and profile

62.

The accumulated expertise of the project team should be utilised in completing the work of
finalising the QTIv2.1 specification and conti
nuing to an expected QTIv2.2
. The desirability of
further extensions of areas of the
specification
has already become apparent and more are to
appear as the community extends
;
experienced practitioners are needed for this work.

63.

In fact, we need a framewor
k in which to maintain not just the specification and profile, but the
body of expertise which has grown up around it. The implication of this is that those who are
currently regarded as experienced practitioners should share their expertise in order to gr
ow a
new generation of experts.

64.

The content collection covers the draft UKHE Profile, but this will need to be extended alongside
the user community.

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
15

of
20

5.2

Maintenance of Tools

65.

There
will

be a need for modifications to the renderers and the authoring tools
in th
e future
.
Experienced developers will perform these modifications more efficiently than new staff who are
unfamiliar with the tools. However, in order to improve sustainability, we recommend that each
tool should have more than one developer who is suffici
ently familiar with the code to be able to
continue development.

In this way the developer community will expand alongside the user
community and ensure that tools and examples continue to develop in response to user needs.

66.

We recognise that with a complex

and comprehensive specification such as QTI, unexpected
nuances in the interpretation of these specifications are likely to become evident as XML is
collected from a widening pool of authors. In particular, while there are now large numbers of
assessment
Items, there are few examples of assessmentTests. During the project, we have
noted
a few
aspects of the test specification which have not yet been implemented, and seem to
us to require some clarification which only an illustrative implementation can achi
eve.
These
features may well prove valuable in some

assessment designs, but have yet to be tried out.

T
he
expertise of the developers in the project

will be needed to bring this about
.

67.

The source code, and in some cases binary downloads, for the tools are
available on
SourceForge in the FETLAR project. Developer webinars and live codebashes would help to pass
on the expertise and take the code forward
; we anticipate that CETIS will be involved in the
organisation of events of this type
.

5.3

Community Building

68.

The construction of profiles of QTI, and of tools for editing and delivering QTI resources is
immaterial unless there are users to adopt them. We put forward the following recommendations:

5.3.1

Extending Interest in QTI

69.

One of the most effective, and earliest,
means by which the message about QTI was spread was
through a series of road shows organised several years ago (in the days of QTI Version1) by
CETIS. With the availability of the greatly improved QTIv2.1, and in particular the tools and
support resources
created in the present project, those who might attend such road shows can be
given greatly increased confidence in the future
.

W
e
consider

that
,

ideally,
a further series of
roadshows

should be mounted.
However, in the current economic climate, it is like
ly to be

seen as

more cost
-
effective to provide online introductory resources.
Nonetheless, we consider that viva
voce demonstrations and informal contacts at meetings are the most effective means of recruiting
interest, and should be encouraged as far as
is possible. We return to this problem in the
appendix.

70.

It has been said that supported community building should await the availability of
comprehensive, friendly editor applications. We would contend that, since the first version of the
Uniqurate editor
will be available in early 2012, that planning for this effort can be
gin in
autumn/winter 2011, and that a “QTI for Beginners” workshop should take place in spring 2012.

71.

Most of the work in developing QTI over the last few projects has concentrated on math
ematics,
which has a very wide appeal since it is fundamental to many subject areas, such as science,
medicine, social sciences and financial disciplines. There are, however, other subjects such as
languages, chemistry and biochemistry which form the found
ation of many other disciplines, in the
same way as does mathematics, which would also benefit from the application of QTI in handling
large classes
.

Staff in these disciplines bemoan the lack of facilities for, for example, chemical
diagrams using ChemML
in chemistry, the ability to check code in computer science, or audio
input and musical notation in music.
It is very desirable to engage the attention of
learning
technologists and
academics across the sector to engender partnerships like those which have

been so successful in mathematics.

The flexibility which enables QTI to provide targeted and
Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
16

of
20

timely feedback in formative assessment and to be used even for subjects which are difficult to
assess properly using normal VLE quiz engines or proprietary soft
ware will also make it of
interest to decision makers across the sector, and dissemination to this group is particularly
important.

72.

The new projects QTI
-
DI and Uniqurate are working with their client partners’ academic
champions to achieve uptake in a vari
ety of disciplines within their institutions
, and to ensure that
the technical aspects of the project become embedded in their VLEs. The academic champions
will also demonstrate the potential of QTI to management in order to achieve wider adoption
.


5.3.2

Traini
ng in QTI

73.

There is a need to establish

and promote

training in
QTI to encourage and facilitate contact
between beginners and experts. This should include a number of facets:



These activities can be carried out online:



Maintenance and (possibly consolidatio
n) of the existing forums, including retaining
the availability of the present experts, to ensure that authors’ questions are promptly
answered,



Maintaining the website which has been developed during the project, including
maintaining and updating the doc
umentation and the collection of examples, in
particular those which are presented as working examples; monitoring and providing
advice to users of the XML test site
,



Online master classes in which experienced practitioners demonstrate techniques
and answe
r questions from users
.



There is a
lso a need for
face to face

events, perhaps in conjunction with conferences and
other meetings where breakout sessions can be arranged
:



Training sessions for different levels of users or subject areas,



C
odebashes and semin
ars through which authors may test their XML and new
implementers can compare the interoperability of their products
.

74.

We anticipate that CETIS will wish to take a leading role in arranging, running and promoting
training in QTI.

5.4

Objectives

75.

What is needed
to sustain these activities can be summarised into t
hree

major objectives:

a.

To support the continued growth of a community
within the HE and FE

sector
who will use

and promote

QTI
-
compliant tools and examples,
with the aid of

the

support tools provided

by
t
he present project,

b.

To assist the

educational and
technical specialists
within the institutions in advocating,
maintain
ing

and develop
ing

the facilities

needed by the academics,


c.

To find means of maintaining and developing the

output of this project
, in
particular

the
support website. The utility of these support

facilities depends upon the availability of a
growing team of specialists, familiar with the QTI specification and the tools themselves, who
can provide advice to users and continue development o
f the specification.


76.

Dissemination is essential to
all

of these objectives; we anticipate that development of the tools
and the bank of examples will be driven by the needs of teachers and students, and hence the
cultivation of the
technical and
academic
community through dissemination is the top priority. We
anticipate that some tutors and some technical staff will have the enthusiasm to become part of
the specialist team.

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
17

of
20

6

Implications for the future

77.

Two new projects have been funded recently by JISC, whi
ch will enable partner institutions to
adopt the tools with support from experienced developers and users:



QTI Delivery Integration (QTIDI) will use IMS Learning Tools Interoperability (LTI) to create a
more robust connection to MathAssessEngine. This will

also be more generally applicable,
and will thus enable users to access MathAssessEngine from a variety of VLEs, including
Blackboard, Moodle and Sakai.



Uniqurate will modify the Mathqurate question editor to provide a user
-
friendly authoring
environment

with a wider subject base. The more sophisticated interactions will be supported
as well as the basic multi
-
choice and text
-
box types. This is regarded as a major step towards
making QTIv2.1 accessible to less technically
-
inclined staff and is expected to

improve
adoption of QTI across the sector.

78.

Both of these projects will utilise the QTI
-
IPS Support Website as a means of providing online
support for novice users as well as giving links to the latest versions of the tools. New users will
be able to downl
oad questions from the support site and modify them using Uniqurate before
constructing tests using Spectatus and loading them into their institutional VLE using the QTIDI
connector.

79.

There is increasing interest in the resources made available by this proj
ect, but the impetus must
be maintained in order to make the use of QTIv2.1 a natural choice for online assessment.


7

References

IMS Learning Tools Interoperability,
http://www.
imsglobal.org/lti/blti/bltiv1p0/ltiBLTIimgv1p0.html


IMS Question and Test Interoperability Version 2.1,
http://www.imsglobal.org/question/



8

Appendices

8.1

Appendix 1: Sustainability for QTIv2.1 Usage in UK
HE

80.

In section 5 above, we have set out a number of activities which we believe are necessary to
ensure that full future value will be obtained from the expenditure of funds and effort in the
present project.
In 5.4 we
distilled these recommendations to th
ree objectives, which can be
summarised as nurturing growth in the academic, technical and specialist communities.

81.

To achieve these objectives we believe that some further funding is necessary. We believe
however that
,

given the foundations laid in the present project, much can be achieved through the
expenditure of relatively small amounts. Each of the suggestions below reflects one or
more

of
these objectives.

8.1.1

Community Building

82.

We put forward the following suggestion
s in relation to community building:

8.1.1.1

Extending Interest in QTI

8.1.1.1.1

Conferences and Seminars

Online and Face to Face

83.

It is suggested that CETIS could provide a framework through which introductory webinars and/or
downloadable slideshows might be delivered, wit
h availability of these resources advertised in the
newsletters and via social media.

Valuable as these provisions are
,

once the interest of potential
participants has been secured, they may not
,

however,
be effective in first engaging this interest.
Past experience has shown that new interest has only been generated in viva voce talks at which
QTI has been described as part of conferences and other live meetings, where the

networking


which accompanies su
ch events is often equally valuable in recruiting interest. Such events have
,

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
18

of
20

in the past
, often

been run by CETIS
,
but present financial constraints make it difficult to fund
events specifically devoted to QTI.
It will be more practical to run pre
-

or p
ost
-
conference
workshops to tie up with presentations given during the main conference; this happened at the
CAA conference last year and appears to be a workable arrangement.
We plan to attend the
CETIS conference in February 2012, which coincides with th
e IMS Quarterly meeting and will be
a splendid opportunity for promoting QTI;
the

codebash session will be
particularly
valuable for
implementers and
may also engage learning technologists new to the specification.


84.

In the course of the QTI
-
DI and Uniqurat
e projects, we
shall

continue dissemination of the
QTIv2.1 specification, the support site created under this project and the tools produced by the
new projects
, particularly by presenting this work at conferences relating to assessment and our
various sub
ject areas
. This is a vital opportunity to enthuse colleagues across UK HE and FE
about the benefits of adopting QTI

and to identify institutions where staff are keen to begin using
the resources and tools in their teaching.

Additional dissemination via th
e JISC Assessment and
Feedback Programme
, which contains these projects, will also be important in improving uptake
across the sector.


85.

A further,

relatively inexpensive means of dissemination that could be used alongside

the ongoing
project activities,
CE
TIS

webinars and other provisions
, would be the provision of some minor
funding to allow our existing experts to attend conferences and other meetings which are already
being held or planned. This would also be an effective way of encouraging the continue
d
participation of these experts in our community. F
or example, f
unding for travel is available

in
2011
-
12

through HEA
http://www.heacademy.ac.uk/travel
-
fund
,

for attendance at specified events
relatin
g to a number of themes, including assessment, education for sustainable development,
flexible learning, retention and success, and the activity should “have a positive impact in the
sector”. However, this appears to be intended for attendees at these meet
ings rather than
presenters, and is available to academic and technical staff of HE institutions. Unfortunately,
several of the consultants are self employed

and hence are not eligible for this funding.

8.1.1.1.2

Starter projects

86.

Following on from the identification

of interested academics, learning technologists and technical
staff,
we need to find the means to introduce QTI into their institutions.

87.

We suggest that we should seek funding, perhaps from JISC or HEA, to run Mini
P
rojects, each
costing a relatively sma
ll amount, in conjunction with individual institutions of Higher

and Further

Education. These might be at the Department or School level or
through

the institution’s Teaching
and Learning Support centre, or equivalent.

88.

These projects would be used to creat
e resources within the department or institution, perhaps in
the form of questions within a particular course, with the side or even main objective of creating
expertise in QTI. New materials would be required to be shared as OER, and new materials
public
ised on CETIS Assessment wiki/newsletter.

89.

For example, a project to produce new questions for a course would require:



The creation of an agreed number of questions,



Assistance with deploying the tools required,



Training for a group of staff, who might be a
cademics or learning technologists, in creating
and editing questions and tests and setting tests in the institutional VLE.

90.

A register of available consultants could be maintained and publicised by CETIS through the
usual channels.


91.

The cost of these proj
ects would be of the order of £3000
-
£5000, depending on the type of
resources required and the need for expenses for travel to training sessions. This is sometimes
available from institutions’ Learning and Teaching budgets, but the provision of small grant
s for
this purpose would significantly increase the spread of QTI.


Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
19

of
20

92.

The HEA is funding some Teaching Development Grants. The deadline for the departmental
grants was too close to allow us to investigate the possibility of bidding for this. However, there
w
ill be a
further
call
, this time

for Collaborative Grants, and we shall prepare a bid for this funding.

Since the project outputs, particularly the assessment resources, will be OER, it would

also

be
appropriate to seek funding within the OER programme.

8.1.1.2

Training in QTI

93.

To establish a 'QTI Academy' we suggest

o
nline activities, hosted and maintained by CETIS with
input from practitioners:



Maintenance and (possibly consolidation) of the existing forums,



The hosting by JISC
-
CETIS of the QTI
-
IPS Support Site
developed during this project,
including provision for maintaining and updating the documentation and the collection of
examples, in particular those which are presented as working examples,



Online master classes in which experienced practitioners demonstr
ate techniques and
answer questions from users.



We are encouraged to hear of the
forth
coming 'codebash' hosted by CETIS, and hope that
these will become a regular feature.

8.1.2

Expanding the Community of Specialists

94.

To maintain the specification and profile we
need not only to retain the body of expertise which
has grown up around it but also to provide means whereby these exper
ienced practitioner
s may
share their expertise in order to grow a new generation of experts.

We expect that
some of the
academics, learn
ing technologists and developers who have adopted QTI through the
dissemination efforts outlined above will
be sufficiently enthusiastic about the ideas of
interoperability and resource sharing to become involved in the development of the specification
and

the tools. However, this will not happen overnight, and the current team of experts will need
to continue their efforts.


8.1.2.1

Maintenance

and Development

of the IMS QTI
S
pecification and
P
rofile
s

95.

T
he
mem
bers of the current project team

have been contributing
to the development of the IMS
QTI specifications for many years, during their involvement with projects and also in their own
time
.

We are endeavouring to communicate our enthusiasm for QTI to colleagues across the
sector, and there are several new users w
ho are likely to join in the
detailed

process

of suggesting
and testing additions and extensions to the specification
, particularly as the needs of the various
disciplines become apparent.
This is more likely to happen during projects which involve
experie
nced practitioners who can guide their colleagues in this work, and we consider that the
starter

projects suggested in 8.
1.1
.1.2 above would be helpful in identifying and training
prospective “new experts”.

8.1.2.2

Maintenance

and Development

of Tools

96.

The involvem
ent of team members with the specification and profiling process has led to the
development of tools, initially to check the specification, but now increasingly as necessary
equipment for authoring and delivery in institutions.
The tools are open source an
d carry licences
which encourage development
. There is evidence that the source code is already being
downloaded by other developers, and we hope that their efforts will also be made available as
open source
.

Project Identifier:

Version:0.8

Contact:sue@elandweb.co.uk

Date: 29/10/2011


Document title: JISC Final Report Template

Last updated : Feb 2011


v11.0

Page
20

of
20

97.

Some of the tools will be maintained for the
next year or so under the current QTI
-
DI and
Uniqurate projects
, whose intention is to establish the tools in the client partners’ institutions, thus
increasing their sustainability
.
In these projects, the developers will aim to improve their familiarity
w
ith each other’s code, and also involve the technical staff at the client institutions.


98.

To
improve sustainability of

the other tools, we suggest that
,

if
a small retainer
wer
e made
available
,

the developers concerned would
be able to maintain the tools an
d
also provide training
to “apprentices” recruited through the activities suggested in the paragraphs
above.

99.

Thus

each

tool
would have

a
t least one

additional developer who can

maintain the tool and
continue development
. In addition
,
we should see the
kernel of a new

developer community which
can

contribute to the pool of expertise for the future.

8.1.3

Community

100.

As the community of users grows, it should be provided with a “home base” online. This could
be part of the Support Site, and would provide communic
ation facilities and news of
developments, new resources, activities, etc. Ideally, it would issue newsletters to participants and
either have its own discussion forums or use the current QTI mailing lists for this purpose.

8.1.4

Summary

101.

It is apparent from the
above discussion that dissemination is crucial to the future
sustainability of QTI. Only by enthusing academics, learning technologists and developers will we
be able to increase the use of QTI resources in institutions and find the new blood needed to
con
tinue developing and extending the specification and the tools. Dissemination efforts will be
greatly assisted by



The new projects Uniqurate and QTI
-
DI, which include dissemination as part of their remits,



Provision of online and face to face meeting facil
ities, specifically aimed at assessment and/or
at QTI itself,



Travel grants enabling non
-
affiliated experts to attend and present at meetings.

102.

Would
-
be new adopters can be assisted in their uptake of QTI by means of starter projects;
these may be funded th
rough a variety of means



Existing calls, for example th
at

from HEA mentioned above,



Future OER funding rounds,



Institutions’ internal learning and teaching programmes,



Subject
-
based community funding, in particular where shared assessment resources with
ta
rgeted feedback would increase the value of formative assessment,



Direct funding designed for these projects.

103.

From among the new users we
expect

to
recruit enthusiasts who will assist in the
dissemination, development and support work which is
at present

undertaken by the current team
.

U
ntil this happens,
we need to retain the experienced practitioners by involvement in projects
where they can propagate their expertise:



QTI
-
DI and Uniqurate,



The starter projects mentioned above,



“Apprenticeship” funding as in paragraph
96

above.

104.

With the support outlined here, the sustainability of QTI usage in UK HE and FE will be
improved and a community of users established for the future.