Item Authoring and Review

fullfattruckΚινητά – Ασύρματες Τεχνολογίες

10 Δεκ 2013 (πριν από 3 χρόνια και 7 μήνες)

90 εμφανίσεις



Item Authoring and Review

Business Requirements



Request for Proposal

Provision of an Online Item Authoring and Review Solution
-

Annexure A

Contents


Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

i


Glossary

................................
................................
................................
................................
.......................

ii

1

Introduction
................................
................................
................................
................................
........

2

1.1

Project background

................................
................................
................................
.................

2

1.2

Bu
siness objectives

................................
................................
................................
................

2

1.3

Document purpose

................................
................................
................................
..................

3

1.4

Assumptions

................................
................................
................................
............................

3

1.5

Constraint
s

................................
................................
................................
..............................

3

2

Business processes in scope

................................
................................
................................
..........

5

3

Business processes not in scope

................................
................................
................................
...

7

4

System overview

................................
................................
................................
...............................

9

4.1

High
-
level overview

................................
................................
................................
.................

9

5

Business requirements

................................
................................
................................
...................

11

5.1

Overview

................................
................................
................................
...............................

11

5.2

Prioritisation

................................
................................
................................
..........................

11

5.3

Actors and roles

................................
................................
................................
....................

11

5.4

Pre
-
conditions

................................
................................
................................
.......................

13

5.5

Post
-
conditions
................................
................................
................................
......................

13

5.6

Item authoring

................................
................................
................................
.......................

14

5.7

Item review, approval and version control

................................
................................
............

17

5.8

Item bank

................................
................................
................................
..............................

18

5.9

Interopera
bility
................................
................................
................................
.......................

18

5.10

Reporting

................................
................................
................................
...............................

19

5.11

Identity and access management

................................
................................
.........................

19

5.12

User interface and accessibility

................................
................................
............................

21

6

Non
-
functional requirements

................................
................................
................................
.........

23

6.1

Accessibility

................................
................................
................................
...........................

23

6.2

Usability

................................
................................
................................
................................
.

23

6.3

System performance

................................
................................
................................
.............

24

6.4

System availability
................................
................................
................................
.................

24

6.5

Maintainability

................................
................................
................................
.......................

24

6.6

Data storage

................................
................................
................................
..........................

24

6.7

Scalabil
ity

................................
................................
................................
..............................

25

6.8

Volumetrics

................................
................................
................................
...........................

25

6.9

Deployment and hosting

................................
................................
................................
.......

26

6.10

Security and privacy

................................
................................
................................
..............

26

7

Appendices

................................
................................
................................
................................
......

28

7.1

Item structure

................................
................................
................................
........................

28

7.2

Process flow

................................
................................
................................
..........................

29



Glossary


Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

ii


Term

Definition

ACARA

Australian Curriculum, Assessment and Reporting
Authority

Adaptive assessment

Provides the ability for an assessment to adjust based on a student
'
s
responses. This can be done on an item
-
by
-
item basis (linear) or on a testlet
basis (staged)

AI

Artificial
i
ntelligence



m
arking that occurs by a
computer and is typically
used in essays or extended responses

APIP

Accessible Portable Item Protocol (v 1.0)


t
he standard
that
allows digital
t
ests and
i
tems to be ported across APIP compliant test item banks. It
provides a test delivery interface with

all the information and resources
required to make a
t
est and an
i
tem accessible for students with a variety of

disabilities and special needs

Archive

R
eferenced in this document as a system function as well as an operational
activity. The system function is the ability for a user to flag an item, content or
other records as archived
(ie

no longer active
)
. The operational activity is the
moving of data th
at is no longer actively used to a separate data storage
device

Authentication

Verification by the system that you are who you say you are

Authorisation

Defines what the specific user can do within the system

Constructed response

A question type that re
quires the student to compose or enter a response
rather than selecting a response from a list of options. This includes all
constructed responses such as short, long, dichotomous, polytomous, open
and closed
:



s
hort



a
one or two word or digit response



long


a sentence or short paragraph



open


an unlimited number of characters permitted



closed


a limited number of characters permitted with the ability to
specify the limit



polytomous


see
Polytomous

item



dichotomous


see
Dichotomous

item
.

Credentials

Physical or digital evidence of identity used in authentication.

Commonly
username and password

DEEWR

Department of Education, Employment and Workplace Relations

Diagnostic assessment

Given at set increments throughout a period of time to det
ermine the progress
of a student
'
s learning and identification of areas of weakness and strengths.
These assessments also provide insight into a student
'
s misconceptions of
concepts. The results from these assessments can then be used in
instructional plan
ning

Dichotomous item

An item whose responses are
scored as correct or incorrect
1

Distributed scoring

Scoring that occurs by sending responses to be marked by persons at set
locations. Marks are recorded and then used for scoring




1


http://www.smarterbalanced.org/wordpress/wp
-
c
ontent/uploads/2012/05/TaskItemSpecifications/TechnologyEnhancedItems/TechnologyEnhancedItemGuidelines.pdf

Glossary


Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

iii


Term

Definition

ESA

Education Services
Australia

Formative assessment

Used with students, usually informally, in order to modify teaching and
learning activities to increase understanding of concepts. The information is
typically qualitative and used by students and teachers for greater learni
ng

ICT

Information and
c
ommunications
t
echnology

IDAM

Identity and
a
ccess
m
anagement

-

t
he business processes and technology
required to manage the life

cycle of identities and their access to resources

Identifier

A set of data items that uniquely iden
tify an identity

Identity

The digital representation of a real
-
world entity (eg student, teacher, school,
class), encapsulated by a set of information about that entity

IMS Global Learning
Consortium (IMS GLC)

A

global, non
-
profit, member organi
s
ation
that strives to enable the growth
and impact of learning technology in the education and corporate learning
sectors worldwide

Item

A test question that contains one or more interactions with response data
collected as a result of the interaction. An item
should contain its own, unique
prompt. It could have its own stimulus, and/or share a stimulus with other
items
1

Item bank

A repository of items and all associated content which can be accessed and
used in tests. More than one

item bank can be used in an assessment
system; however,
only
one should be the authoritative source

Jurisdiction

For the purposes of this document,
jurisdiction

may refer to either the
state/territory or the school sector (government, independent or Catholic)

L
earning resources

O
nline text, videos, audio, interactive activities and other materials that
teachers
use to support student learning

Longitudinal

For the purposes of this document,
longitudinal

refers to tracking the same
type of information on the same subjects at multiple points in time. It is used
for measuring change. These data and information can be focused on
cohorts, assessments or any other

data desired to be tracked to make data
-
informed decisions

Master data
management

Involves determining the processes, governance, standards, policies and
data of an organisation and/or applications

NAP
-
CC

National Assessment Program Civics and Citizensh
ip

NASOP

National Assessment and Surveys Online Program

NOAC

National Online Assessment Capability

NSOSS

National School Opinion Survey System

Glossary


Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

iv


Term

Definition

Paradata

A

type of metadata which refers to the usage data of an item or its associated
content

(
eg how many
times a piece of content was accesse
d)

Performance task

An assessment activity that requires students to construct a response, create
a product, or perform a demonstration
2

PNP

Personal
n
eeds and
p
references



a

standard that provides the detailed
studen
t needs and preferences for specific assessments. These needs and
preferences might include magnification, contrast, foreground/background
colour, audio text and Braille
2

Polytomous item

An item whose responses are scored on a

scale
; for example
, 0

4 or 0

2
1

Prompt

The instruction to the student on how to respond to a question

Psychometrician

An individual who studies and practices the measurement of the knowledge,
skills and abilities of others.
In assessment, this individual ensures that items
and assessments are created according to the constructs and frameworks
desired to be measured as well as reviewing data to ensure that items and
assessments are performing as expected

QTI

Question and
T
est

I
nteroperability (v 2.1)


a

specification standard that
defines a standard format for the representation of item content and results
2

Role

The grouping of a set of access rights to resources/services that can be
assigned to
a user

Rubric

An assessment tool used to evaluate the item response based on a
predefined set of criteria

ScOT

Schools Online Thesaurus


provides a controlled vocabulary of terms used
in Australian and New Zealand schools. It encompasses all subject areas as
well as terms describing educational and administrative processes

SCSEEC

Standing Council on School Education and Early Childhood

S
IF

School Interoperability Framework


an interoperability standard enabling
the

exchange
of
school information between
schools

and
between schools and

external parties

SIS

Student
i
nformation system

SSO

Single sign
-
on



u
ser only needs to sign on once t
o access a range of
services/applications that they would have otherwise had to sign on to
individually

Stem

Beginning part of the item that presents the question asked of the
respondent, or an incomplete statement to be completed

Stimulus

Any
supplementary text, graphic, video and audio, or combination thereof
associated with an item to assist a student when re
sponding to an item or
item set




2

https://parccsharepoint.org/Public_Access/Assessment%20Administration%20RFP%20Q%20and%20A%20Documents/4_High
-
Level%20Application%20Architecture_DIS
TRIBUTE
-
Nov%202012.pdf

Glossary


Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

v


Term

Definition

Summative assessment

Measures a student
's

understand
ing

at a given point such as the end of a
course or

year. These tests are scored and used for more formal purposes
such as accountability or marks

TAA

Test Administration Authority

Technology enabled

Standard selected response or constructed response items that use digital
media as the stimulus (audio, v
ideo, or interactive widget) but do not require
specialised interactions for response and/or accompanying response data
1

TEI

Technology
-
enhanced item



computer
-
delivered items which include
interactions/responses other than t
he standard selected response or
constructed response. These may include embedded media, graphical tools,
drag and drop or text highlight

Test bank

A repository of created tests that can be delivered to students

Test item

An item that has been approved
and added to a test or testlet

Testlet

A smaller portion of the overall assessment. Numerous testlets make up one
assessment. Each testlet measures different constructs. In an adaptive
assessment, different testlets are provided based on previous response
s to
items

User

A person
who

interacts with the system










Introduction

1



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

2


1

Introduction

1.1

Project b
ackgro
und

The Online Diagnostic Tool
s

(ODT) initiative is a 2010 Australian Government election
commitment.

The
Department of Education, Employment and Workplace Relations

(
DEEWR
)

has funded the
National Assessment and Surveys Online Program (NASOP), under the
ODT

initiative, which is
designed to deliver national assessments and surveys online
.
The program comprises three
elements:



development
, testing and delivery of a National Sc
hool Opinion Survey
S
ystem

(NSOSS)
for deployment in 2013 at jurisdictional discretion (Element 1)



development, testing and delivery online of the 2013 National Assessment Program Civics
and Citizenship sample assessment (NAP
-
CC) (Element 2)



development an
d testing of a National Online Assessment Capability (NOAC) for diagnostic,
formative and summative testing, supporting the Australian Curriculum, Assessment and
Reporting Authority (ACARA) National Assessment Program, including National
Assessment Program

Literacy and Numeracy (NAPLAN) if agreed by the Standing Council
on School Education and Early Childhood (SCSEEC) (Element 3)
.

The program is being delivered in partnership by ACARA and Education Services Australia
(ESA), with each responsible for deliver
ing interrelated aspects. ACARA is responsible for
delivering
a
ssessment and
r
eporting
s
ervices, including research into the online delivery of
assessment programs. This research will provide Ministers with evidence
-
based information to
facilitate decision
s regarding the future of national assessment programs.

ESA is responsible for
the delivery of the technical services.

One of the components of Element 3 is an online item authoring and review
system
. This
system

will allow
content specialists to create, e
dit and track items through a specified workf
low process

for
item

creation and trialling.

1.2

Business objectives

The high
-
level business objectives for the
i
tem
a
uthoring and
r
eview system are:



to provid
e

an online solution with the capacity to create, r
eview
, modify and approve items

for diagnostic, formative and summative testing



to support the higher volume and types of items required as part of NASOP



to satisfy compliance obligations by adhering to technical interoperability standards



to improve the user
experience by providing a more streamlined workflow and review
process for items



to enable users to share approved quality content available on the internet



to provide the ability to align items to pre
-
existing curriculum standards
.





Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

3


1.3

Document purpose

The
purpose of this document is to articulate the business requirements of the
i
tem
a
uthoring
and
r
eview system.

In addition, t
his document will provide input to the following activities:



deve
lopment of system and user test cases



development and/or configurati
on of the selected software package
.

1.4

Assumptions

The business requirements outline the future intent of the business processes for
i
tem
a
uthoring
and
r
eview and do not necessarily represent the current processes. Some of these processes
may change after po
licy decisions are made.



Item specifications will be predefined by ACARA
.



The alignment of content to curriculum standards can be accomplished where curriculum
standards already exist
.



All items and associated content created outside the
i
tem
a
uthoring and

r
eview system will
have the same level of functionality as other items and associated content created within
the system.



The system will support formative, summative

and

diagnostic assessment
s.



Item
a
uthoring roles can be assigned to any user type such as

a teacher or third
-
party staff.



Identity and
a
ccess
m
anagement (IDAM) functions will initially be implemented using the
native registration and administration of the selected product
,

however
,

the product must be
capable of leveraging an external
IDAM

service
.



ESA will work with
the
vendor to identify all the user roles for the item authoring system.
User roles will be described in detail in the functional specifications.



The v
endor will be able to comply with the business continuity plan when such a p
lan is
defined for the item authoring
and review
system.



The v
endor will be able to comply with the disaster recovery plan when such a plan is
defined for the item authoring
and review
system.



The v
endor will be able to comply with the failover and redunda
ncy plan when such a plan
is defined for the item authoring
and review
system.



In the initial phase of the project,
an
'
i
tem
b
ank
'



where all the items are stored and
retrieved



will be implemented as an internal
data store
of the solution.

1.5

Constraints



T
he solution must be capable of leveraging an external
IDAM

service.



Policy decisions related to minimum technical requirements may impact item authoring.











Business processes in scope

2



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia
Limited

5


2

Business processes in scope

Item
a
uthors will create items used in NOAC. These items need to support the item stimuli
creation, assets, marking information (including rubrics) and prompts.

The
i
tem
a
uthoring
and review
system will enable the entire authoring process including:



creation of the item by the item author



review from a content and technical perspective by
revie
wer,
psychometricians, curriculum
experts and the item author



approval of items by
reviewer,
psychometricians, curriculum expert
s

and test managers.

All items will then be made available to the test authoring system. All items will be assigned an
ID for tr
acking and management purposes.

The following additional processes are within the scope of the
i
tem
a
uthoring and
r
eview system:



provision of a stand
-
alone online item authoring system compliant with interoperability
standards for communicating with all re
quired NOAC components



a reporting mechanism to enable administrators to report on all item
-
related data.

User access management will require the following minimum functions:



a
bility to create, update, suspend, delete user accounts



a
bility to validate a
user when accessing the system (authentication)



a
bility to manage access to th
e data and functions using roles



a
bility to define, assign and maintain roles for a user
.











Business processes not in
scope

3



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

7


3

Business processes not in scope

The following business processes are not in
the
scope of the item authoring
and review
project:



p
rovision of a test authoring

solution



p
rovision of a test
delivery solution



provision of a learning repository and the ability to create, edit and update learning
resources



c
apability to create, edit, review and approve curriculum standards



integ
ration with school, jurisdiction and national systems

including
i
ntellectual
p
roperty
management systems



f
inancial management of items or associated contr
actor costs



delivery, report
ing and analysis of assessments



provis
ion of an external IDAM service



desi
gn and creation of
all
media and graphical assets
associated with an item
.











System overview

4



4

System overview

The NOAC reference model
shown
in diagram
1

below
represents

a high
-
level view
of
the
proposed
NOAC

componentised structure

and the functions that it will support
. It does not

represent the detailed
data

that wil
l be necessary for functional requirements

and may be
subject to change pending the agreed system architecture
.

4.1

High
-
level overview

The highlighted components
within the
'
Assessment Creation
and

Management
'

section
indicate
the scope of this requirements d
ocument.


Diagram
1
: NOAC
r
eference
m
odel










Business requirements

5



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

11


5

Business requirements

5.1

Overview

Th
e following requirements have been identified
to support the creation of items and all
elements of
th
e

business process including editing, review and approval.

5.2

Prioritisation

The r
equirements that will deliver the greatest business value or present the highest degree of
technical complexity or dependence are
prioritised
. High priority requirements are elaborated
and developed as early as possible in the development process.

These re
quirements have been
prioritised

using the following approach:

Priority

Description

C


Critical

A

requirement that must be satisfied in the final solution for the solution to be
considered a success

I


Important

A

high
-
priority item that should be included in the solution if possible. This is often a
critical requirement but one which can be satisfied in other ways if strictly necessary

D


Desirable

A

requirement that is considered desirable but not necessary. Th
is will be included
if time and resources permit

O


Out of scope

A

requirement that stakeholders have agreed will not be implemented in a given
release, but may be considered for the future

5.3

A
ctors

and r
oles

The following
actors

and
roles

have been
identifie
d for
interact
ing

with the system
3
.



i
tem author



r
eviewer



p
sychometrician



s
ystem
a
dministrator



c
urriculum expert



t
est manager
.






3

The details of each actor

and
role will be
further clarified

in the functional specification
(
s
)
.



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

12


5.3.1

Item author

Description

A
n authorised user who specialises in creating test questions and content
according to
established specifications

Responsibilities

Designs and creates items
;

specifies associated content and metadata

Success criteria

Is able to
:



authenticate themselves to the system



create and modify items according to specification



track item history and
version



assign items and associated item content in the workflow process

5.3.2

Reviewer

Description

A
n authorised user who reviews items and associated content according to
established specifications

Responsibilities

Reviews and provides comments on assigned
items and associated item content

Success criteria

Is able to
:



authenticate themselves to the system



review all assigned items according to specification



add and modify review comments



view all review comments and version history



assign items and associat
ed item content in the workflow process



flag items during the review process

5.3.3

Psychometrician

Description

A
n expert

at
analysing

educational data and measurements from student
responses

Responsibilities



Reviews and provides comments on assigned items and
associated item
content



Analyses and updates
established

psychometric data associated with items

Success criteria

Is able to
:



authenticate themselves to the system



review all assigned items according to specification



add and modify review comments



view
all review comments and version history



add or modify psychometric data associated with items





Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

13


5.3.4

System a
dministrator

Description

A
n authorised user
who

is able to administer user access and settings

of this
system

Responsibilities

Create
s

and updates user access and authorisation

Success criteria

Is able to
:



authenticate themselves to the system



create, update and delete other system users



manage user roles and access

5.3.5

Curriculum expert

Description

A

subject matter expert for a particular learning area within the curriculum

Responsibilities

Reviews item content relevant to the curriculum and specified learning area

Success criteria

Is able to
:



authenticate themselves to the system



review and comment

on item content



assign items and associated item content in the workflow process

5.3.6

Test manager

Description

A
n
authorised

user
who

is able to review items and associated item content
according to established specifications

Responsibilities

Reviews and
provides comments on assigned items and associated item content

Success criteria

Is able to
:



authenticate themselves to the system



review and comment on item content



assign items and associated item content in the workflow process

5.4

Pre
-
conditions



All actors possess
authorised

access to the system.



Assessment frameworks have been created.

5.5

Post
-
conditions



National summative assessment items are created
,

l
oaded (if necessary) and ready for
delivery for trial.



Other summative assessment items are creat
ed and ready for delivery.



Diagnostic assessment items are created and ready for delivery.



Formative assessment items are created and ready for delivery.



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

14


5.6

Item authoring

Requirement ID

Requirement description

Priority

BRS
reference

Item
c
reation

IU
-
001

The system shall provide
a web
-
based
item authoring
s
olution
.

C

IA
-
034

(NOAC)

IU
-
002

The system shall provide the ability for the

creation of
the following item types
:



constructed response



long
and

short



polytomous
and

dichotomous



order phrase/sequence



extended response



technology
-
enhanced (TEI)



r
efer to IU
-
003



technology
-
enabled (embedded media)



r
efer to
IU
-
004.

C

IA
-
004 (NOAC

IU
-
003

The system shall provide the ability to create a number
of
TEIs

and capabilities including:



association



create a
line (graphic tool)



move one or more objects to given set of locations
(drag
-
and
-
drop)



produce a geometric shape



select blocks of text (hot text)



mark a visual area (hot spot)
.

I

New

IU
-
004

The system shall provide the ability to create
t
echnology
-
enabled

item
s

which
can

contain:



video clips

(in agreed formats)



audio recordings (in agreed formats)
.

I

New

IU
-
005

The system shall provide the ability
to create a
performance task item type
.

D

IA
-
004 (NOAC)

IU
-
006

The system shall provide the ability for item

authors to
create, update and delete item templates for the
creation of items in accordance with set specifications.

C

IA
-
030

(NOAC)

IU
-
007

The system shall provide the ability for item authors to
create, update, and delete

item
s
.

C

IA
-
031

(NOAC)

IU
-
008

The system shall
automatically generate

a unique
identifier
for each
item.

C

IA
-
032

(NOAC)

IU
-
009

The system shall provide the ability to capture
at least

three additional unique identifiers to each item.

C

New

IU
-
010

The system shall allow authorised
users to create,
update and delete an individual item and all associated
content such as stimuli or prompts, graphics, rubrics,
marking information.

C

IA
-
005

(NOAC)

IU
-
01
1

The system shall allow the authori
s
ed users to create
multiple items or item sets v
ia a defined bulk process.

C

IA
-
006 (NOAC)



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

15


Requirement ID

Requirement description

Priority

BRS
reference

IU
-
012

The system shall allow the authori
s
ed users to update
and delete multiple items or item sets via a defined bulk
process.

C

IA
-
006 (NOAC)

IU
-
01
3

The system shall have the ability to link the items to
a
learning resource
.

C

New

IU
-
01
4

The system shall provide the ability to preview an item
with associated content as it will appear to the student
in the test player.

C

IA
-
013

(NOAC)

IU
-
015

The system shall provide the ability to attach a file to an
item o
r an item set in agreed formats
such as
,

but not
limited to
,

png
,

docx
,

xlsx
,

ai
,

indd

and

HTML

for review
purposes only.

C

New

Item m
odification

IU
-
01
6

The system shall provide the ability

for
authorised
users

to save
an item at any stage during the
item creation
process
.

C

IA
-
008

(NOAC)

IU
-
01
7

The system shall
allow
authorised
users
to edit an
individual item
and all associated content such as
stimuli or prompts, graphics, rubrics and marking
information
at

any stage of the
item creation and review
process.

C

IA
-
009

(NOAC)

IU
-
01
8

The system shall allow authorised users to edit multiple
items or item sets and all associated content such as
stimuli or prompts, graphics, rubrics and marking
information at any stage of the item creation and review
proce
ss.

C

IA
-
010

(NOAC)

IU
-
01
9

The system shall provide the ability to move, replace
and resize images belonging to individual items, stimuli
or prompts and item sets.

C

IA
-
027

(NOAC)

IU
-
0
20

The system shall provide the ability for authorised users
to
archive one or more items from the system.

C

IA
-
025

(NOAC)

IU
-
0
2
1

The system shall provide the ability for authorised users
to un
-
archive one or more items
that had been archived
in

the system.

C

New

IU
-
02
2

The system shall provide the ability to make a
copy of
one or more

item
s including all associated content, and
assign a unique identifier to the new item(s).

C

IA
-
039
(NOAC)

IU
-
02
3

The system shall provide the ability for an authorised
user to share
one or more

item
s and all associated item
content

with other authorised system users
.

C

IA
-
040

(NOAC)

IU
-
02
4

The system shall
have the ability to notify users of
different events
.

C

New

Item m
etadata



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

16


Requirement ID

Requirement description

Priority

BRS
reference

IU
-
02
5

The system shall provide the ability to specify and
update metadata information belonging to
individual
items and all associated content such as
,

but not limited
to:



unique identifiers



title



description



item type



stimulus and assets



year level and difficulty



learning area



ScOT terms



correc
t response, scoring and rubrics



the accommodations, accessi
bility features/

embedded supports



item tools
; for example, c
alculators



established psychometric data



intellectual property

and copyright



learning resources.

C

IA
-
007 (NOAC)

IU
-
02
6

The system shall provide the ability to specify metadata
information about item tools
to support students during
testing
such as but not limited to:



calculator



notepad



highlighter



drawing function



ruler
.

I

New

IU
-
02
7

The system shall provide the ability for version control of
copyright and
intellectual property

information on any
assets, items, stimuli or prompts, rubrics or other
materials associated with each item.

C

IA
-
019

(NOAC)

IU
-
02
8

The system shall provide the

ability for an authorised
user to restrict the display of any metadata based on
user roles.

C

IA
-
037

(NOAC)

IU
-
02
9

The system shall record creation, modification and
usage
metadata
(
paradata
) for each item, items sets
and associated content such as stimu
li or prompts and
graphics.

C

IA
-
017

(NOAC)

Item
s
earch

IU
-
0
30

The system shall provide the
ability

to search for all
content and items
based o
n multiple established criteria
such as
,

but not limited to
,

year level, learning area and
ScOT terms.

C

IA
-
026

(NOAC)

IA
-
015

(NOAC)

IA
-
016 (NOAC)

MD
-
006 (NOAC)

IU
-
0
3
1

The system shall provide the ability to search on item
related paradata.

D

PD
-
005

(NOAC)



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

17


5.7

Item review, approval and version control

Requirement ID

Requirement description

Priority

BRS
reference

IR
-
001

The system shall provide the ability to follow
established business rules for finalising items including
committee review and approval process.

C

IA
-
029

(NOAC)

IR
-
002

The system shall provide the ability to associate
specific business rules, workflow
s and notifications to
the flag status assigned to individual items, item sets
and all associated content such as stimuli or prompts,
graphics, rubrics, marking information
.

C

IA
-
014

(NOAC)

IR
-
003

The system shall allow authorised users to flag an
individual item and
all associated content such as
stimuli or prompts, graphics, rubrics, marking
information
as approve, modify, replace, terminate or
other required status.

C

IA
-
011

(NOAC)

IR
-
004

The sy
stem shall allow authorised users to flag multiple
items and all associated content such as stimuli or
prompts, graphics, rubrics, marking information as
approve, modify, replace, terminate or other required
status.

C

IA
-
012

(NOAC)

IR
-
005

The system shall

allow comments and ratings to be
added to items.

C

IA
-
018

(NOAC)

IR
-
006

The system shall provide the ability for authorised users
to view and/or revert back to previous versions of
individual items, stimuli or prompts or item sets.

C

IA
-
028

(NOAC)

IR
-
007

The system shall provide version control and audit
history on individual items, item sets and individual
reading stimuli or prompts.

C

IA
-
020 (NOAC)

IA
-
033 (NOAC)





Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

18


5.8

Item bank

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

IB
-
001

The system shall provide the ability to move finalised
items to a NOAC
i
tem bank component

based on
established business rules
.

C

New

IB
-
002

The system shall provide the ability to integrate with a
number of NOAC
i
tem bank components
(eg

formative,
diagnostic, jurisdiction
)
.

C

New

IB
-
00
3

The system shall provide the ability to store

items, items
sets
and
all associated content such as stimuli or
prompts, graphics, rubrics and marking information.

C

IA
-
035

(NOAC)

IB
-
00
4

The system shall
provide the ability to store all
established

psychometric data associated with items
4
.

C

IA
-
022

(NOAC)

IB
-
00
5

The system shall provide the ability to store interactive
item types, including

TEIs, technology
-
enabled items
types and all associated assets.

C

New

IB
-
00
6

The system shall enable individual items
and all
associated content to be imported
from external third
-
party item repositories
ready for

delivery
.

C

IA
-
023

(NOAC)

IC
-
003

(ACARA)

IB
-
00
7

The system shall enable multiple items and all
associated

content to be imported from external third
-
party item repositories ready for delivery via a defined
batch process.

C

IA
-
024 (NOAC)

IC
-
004 (ACARA)

IB
-
00
8

Th
e

system shall

support multiple methods for
uploading items such as ftp.

C

New

5.9

Interoperability

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

IO
-
001

The system shall
provide the ability to export and import
individual items

and all associated content in

agreed

technical interoperability standards
5
.

C

IA
-
002

(NOAC)

IO
-
002

The system shall provide the ability to export and import
multiple

items and all associated content in agreed
technical
interoperability standards
5
.

C

IA
-
002 (NOAC)

IB
-
005

IO
-
003

The system shall
export items for delivery using
technical
interoperability standards
5

for

accessibility and
accommodations
.

C

IA
-
003

(NOAC)

IO
-
004

The system shall adhere to
published
technical
interoperability standard
s
5

f
or

communicating with the
item bank.

D

I
A
-
001

(NOAC)




4

Further details
will

be provided in functional specifications.

5

C
urrent standards established by IMS Global (QTI 2.1, APIP 1.0)
must be used
where possible
.



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

19


Requirement ID

Requirement description

Priority

BRS
r
ef
erence

IO
-
005

The system shall support published technical
interoperability standards
5

fo
r communicating with all
required
NOAC
components.

C

New

IO
-
006

The system shall communicate with the test delivery
system
, data warehouse
or
any
other
NOAC component
to
record

necessary e
stablished psychometric metadata
on items.

C

IA
-
021

(NOAC)

IO
-
007

The system shall support published technical
interoperability standards
5

for linking the ite
ms to a
learning resource
.

C

New

IO
-
008

The system shall have the ability to function as
a
stand
-
alo
ne online item authoring and review system.

C

New

5.10

Reporting

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

IR
-
001

The system shall
support the specification, generation
and execution of an agreed number of standard reports
and/or extracts of item related data.

C

New

IR
-
002

The system shall support the specification, generation
and execution of any number of
ad hoc

reports and/or
extracts

of item related data.

I

New

IR
-
003

The
system

shall provide the ability to generate reports
in one or more agreed report formats
.

C

New

5.11

Identity and access management

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

Identity and
a
uthentication

IM
-
001

The system shall associate a unique identifier with each
user
.

C

New

IM
-
002

The system shall generate a unique credential for each
user
.

C

New

IM
-
003

The system shall require users to authenticate
themselves using
unique credential
s.

C

New

IM
-
004

The system shall provide the facility to create any
number of users
.

C

New

IM
-
005

The
system

shall allow importing of users from an
external source
.

I

New

Access and
a
uthorisation



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

20


Requirement ID

Requirement description

Priority

BRS
r
ef
erence

IM
-
006

The system shall control levels
of access to data and
functionality based on predefined roles and other
contextual data (
eg organisation

name) associated with
a user.

C

New

IM
-
007

The system shall associate each user with at least one
system role
.

I

New

IM
-
008

The system shall allow a
user to be associated with
more than one role
.

I

New

IM
-
009

The system shall provide the ability to manage roles,
including separation of duties
.

I

New

Administration and
s
ecurity

IM
-
010

The system shall record an audit trail for all interactions
by the

user
'
s unique identifier
.

C

New

IM
-
011

The system shall enforce a minimum password
complexity as defined in the Australian Government
'
s
Protective Security Policy Framework
6

and associated
Information Security Manua
l
7
.

I

New

IM
-
012

The system shall
provide delegated user administration
of identity data, user accounts, access and roles within

organisational

boundaries.

C

New

IM
-
013

The system shall provide delegated system
administration and configuration (
ie p
rivileged access)
that enables control
of access along organisational
boundaries.

C

New

IM
-
014

The system shall support multi
-
tenanted storage of data.

I

New

Internal system IDAM

IM
-
015

The system shall allow an administrator to manually
create, update, suspend and remove a user account.

C

New

IM
-
016

The system shall allow an administrator to manage user
passwords.

C

New

External IDAM

IM
-
017

The system shall allow an external system to create,
update, suspend and remove a user account.

C

New

IM
-
018

The system shall allow
s
ingle
sign

on

from an external
identity provider.

C

New

IM
-
019

The system shall be able to federate to an external
identity provider for role
-
based access control.

I

New

IM
-
020

The system shall provide the facility to ingest external
identifiers and associate these
with the system
'
s
identifiers.

I

New




6

http://www.protectivesecurity.gov.au/Pages/default.aspx

7

http://www.dsd.gov.au/infosec/ism/



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

21


5.12

User interface and accessibility

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

UA
-
001

The system will
be compliant with
WCAG 2.0
standards

for all formative user interactions and interface designs.

C

New

UA
-
002

The system
will

be compliant with
WCAG 2.0
standards

for all summative user interactions and interface
designs where possible.

C

New

UA
-
003

The system will follow published technical
interoperability standards
5

for standardisation of all
items.

C

AC
-
002

(NOAC)

UA
-
004

The system shall be covered by the
Disability
Discrimination Act

1992
. Equal access for people with a
disability is required by the D
isability Discrimination Act
where it can reasonably be provided.

C

New

UA
-
005

The system shall be designed to accommodate
meaningful associated text for every graphic image.

C

New

UA
-
006

The system shall be designed to accommodate
alternatives to image
maps.

C

New

UA
-
007

The system shall be designed to accommodate
descriptive comments for all photographic images.

C

New

UA
-
008

The system shall be designed to accommodate text
transcription or descriptions for audio clips.

C

New

UA
-
009

The system shall
be designed to accommodate
alternatives to other embedded objects.

C

New

UA
-
010

The system shall be designed
to
ensure
that
there is
colour and tone contrast.

C

New

UA
-
011

The system shall be designed to accommodate a
screen resolution of at least 1024 x

768.

C

New

UA
-
012

The system shall

not be built using frames.

C

New

UA
-
013

The system shall permit font size to be altered by the
user.

C

New

UA
-
014

The system shall provide online support and material
such as contextual help, user guides and support
tools
for administrators.

C

New










Non
-
functional requirements

6



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

23


6

Non
-
functional requirements

6.1

Accessibility

Assumptions

All accessibility and accommodation requirements have been identified in section 5.
12

of this
document (
'
User
i
nterface and
a
ccessibility
'
)
.

6.2

Usability

The system shall be designed in line with web
standards detailed as part of ISO 9126 Software
Engineering Product Quality and Web Content Accessibility Guidelines (WCAG 2.0) issued by
the World Wide Web Consortium (W3C). The system must comply with W3C AA require
ments.

Assumptions

Most of the

usabilit
y requirements have been identified in
section 5.12 of this document.

Some
additional usability requirements are documented below
.

Requirement ID

Requirement description

Priority

BRS
reference

UB
-
001

The system must be designed to support minimum
agreed
device specifications such as:



Windows
®

desktop



laptop/notebook



Android


tablet/iPad
®



Chromebook

/Chromebox

.

C

New

UB
-
002

The system shall be designed to support as a
minimum the following browsers and operating
systems:



Internet Explorer
®

9 (Vista, 7)



Internet Explorer 8 (XP, Vista)



Chrome


21+ (XP, Vista, 7, OSX, Chrome OS)



Firefox
®

15+ (XP, Vista, 7, OSX)



Safari
®

(XP, Vista, 7, OSX).

Note: Where a browser version is given, all
subsequent versions are included.

I

New





Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

24


6.3

System performance

Requirement
ID

Requirement description

Priority

BRS
r
ef
erence

SP
-
001

The system shall support secure and concurrent
access for at least
10,000
users

such

as
i
tem
a
uthor,
r
eviewer,
p
sychometrician,
a
dministrators,
c
urriculum
e
xpert and
t
est
m
anager
.

C

New

SP
-
00
2

The
system shall ensure that the performance is
monitored

in

real time

and
a
ny degradation in
performance is managed
.

C

New

SP
-
00
3

The system shall have a response time that does not
exceed
two
seconds
.

C

New

6.4

System availability

Assumptions

These
requirements may change when architecture and policy decisions are made.

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

SY
-
001

The
system shall be
online and available 24/7

excluding scheduled maintenance periods.

C

SA
-
001

SY
-
00
2

The
system will not exceed one hour per week of
unscheduled maintenance.

C

SA
-
007

6.5

Maintainability

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

MB
-
001

The routine maintenance on the system shall be
performed during
non
-
peak hours
.

C

MT
-
001

6.6

Data storage

Assumptions

These requirements may change when additional policies for archiving are established.

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

DT
-
00
1

The system will store all data regarding item
information and item
performance related data
according to archiving policies.

C

DS
-
007

DT
-
00
2

The system must comply with the
Archives Act 1983
.

C

DS
-
008

DT
-
00
3

The system must archive records according to
configurable
parameters
.

C

DS
-
009



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

25


Requirement ID

Requirement description

Priority

BRS
r
ef
erence

DT
-
00
4

The system shall support data
cent
r
es

across different
states of Australia.

C

DS
-
012

DT
-
005

The system

shall support database configurations
designed to enable redundancy of data.

C

DS
-
012

DT
-
006

Item related data and item performance data will be
contained within Australia.

C

DS
-
013

DT
-
007

The system must be able to retrieve archived records
as required by the business.

C

DS
-
010

6.7

Scalability

Assumptions

These requirements may change when policy decisions related to deployment are made.

Requirement

ID

Requirement description

Priority

BRS
r
ef
erence

SL
-
001

The system shall be able to
scale horizontally and
incrementally.

C

SC
-
003

6.8

Volumetrics

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

VL
-
00
1

The system must support the authoring of
approximately
85,000 items for
each tenant

annually
.

C

VM
-
003

VL
-
00
2

The system shall support the creation and
management of an unlimited number

of

administrative
and user accounts.

C

DS
-
004

VL
-
00
3

The system must store psychometric data on all
national assessments and
test
items
.

C

DS
-
011





Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

26


6.9

Deployment and hosting

Assumptions

Policy decisions regarding deployment and the development of the architecture may change the
deployment and hosting
non
-
functional requirements.

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

HS
-
001

The system shall support a m
ulti
-
tenancy
model
at the
jurisdiction and organisation level.

C

DH
-
002

HS
-
002

The system shall provide 24
/7 monitoring of the
environment.

C

DH
-
003

HS
-
003

All hosting must be in a secure environment and
follow all security policies.

C

DH
-
004

HS
-
004

The system shall have a

backup and restore
procedure.

C

DH
-
005

HS
-
005

The system shall have a

disaster recovery procedure.

C

DH
-
005

6.10

Security and privacy

Assumptions

Some of these requirements may change when policy decisions are made regarding security
and privacy.

Requirement ID

Requirement description

Priority

BRS
r
ef
erence

SV
-
001

The system shall minimally enable compliance with
personnel, physical and information security
obligations as required by the
Australian
Government
'
s

Protective Security Policy Framework
6

and
'
Information Security Manual
7
.

C

SP
-
0
1
3

SV
-
002

The system shall support
all
security policies set at a
national and
jurisdiction level.

C

SP
-
001

SV
-
003

The system shall support any privacy policies set at a
national and jurisdiction level.

C

SP
-
002

SV
-
004

The system shall provide item level security.

C

SP
-
003

SV
-
005

The system shall provide strong security for data
at
rest and support strong security for data in transit.

C

SP
-
005

SV
-
006

The
system shall support
user session timeout after a
certain period of inactivity. The system
shall

provide
the ability to configure the value for session time out.

C

New

SV
-
00
7

The system

shall support integration with third party
security and privacy services and products.

C

New










7

Appendices

7



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

28


7

Appendices

The following section provides supporting information to the abovementioned functional requirements.

7.1

Item structure




Stimulus

The text, graphic or media which enables
the student to
respond to the question

Stem

The question being posed to the student

Prompt

The
instruction on how to respond to the
questions

Interaction type

Indicates the item type, such as multiple
choice or constructed response

Metadata

Further educational details of an item
including psychometric and usage data

Marking information

Indicates the correct response(s) if
applicable for the item type and can be
used during auto
-
marking

Rubrics


Indicates the marking guide for certain
item types such as extended response


Also indicates the weighting of an item


Typically used during
human/distributed marking



Item Authoring and Review

Business Requirements

©
2013 Education Services Australia Limited

29


7.2

Process flow

The following diagram represents the actors, entities and functions involved in the item creation and review workflow.