Available

candlewhynotData Management

Jan 31, 2013 (4 years and 9 months ago)

230 views

A Cloud
-
based S
elf
-
assessment
Tool
for Measuring Information
Literacy

Wesley Leonard

Senior Application Programmer

Health Technology Group @ Central Michigan University

INTRODUCTION

An online self
-
assessment tool,
Ready:Set
, was developed to measure
information literacy and provide immediate feedback to the
user. This
tool
made heavy use of visual analog scales or, as they are commonly known,
“sliders.” Adaptive testing was added to the latest version of the
assessment. This allowed the difficulty of questions to fluctuate based upon
the user’s perceived (or actual) skill level.
Ready:Set

was deployed on the
Amazon Elastic Computing Cloud. Since its original incarnation in 2003, the
assessment has been completed by students from Central Michigan
University and partner institutions around the world.


ABOUT ME


Studied computer engineering and computer science at Western
Michigan University


Participated in the Computer Club @ WMU and the Kalamazoo Linux
Users Group


Worked for a failed dotcom company, 2000


2001


Worked as a freelance programmer


Started as a student at Central Michigan University in 2003


Worked as a grad assistant and was hired full
-
time in the

new college of health professions


Was introduced to Dr. Lana
Ivanitskaya

and “RRSA”


Completed M.S. in computer science in 2008

HISTORY OF
READY:SET


Development started in 2003


Original idea from 1998


Perl, MySQL, IBM AIX server


Developed by Lana V. Ivanitskaya, Ph.D. and Anne Marie Casey, A.M.L.S
. with assistance
from student programmers


Original version called “RRSA”


Research Readiness Self Assessment


Questions selected for individual subject areas and designed to accurately gauge
users abilities


Original topic was students’ ability to utilize library resources and effectively find
information



SUBJECT AREAS


Major topics: Library, Health


Information Literacy


Library resources


Search engines


Peer
-
reviewed journals versus less trustworthy content


Plagiarism


Evaluating quality of web sites


Online information (commercial sites disguised as research)


Online pharmacies


Dietary supplements

MAJOR VERSIONS


30 different versions have been created for various groups and
institutions from around the world (England, China, Brazil, Taiwan, Texas,
Calgary… to name a few)


Questions, instructions, and feedback translated (as necessary) by
volunteers


Questions customized for each institution


FEEDBACK SYSTEM


After completing the assessment, the user receives feedback based
upon their results


User may also opt to receive an email with additional resources to help
them improve their skills


User may email results to themselves at any time after completing
assessment


User may not retake assessment using same PIN

VISUAL ANALOG SCALES


Visual Analog Scales (VAS) or “Sliders”


Replaced discrete questions


400 distinct points


Customizable


SLIDER CONFIGURATIONS


With/without feedback


With/without labels (midpoint, endpoints)


Custom widths


Reverse scale (less is better)


Custom background images and custom granularity

SLIDER RESULTS

Distribution of decimal values

USERS LIKE WHOLE NUMBERS

MINOR (COOL) FEATURES


PIN


Password used to access users’ version of the assessment


Self
-
enrollment


User is given a “group code” which can be used to
generate a unique PIN


Fake web sites


Dietary supplement, online pharmacy, and several
other fake web sites were created for users to evaluate while taking the
assessment


Reporting


Completion, placement, performance reports were created
so instructors from each institution can see what their students are
doing


Export


All user information can be exported to CSV format for
analysis

ADAPTIVE TESTING


Original RRSA was long (30


40 minutes)


Adaptive testing was added to shorten test time


Currently, using simple “up or down” algorithm


Question “difficulties” determined by how often they were answered
incorrectly in the past


A set of “shadow questions” determines initial starting difficulty


In our case, shadow questions are actually questions about the user’s
perceived skills (the hypothesis being that perceived skills and actual
skills are inversely
-
related)



ADAPTIVE EXAMPLE

User completed
shadow test

~ 0.85

User receives
question with
difficulty rating
closest to 0.85

Question
answered
incorrectly, user
score dropped to
0.75

User receives
question with
difficulty rating close
to currently score

Repeat until user
score stabilizes

OUTCOMES


Subjects showed significant improvement on all measures from pre to
post when time lag is great


Short
-
term effect: strong change in self
-
perceived skills and beliefs


Research with or based on RRSA has generated 20 publications


This work has been cited 21 times


(See
http://rrsa.cmich.edu/twiki/bin/view/RRSA/Publications

for a complete list)


Collaboration with groups of students and faculty from around the
world


a truly global perspective on information literacy in the digital
age


Additional research topics: “Rogue online pharmacies”


Revenue generation


PART 2:


THE CLOUD

How we brought
Ready:Set

to the
Amazon Elastic Computing Cloud

HOW THE AMAZON EC2 WORKS


Sign up for account


Launch instance


Preconfigured or create your own


Connect to instance


Create security keys, connect as root/administrator with key


Install and configure software as needed


Execute applications


Terminate instance when completed

OUR CONFIGURATION


Small Linux instance (EC2)


Small storage space (Volume)


Simple email service


Custom script to install necessary software and bring up
Ready:Set

server

AWS CONSOLE

AWS CONSOLE

AWS CONSOLE

AWS CONSOLE

REDESIGN


In late 2009, the RRSA was to be redesigned and rebranded


Python framework


PostgreSQL

database


Re
-
implement existing features but designed for the future


Decided to bring it to the cloud…

DJANGO


A
high
-
level Python Web framework that encourages rapid development
and clean, pragmatic
design


Follows MVC paradigm


Designed
to handle two challenges: the intensive deadlines of a
newsroom and the stringent requirements of the experienced Web
developers who wrote it.



Lets
you build high
-
performing, elegant Web applications
quickly


Focuses
on automating as much as possible and adhering to the DRY
principle


http://
www.djangoproject.com


STARTUP SCRIPT

#
Script to start up RRSA server on Amazon AWS

###
Install necessary packages
:

apt
-
get install postgresql
-
8.3

pg_createcluster

8.3 main
--
start

apt
-
get
install python
-
pygresql

apt
-
get install python
-
django

apt
-
get install python
-
psycopg2


###
Point
Postgres

to static volume:

echo 'Now changing
postgres

directory... (hit ENTER to continue)'

/
etc
/
init.d
/postgresql
-
8.3 stop

read

mv /
var
/lib/
postgresql

/
var
/lib/
postgresql.BACKUP


echo '(hit ENTER to continue)...'

read


ln

-
s /app/
postgresql

/
var
/lib/
postgresql

echo '(hit ENTER to continue)...'

read

/
etc
/
init.d
/postgresql
-
8.3 start


### Start the server in the background:

python rrsa/manage.py
runserver

0.0.0.0:80 2>/
dev
/null 1>/
dev
/null &


echo 'DONE'

ADVANTAGES OF THE CLOUD


Scale quickly


High security


Failover


Reliability


Can dedicate resources to certain institutions if necessary


No need to purchase or maintain infrastructure


Cost is comparable to utilizing CMU’s infrastructure


Only pay for resources used


AMAZON PRICING

OTHER CLOUD PROVIDERS


Rackspace.com


Google


Vmware


Microsoft


+ many, many more!

ACKNOWLEDGEMENTS


Lana Ivanitskaya, Ph.D.


CMU, DHA Program


Ann Marie Casey, A.M.L.S.


Formerly with CMU’s Park Library


Programmers:
Sherzod

Ruzmetov
, Ryan Laus,
Xinxin

Wu, Jeff
Pipas
,
Weber Schulz


Lawrence Fulton, Ph.D.


Texas State University, Computer Information
Systems


Paul Albee, Ph.D.


CMU, Computer Science department


ADDITIONAL INFO


For more information, see our wiki:

http
://
rrsa.cmich.edu/twiki/bin/view/RRSA



We are always seeking research partners, so contact us to
try out
Ready:Set



Currently beta testing both cloud and
adaptive versions