September 30, 2010 Recommendation

safetroubledΚινητά – Ασύρματες Τεχνολογίες

24 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

224 εμφανίσεις

1



September 30, 2010

Recommendation


Villanova University’s LMS Replacement



Over the last two years, a faculty

pilot group
,
UNIT
staff and student

volunteers have actively researched and tested the various LMS
applications. A
total cost of ownership
model
was develop
ed
, surveys were completed, collaborative efforts with other schools
,

and testing of
the

vendor/product options

were all part of the
evaluation/due diligence process

used

to make a good decision for not only the University as a
whole but
for the individual faculty and students with varied teaching and learning needs. It is significant to note that for Villanov
a, neither
cost nor the core feature sets stood out as the bigge
st differentiators between the
competing LMS products, but the

extra features that add
enhancements to our current platform that will provide more diverse tools and capabilities in the learning environment.




During the process there was discussion about the possibility of Villanova moving towards an open
-
source so
lution for the next LMS product.
After review by the faculty, it was determined that the current release of
the more widely used open
-
source product (
Sakai
)

does not provide
the broad spectrum of features required by the faculty as a whole. The next rele
ase from the Sakai community is expected in early 2012. This
release appears to meet many of the functional requirements cited by the faculty. This could potential
ly

be a longer range strategy to be
considered by the University.


In the final discuss
ion, the main factor considered by faculty was the added functionality to the environment
-

new innovations without losing
the features of the current LMS. Support for mobile devices, more flexibility with testing, easier to use interface, etc., w
ere some

of the
discussion points that lead to the final recommendation. While all the evaluated products did retain most of the core funct
ionality required,
the Blackboard Learn release is the only application that provided enhancements and also supported the i
ntegration of the many third party
products that are currently accessed through WebCT and used across campus. Thus the Blackboard Learn system, with its famili
ar look and
feel and significant improvements, is the recommendation from the faculty as the out
come of this research process.


The attached report will be submitted to the Governance Committees for final approval.




2


The Next Step: Villanova’s Learning Management System

Prepared by UNIT’s Instructional Technologies


This paper summarizes Villanov
a University’s search for its next Learning Management System (LMS). It describes the needs assessment
process that the University undertook to gather requirements, the survey process for comments and feedback and other associat
ed documents
that resulted,
and the process by which an LMS was chosen. In the final phase, three Learning management Systems were considered as
possible alternatives:

Blackboard and

Desire2Learn
, both leading proprietary systems,
a
s well as
Sakai, open source initiative. This pape
r
summarizes all of the activity that occurred during this process and the implications of the decision for Villanova.


Introduction


During the academic year 2008
-
2009, Instructional Technologies began the process to determine what LMS should replace the
current WebCT
system. For a brief history, the University first contracted with WebCT in 1998. The company has proven to be a good partn
er with its
continual focus on teaching and learning. The University benefited from having an extensible learning pl
atform and predicable small increases
in licensing costs

throughout the years. In October

of 2005, Blackboard purchased the WebCT

company in an attempt to merge the best
-
of
-
breed in learning management systems. Once again this relationship has proven to
be beneficial to the University. Unit has been able to
continue to provide a very cost
-
effective learning management system by leveraging our relationship with the vendor and using our current
infrastructure to deliver an enterprise learning system with a

wide variety of 3
rd

party teaching tools for faculty and student.


In 2008 we renewed our contract and agreed to test a learning adaptor for Blackboard. This agreement enabled us to have a th
ree year freeze
on our license cost while awaiting delivery of a next
-
generation product as promised by Blackboard. In 2011 our c
ontract with Blackboard for
the WebCT product ends. This end of contract has been coordinated to give the University sufficient time to make a thoughtfu
l and diligent
decision in the selection of our next LM system.


Over the years we have seen a tremend
ous extension of the WebCT platform from the delivery of thousands of journal articles which are part
of the Library’s eReserve to departmental training of staff and the development of certificate programs. In reviewing usage
statistics from year
to year
over 9300 students are enrolled in a least o
ne WebCT course each semester;
we see usage of faculty as high as 62% with a
n average of
approximately 55%
on a yearly basis. Many faculty deliver video and audio content through We
bCT which enables them to have

copyrighted
material in a password protected environment thus keeping them in compliance with copyright law. Several faculty are now usi
ng a
n
eP
ortfolio system wh
ich

is also integrated with the WebCT environment. In addition to the collaborative, testing and tracking to
o
ls which
come packaged as part of WebCT, faculty also routinely use “powerlinks” (3
rd

party applications integrated with WebCT). These include
Respo
ndus, a test generator and security testing package; SafeAssign


a plagiarism dete
c
tion program; iTunes University and Wimba for
synchronous collaboration. Other third party applications have also been added in
to the environment to address
specific needs

of individual
faculty or groups of faculty, e.g., Scholar, etc.


As we move forward, several drivers will determine the direction the University will take in pursuit of a new learning manage
ment system. The
primary one, as previously mentioned, is the
software is at end
-
of
-
life. The company will no longer develop new tools for the software
platform. Blackboard will also at the end of year 2012 no longer support the
WebCT
product. This means that any further updates for product
fixes to the current t
ools, security patches, etc.., will not be released. As we look at advances in technology over the last few years, such as
mobile computing, improvements in collaborative tools, drop and drag functionalities, etc. , unfortunately faculty and studen
ts wil
l not have
access to these tools in the learning environment until we move on to a new platform.


From a central computing perspective, many things have to be considered regarding the administrative and support requirements

of a new
system. Will we need a
dditional staff in place to support a new environment? What are the integration requirements for 3
rd

party
applications? What will it take to integrate all the third
-
party applications with respect to financial and staff?



In Appendix 1, you will see a

list of functional requirements that have been aggregated from need assessments developed with input from
students, faculty and UNIT’s Instructional Technologies group. What we have attempted to do is to get feedback from all stak
eholders and
determine w
hich tools are the most used, the most effective and which tools must to be part of our next LMS. We consider gaps that exis
t and
how we can address those with 3
rd

party application
s
.


Other drivers being considered: what is the health of the vendor in

terms of financial stability
-
is the vendor or community here to stay or are
they subject to the negative effects of the current economic climate? Is the platform scalable
---

not only capable of meeting the educational
needs
of

our classrooms but the bro
ader needs of the University as a whole
---

we have many departments across campus that make use of the
LMS system to provide training for their staff, certificate programs to external users and as an environment where online col
laboration can
occur within

departments and/or among groups. The LMS has to handle many of the University's functional requirements, from educational to

institutional use. These requirements include integration with existing systems, Banner, LDAP, MyNova, etc., and functionalit
y to

meet staff
workflow and governmental regulatory standards.

3



Is the company or organization innovative


are they looking at future technology trends and actively updating their application’s tool sets?
This is an important aspect as we move forward wi
th our next learning management system. As we've seen over the last couple of years some
of the tools in the WebCT environment have become outdated since the purchase of the application by Blackboard. When we firs
t acquired
the current version, the tools
et was varied and adaptable to the learning environment capable of meeting the needs of a broad spectrum of
faculty users. With very limited development over the last few years, th
e toolset has become stagnant.

It is important that the next company
and p
roduct that we select for our new learning management system has the vision to not only put in place the current technologies

but also
continue to look at the direction of technology and incorporate tools into their suite to keep it current.


The financia
l impact in the selection of the next learning management system is also an extremely important consideration.
A separate
document
provide
d

a total of cost of ownership for the three packages being considered. The cost estimates
were

extended over a five
-
year
period.
I
n reviewing the document
, a local installation of
the two proprietary products

is

less expensive than the hosted option. However, the
costs to purchase and maintain hardware necessary to support a local installation, the staff resources and the overall enviro
nment as a whole
in terms of h
eating, air
-
conditioning,
physical space,
cabl
ing, routers, switches,
etc.,
-

all play into the final total cost of ownership. All factors
have to be considered in making a truly diligent decision in terms of our next learning management system. Although the fina
ncial impact in
terms of budget allo
cation is a considerable one
, it

is not the primary focus or the primary factor in determining the direction of our next
product. The functional use of the product in terms of meeting faculty
and student
needs in the learning environment is the primary
factor in
the selection process.


Process and Input


I
n 2008 academic year, Unit staff began reviewing the applications that existed in the LMS market. At that time the top five
products were:
Blackboard Learn,
Sakai, Moodle, Angel and Desire
2Learn. Thi
s represented three proprietary and two open source products. UNIT actively
reviewed these products in terms of the toolsets inclusive at the time, the cost and the ability to make a change and sustain

the application in
our environment.


In conjunction
with one of the classes in the Business School, UNIT reviewed the five products previously mentioned in structured process to

isolate and weigh all the parameters within the applications and place a value on the usage of the tools, the cost, and the a
bilit
y to integrate
into the environment. This process was to quantify a forward direction in the s
election process. In Appendix 3
, attached to this document you
can review some of the tables that were included in the final report. The final report was based

on student input on the usability of the
products and what made sense in terms of going forward with a replacement application. At the time, It was determined that t
o replace
WebCT with another product would not be a cost
-
effective initiative. There wo
uld be no cost savings or enhancements in toolsets that would
add value sufficient to shift another product
. A

decision was made to stay with WebCT for the interim until the next generation products
became available.

If cost were not a factor at the time, the student preference would have been for the Angel system. Subsequently Angel
was bought by Blackboard.

A
s we went through this process with the students, we were able to glean some valuable information in term
s of
student usage of a learning management system. The likes and dislikes of the student groups were tabulated as part of the Pr
oject. Student
input was captured as to what they would like to see in any of the next generation products; what tools were c
onsidered
helpful

and what
tools were absolutely
necessary
.



In the 2009/2010 academic year, UNIT has also reviewed additional applications which have either
e
merged as new possibilities in the LMS
marketplace or have been bought and rebranded by other
companies. These addi
tional applications included Cou
rseCruiser
, Edvance360,
Instructure, Web S
tudy and Moodle for the second time. This was done in conjunction with faculty input and with UNIT’s knowledge of the
toolsets that are currently used across c
ampus.


In the ACIT meeting of October 2009, the selection process of the next learning management system was discussed. A recommend
ation from
the committee was to convene a pilot group of faculty who could research and have input into the selection
process. One of the faculty
offered to present the request for faculty participants to the Faculty Senate. Other faculty present at the ACIT meeting wer
e asked to take the
request back to the individual colleges. A core group of faculty representative o
f all the colleges was convened with the task of reviewing and
researching the top applications in the learning management marketplace in November 2009


At the onset the faculty committee consisted of 40 faculty members representing each college and staff
from the Law school, UNIT and the
Falvey
Library. The Faculty Committee was asked to review, research and score the top three products in a pilot environment: Black
board
Learn Suite in a hosted environment; Sakai in a hosted and locally installed environ
ments and Desire2Learn in a hosted environment. Meetings
were scheduled to address the milestones of the project. Information was disseminated that included introductions to the app
lications being
reviewed; training session schedules, webinars for each o
f the applications; a cost estimates for each of the applications and score sheets were
provided for each of the application. A needs assessment survey also was made available for faculty to have input regarding
desired tools and
functions within the LMS
application. The first application that was introduced to the faculty was Sakai. Arrangements were made with a Sakai
reseller, RSmart, to set up a hosted environment for faculty testers for Villanova. We also installed a Sakai instance on Vi
llanova serv
ers. The
4


purpose of this installation was to ascertain whether we as a department could do the installation, the administration and th
e future support
of an open
-
source product on campus. The review and research of Sakai ran from December 2009 and contin
ued through the end of the
summer session.


The next application for review began in February 2010. The application was Desired2Learn which was hosted by the vendor.
The Faculty
Committee was given access to use sandboxes on the vendor site. Addition
ally one faculty opted to teach a summer course with the
Desire2Learn product (survey included in Appendix 3). The last application reviewed was Blackboard Learn 9.0 suite of softwa
re.
Arrangements were made with the vendor to host a Villanova installati
on. Again, faculty were given IDs and passwords with the ability to set
up a content, investigate the toolset and were provided a score sheet to rate the application. In summary beginning the firs
t of December
through August 30
th
, faculty had access to r
esearch, review and score the top three applications.


Additionally relevant documents available in the public sector, as well as LMS vendor literature were made available and play
ed a crucial role in
the review process. This literature review, as well a
s having access to the vendor applications, allowed the University to gain knowledge of the
benefits and drawbacks of each system. Public documentation in the form of surveys of technical requirements, faculty and s
tudent
evaluations, as well as vendor r
esponses to RFQs were all very informative,


As we have gone through this process, there have been additional opportunities for faculty to have input into the selection p
rocess or at a
minimum to gain knowledgeable about the possibilities of the different
alternatives. On April 22 during the Technology Expo, all vendors were
invited to participate. The vendors had the opportunity to have a table with the ability to do a half an hour or hour presen
tation. Many faculty
took advantage of this opportunity an
d spent time with the vendors discussing their products and having a demonstration of the user interface
and software functionality. On May 13th during the Teaching and Learning Strategies Day hosted by VITAL and CIT, a presentat
ion was made
describing th
e learning management selection process and its implications for Villanova over the next year. More than 40 faculty attended

the
session and information was collected in regards to faculty input and concerns about the next learning management system at V
i
llanova. On
September 8th the entire day was devoted to vendor presentations by representatives from Blackboard, Desire2Learn and RSmart.

All faculty
were invited to the presentations via Campus Currents and multiple emails were sent to the faculty parti
cipants of pilot group.
Twenty
-
two
of
faculty attended the session and were introduced to the vendor representatives and able to ask questions and have demonstrati
ons of the
individual tools of their learning management systems.

Additionally, members of
the Northeast eLearning Consortium were invited to provide
some different perspectives.


During the second week of classes (Fall 2010 semester), a survey was generated and sent to all faculty in all the colleges re
questing their input
into the selection p
rocess. The survey consisted of a series of questions and ratings of the different tools in an attempt to get a broad
indication of the functionality that faculty would like to see in the next platform. A representation of the feedback is in
cluded below

in
Figure
I
. This represents all the tools and features that have been indicated by the faculty as required in the next learning manage
ment system.
Across
-
the
-
board all of the products we are currently reviewing have these features included. The produc
ts differentiate themselves by how
well these features actually function within the toolset or if the feature is an add
-
on application. For example Sakai as an open
-
source solution
is a series of modules that are integrated in one application. The module
s are written by different people at different schools and thus
sometimes have a different look and feel. This is particularly noticeable, for example, when an HTML editor is part of one t
ool and not part of
anot
her. As a functional example, i
n the case of a text posting for a Wiki, a faculty member or student would have to know some basic HTML in
order to post formatted text.



5


Figure I


A Model for Proposed System









Appendix 4 and Appendix 5 include summaries of the score sheets and the tool survey sheets that faculty prepared as part of t
he review process. Appendix 4 is
a tabulation of all the fea
tures and tools that faculty would like to see in the next learning management system inclusive in this table are the actual
highest
ratings to the lowest ratings of the designated tools. In Appendix 5
is a su
mmary of the score sheets that faculty submitted after using the applications in a
hosted environment.


Financial Costs and Implications


A

separate document

included in the summary is the total cost of ownership for the individual products. As can be s
een the hosted solutions
versus a local installation is significantly different for all products. We have attempted to include all aspects of hosting

and self hosting to
adequately reflect the money that would have to be spent for the next
-
generation prod
uct. What is not included in this analysis is the cost that
would be absorbed by the University in terms of the physical requirements for server hosting on campus. A

study was done last year with
F
acility
M
anagement and
UnIT

to ascertain what improvement
s would be needed to maintain the data center as it was in the Mendel facility.
The cost for the physical equipment and renovations to the building to include improvements in environmentals such as heat an
d air
conditioning were substantial enough to cons
ider an option for off
site
hosting for Villanova
’s

major computing systems. A decision was made
to move the Data Ce
nter in Spring of this year (20
10) to an off
-
site location. To determine a complete and total cost of ownership right for a
local install
ation of the future LMS product a cost factor of the off
-
site hosted environment for the LMS would need to be included. The
addition of these costs
added
to the items already included in the TCO makes the opportunity for a vendor hosted solution appear to

be cost
-
effective and fiscally responsible.


Conclusion


The process used to select the next learning management system at Villanova has been laborious for both faculty and staff but

necessary in
order to make a good decision for not only the University
as a whole but for the individual faculty and students with varied teaching and
learning needs. It is significant to note that for Villanova, neither cost nor the core feature sets stood out as the bigge
st differentiators
Content MGT for
documents & video assets


Email, Chat & IM

Security

Application Sharing

Content Delivery

Online Testing

Students Tracking

Online Grade book

Course Calendar

SIS Integration

Blogging

Threaded Discussion

Photos in Grade book

LDAP Integration

Mobile Component


Respondus



ePortfolios


Respondus



Plagiarism

Detection

Drop & Drag
Functionality

Send/Receive mail
external to LMS

Grade Exchange


Respondus




Next

LM Sy
s
tem





MyNova Integration

Wikis


Respondus



6


between the
competing LMS product
s, but the extra features that added value for small groups with specific needs, e.g., wikis, mobile
computing, selective release capability, etc., seemed to drive the decision.

There are two areas that potentially could provide cost savings to
the Unive
rsity in the future. First


ePortfolios are becoming more
widely used

on campus as indivi
du
al departments and programs are
realizing their value. Blackboard is the only product with a built
-
in
portfolio

that would meet the current need
s

across campus.

We are
currently using a 3
rd

party application
(Taskstream)


our portfolio initia
ti
ves over time could
potentially

be
transitioned

t
o

the Blackboard
platform

thus saving the annual licensing cost on the Taskstream application
. The second area where cost savings is a potential would be the
consolidation of
licenses
. Currently the Law School carries a
separate

Blackboard license. Future upgrades to the next generation product
(Learn 9.2) would allow for one license for the
ca
mpus

with se
parate

administrative control for each entity.


During the process there was discussion about the possibility of Villanova moving towards an open
-
source solution for the next LMS product.
After review by the faculty, it was determined that the

current release of Sakai does not provide the broad spectrum of features required by
the faculty as a whole. As indicated by the Gartner targeted research, Appendix 6, open source solutions for higher educatio
n are expected to
have mainstream adoption wi
thin the next two years. The next release from the Sakai community is expected in early 2012. This release
appears to meet many of the functional requirements cited by the faculty.

This could potential
ly

be a longer range strategy to be considered
by t
he University.



In the final discussion, the

main factor considered by faculty was the
add
ed

functionality
to the environment
-

new

innovation
s

without losing
the features of the current LMS.
Support for mobile devices, more flexibility with testing, e
asier to use interface, etc., were some of the
discussion points that lead to the final
recommendation
. While all the
evaluated products did retain
most of
th
e

core functionality
required,

the Blackboard
Learn release is the only application
that
provided enhancements and
also support
ed

the
integration of the many
third party
products that
are
currently accessed through

WebCT. Thus the Blackboard Learn system, with its familiar look and feel
and

significant
improvements, is the recommendation from

the

faculty as the outcome of this research process.

7


Appendix 1: Functional Requirements







Course Content

Ability to post various types of content

Reporting/Access to updated roster

Require the ability to access detailed downloadable reports of student
progress, grades, attendance, and enrollment based on individual or group
criteria

Security

System must be secure with SSl encryption and password protection for
copyright compliance

Integrated Gradebook

Instructors have the ability to record and
release grades and provide feedback
privately to students.

Assignment Submission

An integrated assignment submission area gives instructors more control over
when and how assignments and content is released.

Discussion Board, Wikis, Blogs

Require intuitive interface with multiple forum capability; posts must be
gradable items with html editor

Easily Edit Course Elements

Require the ability to make edits to course structure without disrupting student
progress.

Integration support

System must be integrated w/LDAP/MyNova & Banner as well as 3
rd

party apps
currently in use

Automated Testing and Scoring

The testing tool must be able to create assessments that use the following types
of questions: Multiple Choice, Calculated Answer,
and Short Answer/Essay.
Questions can be imported from existing test banks or can be both built with the
tool. An equation editor must be available. The testing tool can support timed test
submission and completion. The automated scoring can s
core multiple

choice and
short
answers type questions with optional immediate feedback.


Student Tracking

Software must track both student
-
centered information (first access date, most
recent access date to all parts of course, e.g., readings and contributions, et
c)
and content
-
centered information (number of accesses to each page of content,
average time spent on each page of content, etc).

Ease of Use

The interface must intuitive; tools are easy to find and use.

Calendar

Ability to provide a calendar to list
course events, requirements and
assignments

Notifications

Notification or Announcement capability

Academic Integrity

Must have a core feature to detect plagiarized submissions or capable of
integration with 3
rd

party app; must also integrate with Lockdown browser or
other feature to deter cheating

Synchronous collaboration tools

Ability to collaborate with students through chat , whiteboards or 3
rd

party
software based integrated conferencing tool

8


Assigned Relative

Weights to criteria

Organized Criteria by

topic/function;

Created parent and sub criteria

Segmented criteria into Technical and

Business Hierarchies

Aggregated Similar Criteria, resulting in 75 Criteria

Began with 105 Criteria

Appendix 3:
Business
Decision Criteria


LMS Alternative Evaluation

BDM Consulting Group


The BDM Consulting Group

an organization comprised of student
s from the
Business Decision Making
(DIT)
class
of Spring 2008
at
the Villanova Business School evaluated potential alternatives to the WebCT Learning Management System. To evaluate
alternatives,
A
nalytical Hierarchy Process (AHP) and Decision
Lens software was used.



After performing the evaluation of pote
ntial LMS alternatives it was the

opinion of the BDM Consulting Group that Villanova University
should
continue to license the WebCT
(Blackboa
rd)
product
for the duration of
their contract
. This recommendation
stems from the results calculated using
the Decision Lens Suite and based on the
t
echnical,
functional and cost
comparisons provided by UNIT


A
matrix of 105 criteria
was
considered
duri
ng the evaluation. These criteria formed the

decision making model
that lead to a LMS recommendation.


A complete copy of the report prepared by BMD Consulting (

Sean Coady,
Marie Lokey, Duane Lacsamana, Matthew Didden, Scott Kosman, Seth
Strohl, Brian Cotroneo)
is available.



9


Course content posting

4.66

Discussion board

2.85

Blogs

2.18

Wikis

2.2

Whiteboard

2.71

Synchronous
collaboration

3.11

Assignments/Drop box

3.72

Quizzing

3.66

Gradebook

3.66

Rubrics

2.71

Announcements

3.5

Calendar

3.08

ePortfolio

2.2

Plagerism detection

3.46

0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Average

score out of 5; 114
respondents

Appendix 4:

Survey on importance of Tools in next LMS


Tool

Average

Score

Tool

Average

Score

Tool

Average

Score

Course content


posting

4.66

Synchronous


collaboration

3.11

Announcements

3.5

Discussion board

2.85

Assignments/

Drop box

3.72

Calendar

3.08

Blogs

2.18

Quizzing

3.66

ePortfolio

2.2

Wikis

2.2

Grade

book

3.66

Plagiarism

detection

3.46

Whiteboard

2.71

Rubrics

2.71






10


Appendix 5:

Average Scores of Products by Pilot Group

Tool

Sakai

Avg score out of 5

Blackboard

Avg score out of 5

Desire2learn

Avg score out of 5

Discussions

3.2

4.75

4

Assignments/Drop box

3.6

4.5

3.85

Group tools for
presentations/share files

3

4.5

3.75

Chat (in course)

3.5

4.75

3.87

Streaming media (w/out HTML)

2.5

5

4

Whiteboard

3.1

4.3

3.5

Assign students to groups
(instructor)

4

4.75

4.4

Self tests

3.6

4.75

4.6

Chat (system wide)

2.3

3.6

2.3

Assign students to groups (random)

3.4

4.65

4.13

Course Builder (guide for
instructors)

3.1

4.15

4.65

Templates for standard pages

3.2

3.9

4.15

Quiz tool

3.85

4.6

4.27

Selective release

3.5

4.5

4.2

Context
-
sensitive Help

3

4.8

3.8

Student tracking

4.2

4.8

4.5

Automatically graded assessment
questions

3.9

4.9

4.6

Intuitive interface

3.1

4.5

4.25

Contemporary look and feel

3.2

4.7

4.1

N=

14

8

10

Total Average

3.33

4.55

4.05




11



Appendix 6:
Gartner T
argeted
R
esearch
P
ublished in July 2010