Emma Thomson 071769431 BSc Computer Science HWU Knowledge Hub

piegazeInternet and Web Development

Dec 7, 2013 (3 years and 8 months ago)

124 views













Emma Thomson
071769431
BSc Computer Science
HWU Knowledge Hub

























Dissertation Supervisor  Rob Pooley

Dissertation Second Reader  Brian Palmer

HWU Knowledge Hub
Emma Thomson

I

Declaration


I, Emma Thomson confirm that this work submitted for assessment is my own and is
expressed in my own words. Any uses made within it of the works of authors in any form
(e.g., ideas, equations, figures, text, tables, programs) are properly acknowledged at any point
of their use. A list of references employed is included.

Signed:


Date:
HWU Knowledge Hub
Emma Thomson

II

Acknowledgements


There a number of people whom I would like to recognise for their invaluable input and
support throughout this project. Firstly I would like to thank Rob Pooley, my dissertation
supervisor, for providing continual encouragement and guidance for the duration of this
project. I would also like to thank my second reader, Brian Palmer who not only provided
essential feedback but also volunteered valuable expertise throughout my evaluation.

Special recognition must be given to Fiona Grant, who proposed the idea of the HWU
Knowledge Hub along with Helen Ashton for initially implementing the system as well as her
extensive input especially during the first stages of this project. As well as being major
contributors to the HWU Knowledge Hub they have continually guided and motivated me and
for this I owe them much gratitude. It has been a pleasure to work with them on the
evaluation and development of this system.

The time commitment given to this project by the MACS Computer Officer Iain McCrone
was very much appreciated and without his invaluable advice and skills, the initial launch of
this system would not have been possible.

Finally, I must recognise the ongoing support and patience from my friends and family, who
have continually motivated me and provided encouragement in every way possible.
HWU Knowledge Hub
Emma Thomson

III

Abstract


This project focussed on implementing and thereafter evaluating a web portal based system
created for academics and students at Heriot-Watt University to allow for easy sharing of
learning and teaching experiences, resources and practices. This system will be used to direct
people to relevant information and resources already available on the internet, therefore,
avoiding the possible duplication of content.

Evaluation played a vital role within this project and it was therefore essential to conduct a
thorough literature review in order to be confident that the correct evaluation methods were
selected for inclusion. Upon completion of the literature review, the following user evaluation
methods were selected for use: paper prototyping, questionnaires and interviews. It was very
important that feedback was collected from a large spectrum of users to ensure as many
viewpoints as possible were taken into consideration.

An initial evaluation and system development was undertaken and on completion it was
deemed that there were no substantial changes required to the system. This thereafter led to
investigating how accessible the system was from the other Heriot-Watt University campuses
as well as evaluating additional features that in future may be implemented into the system.
On completion of the various evaluations undertaken as part of this project, it can be
concluded that the HWU Knowledge Hub is ready for deployment across the university.
HWU Knowledge Hub
Emma Thomson

IV

Contents

Declaration ..................................................................................................................... I
Acknowledgements ....................................................................................................... II
Abstract ........................................................................................................................ III
Contents .......................................................................................................................IV
Table of Figures ......................................................................................................... VII
Chapter 1 - Aims and Objectives ................................................................................... 1
Chapter 2 - Literature Review ....................................................................................... 3
2.1. Quantitative Evaluation versus Qualitative Evaluation .................................................. 3
2.1.1 Quantitative Evaluation ............................................................................................ 3
2.1.2 Qualitative Evaluation .............................................................................................. 3
2.1.3 Summary................................................................................................................... 4
2.2 Assessment of Websites .................................................................................................. 5
2.2.1 Different Types of Websites ..................................................................................... 5
2.2.2 Components of a Website ......................................................................................... 5
2.2.3 Drupal ....................................................................................................................... 7
2.3 Usability Evaluation Methods (UEMs) .......................................................................... 7
2.3.1 Types of Usability Evaluation Methods ................................................................... 8
2.3.2 Empirical Methods ................................................................................................... 9
2.3.3 Inspection Methods ................................................................................................ 11
2.3.4 Inquiry Methods ..................................................................................................... 12
2.3.5 Summary................................................................................................................. 15
2.4 Social Networking Sites ................................................................................................ 15
2.4.1 Features of Social Networking Sites ...................................................................... 15
2.4.2 Social Networking sites in Education ..................................................................... 16
2.4.3 Social Networking sites in Business ....................................................................... 18
2.4.4 Summary................................................................................................................. 18
Chapter 3  Approach and Implementation ................................................................. 20
3.1 Project Methodology ..................................................................................................... 20
3.2 Risk Analysis ................................................................................................................. 21
3.2.1 Risks ....................................................................................................................... 22
3.2.2 Mitigating Methods ................................................................................................ 22
3.3 Project Implementation ................................................................................................. 26
Chapter 4  Evaluation ................................................................................................ 29
HWU Knowledge Hub
Emma Thomson

V

4.1 Paper Prototype ............................................................................................................. 29
4.1.1 Design ..................................................................................................................... 29
4.1.2 Conducting the Experiment .................................................................................... 30
4.1.3 Results .................................................................................................................... 30
4.1.4 Conclusions ............................................................................................................ 32
4.2 Initial Launch & Questionnaire ..................................................................................... 33
4.2.1 Design ..................................................................................................................... 33
4.2.2 Conducting the Experiment .................................................................................... 34
4.2.3 Results .................................................................................................................... 34
4.2.4 Conclusions ............................................................................................................ 36
4.3 Resulting Changes ......................................................................................................... 37
Chapter 5  Current and Future Accessibility of the System ...................................... 39
5.1 VISION ......................................................................................................................... 39
5.2 Praxis - Expert Interview ............................................................................................... 40
5.2.1 Design ..................................................................................................................... 40
5.2.2 Conducting the Interview ....................................................................................... 41
5.2.3 Results .................................................................................................................... 41
5.2.4 Conclusions ............................................................................................................ 41
5.3 Different Campuses Evaluation ..................................................................................... 42
5.3.1 Design ..................................................................................................................... 42
5.3.2 Conducting the Experiment .................................................................................... 43
5.3.3 Results  Dubai Campus......................................................................................... 43
5.3.4 Results  Scottish Borders Campus ........................................................................ 45
5.3.5 Conclusions ............................................................................................................ 47
Chapter 6 Additional Work and Enhancement Themes C onference ......................... 50
6.1 Additional Features Questionnaire ................................................................................ 50
6.1.1 Design ..................................................................................................................... 50
6.1.2 Conducting the Experiment .................................................................................... 51
6.1.3 Results .................................................................................................................... 51
6.1.4 Conclusions ............................................................................................................ 54
6.2 Enhancement Themes Conference ................................................................................ 55
6.2.1 Preparation for the Conference ............................................................................... 55
6.2.2 Conference Summary ............................................................................................. 57
6.2.3 Conference Presentation Session ............................................................................ 57
6.2.4 Conclusions ............................................................................................................ 58
HWU Knowledge Hub
Emma Thomson

VI

Chapter 7  Conclusions and Future Enhancements ................................................... 60
7.1 Conclusions ................................................................................................................... 60
7.2 Future Enhancements .................................................................................................... 62
Appendix 1  Gantt Chart  Original ............................................................................ A
Appendix 2  Gantt Chart  Revised ............................................................................ B
Appendix 3  Paper Prototype Document .................................................................... C
Appendix 4  Results from Paper Prototype Document ............................................... D
Appendix 5  Staff Example Content ........................................................................... E
Appendix 6  Student Example Content ........................................................................F
Appendix 7  Initial Launch Evaluation Questionnair e ............................................... G
Appendix 8  Initial Launch Evaluation Questionnair e  Edinburgh Results .............. H
Appendix 9  Expert Interview Questions ..................................................................... I
Appendix 10  Expert Interview Results ....................................................................... J
Appendix 11  Initial Launch Questionnaire  Dubai Results ..................................... K
Appendix 12  Initial Launch Questionnaire  Scotti sh Borders Results .................... L
Appendix 13  Additional Features Questionnaire ...................................................... M
Appendix 14  Additional Features Questionnaire  R esults ....................................... N
Appendix 15  Enhancement Themes Conference  HWU K nowledge Hub
Presentation Slides ........................................................................................................ O
Appendix 16  Enhancement Themes Conference  Staff Example Content ...............P
Appendix 17  Enhancement Themes Conference  Stude nt Example Content .......... Q
Appendix 18  Enhancement Themes Conference  Activ ity Worksheet ................... R






HWU Knowledge Hub
Emma Thomson

VII

Table of Figures


Figure 1 - Usability Evaluation Methods ....................................................................... 8


Figure 2 - Aspects Increasing and Decreasing Students' Motivation for Study Oriented
Social Network Site Usage ......................................................................................... 17


Figure 3 - Screenshot of the Staff View within the HWU Knowledge Hub ................ 30


Figure 4 - Screenshot of the Student View within th e HWU Knowledge Hub ........... 31


Figure 5 - Representation of responses collected re garding how easy participants
found it to 'add content' during the initial launch and evaluation - Edinburgh Campus
...................................................................................................................................... 35


Figure 6 - Representation of responses collected re garding how easy participants
found it to 'add content' during the initial launch and evaluation - Dubai Campus ..... 44


Figure 7 - Representation of responses collected re garding how easy participants
found it to 'add content' during the initial launch and evaluation - Scottish Borders
Campus ........................................................................................................................ 46


Figure 8 - Representation of responses collected re garding whether participants would
use the HWU Knowledge Hub in future - Dubai Campus ........................................... 48


Figure 9 - Representation of responses collected re garding whether participants would
use the HWU Knowledge Hub in future - Scottish Bord ers Campus .......................... 48


Figure 10 - Representation of responses collected r egarding whether participants felt
the inclusion of an 'Events' section would be advan tageous ........................................ 53



HWU Knowledge Hub
Emma Thomson

1

Chapter 1 - Aims and Objectives


The aim of the project was to implement and thereafter evaluate a web portal based system
created for academics and students at Heriot-Watt University to allow for easy sharing of
learning and teaching experiences, resources and practices. This system will be known as the
HWU Knowledge Hub and will initially be hosted on the MACS (Mathematics and Computer
Science Department) computer network server, with the longer term aim of being
incorporated into the Heriot-Watt University online blackboard entitled VISION.

One main focus during the implementation of this system was to ensure people were directed
to relevant information and resources already available on the internet, therefore, avoiding the
possible duplication of content. Past experiences indicated that many good resources were
rarely discovered, or if so were done by mistake or through word of mouth. The introduction
of this project would hopefully help structure the current approach to finding relevant
information.

A prototype system had already been developed over the previous year therefore, the initial
objective of this project was to further develop the system until it was at a level which could
be launched, to allow for user testing and evaluation by a group of volunteers. Upon
collecting and analysing the feedback, any required changes were implemented relating to the
opinions displayed during the evaluation.

To begin with, an independent evaluation preceding the trial launch was carried out to
determine the scope of any changes required. This consisted of a thorough evaluation
completed by the author to produce suggested changes, followed by the distribution of paper
prototypes to a sample group of potential users. Throughout the duration of the project
HWU Knowledge Hub
Emma Thomson

2

relevant literature was referred to, to ensure that the most optimum web development and
evaluation techniques were being applied.

The original plan was that following analysis of the initial feedback changes to the system
would be implemented and the system would be re-launched early in 2011. Thereafter, further
testing and evaluation would be undertaken, by both the volunteers who evaluated the initial
system and also by new users of the system. This would have allowed for assessment of
whether or not the changes made had been an improvement to the system. However, on
completion of the initial user feedback session, it was deemed that there were no significant
changes required which resulted in changes to the above planned activities. These are outlined
in detail later in this report.



HWU Knowledge Hub
Emma Thomson

3

Chapter 2 - Literature Review


2.1. Quantitative Evaluation versus Qualitative Evaluation

As part of the research while examining options for the evaluation of the HWU Knowledge
Hub, the differences between quantitative and qualitative evaluation techniques were assessed
to ensure that the best suited methods for evaluation were applied.

2.1.1 Quantitative Evaluation

This type of evaluation focuses on data that is in the form of numbers, or that can easily be
transformed into numbers (Sharp, Rogers and Preece, 2007, p356). This therefore allows
statistical representation of the data, which could be very useful when trying to gather an
overall majority opinion.

Quantitative evaluation can be included in numerous techniques. Sharp, Rogers and Preece
stated in 2007 that, Quantitative analysis uses nu merical methods to ascertain the magnitude,
amount, or size of something, for example the attributes, behaviour, or opinions of the
participants (Sharp, Rogers and Preece, 2007, p356 ). This was especially useful during
evaluation, as it allowed majority opinions to be gathered throughout the initial launch of the
HWU Knowledge Hub and thereafter these could be easily compared to the results collected
from the other Heriot-Watt University campuses.

2.1.2 Qualitative Evaluation

Kaplan and Maxwell describe qualitative research as follows; qualitative research is
conducted in natural settings and uses data in the form of words rather than numbers.
HWU Knowledge Hub
Emma Thomson

4

Qualitative data are gathered primarily from observations, interviews and documents, and are
analyzed by a variety of systematic techniques (Ka plan and Maxwell, 2005, p30).

From this definition it was concluded that the purpose of this evaluation technique was to
develop an understanding of how a system could be utilised within its natural environment.
Conclusions are not drawn using numbers or statistics, therefore the developer can gain a
more in-depth understanding of the users view. This evaluation method was particularly
interesting as it provided the tester with the opportunity to express their opinions in an
unrestricted environment.

This type of data could be collected using a variety of methods, examples being interviews,
observation and questionnaires. All of these methods could be applied during evaluation; but
after consideration, questionnaires (with a mixture of open and closed questions) and
interviews were selected.

2.1.3 Summary

During the evaluation of the HWU Knowledge Hub both quantitative and qualitative
evaluation methods were included. Although these are very distinct evaluation methods, it is
clear that The effectiveness and efficiency of qua ntitative analysis is driven to an important
extent by the quality of the qualitative analysis and the joint interpretation of both (Chapman
and Ward, 2003, p35).

Quantitative analysis allowed majority conclusions to be drawn while the qualitative analysis
ensured more precise views and opinions, which people expressed were considered.

HWU Knowledge Hub
Emma Thomson

5

2.2 Assessment of Websites

Within this section findings of different types of websites are presented as well as essential
components which must be considered when creating a website. It concludes with an
overview of the software used to create the HWU Knowledge Hub. As the website was
already partially implemented when the project was undertaken, this shall not cover reasoning
as to why this software was selected but more an overview of the capabilities and limitations
which were considered.

2.2.1 Different Types of Websites

De Troyer and Leune propose that, There are two di fferent kinds of Web sites: the kiosk
type and application type. A kiosk Web site mainly provides information and allows users to
navigate through that information. An application Web site is a kind of interactive
information system where the user interface is formed by a set of Web pages (De Troyer and
Leune, 1998, p86). This was a very interesting paper to read as it prompted the debate as to
which category the HWU Knowledge Hub would be included in. After careful consideration
it was clear that it fell under the category of application website, with reasoning being that
within the site there will be many links to other resources, which will navigate the user to
other pages, therefore being an interactive information type site. One of the principal HWU
Knowledge Hub focuses is to direct, not to reinvent, information already present on the
internet. Under these circumstances a kiosk web site would not be an appropriate definition
for this type of application.

2.2.2 Components of a Website

It is clear that there are many individual components which must be taken into consideration
when developing a website. These are: its content, architecture or structure and organization,
HWU Knowledge Hub
Emma Thomson

6

the presentation of the content, and the programming logic that is used to integrate the content
and its presentation within the sites structure ( Vu and Proctor, 2006, p1317).

Although the system had already been designed and partially implemented, consideration was
given to each of the components listed above when preparing the system for the initial launch,
implementing changes and while considering future additional features which may be
implemented.

Newman et al (2003) provided an outline of the current design practices. During their
research, they conducted an investigation into the different practices used when creating a
website. They invited a number of professional website developers to participate as testers
and examined recently undertaken projects from each developer. They assessed documents
and techniques used by the developers when creating these projects and from their study
created DENIM, a website design tool.

DENIM stands for Design Environment for Navigation and Information Models. DENIM is
intended for prototyping in the early stages of design but not for the creation of finished Web
sites (Newman et al, 2003, p32). It focuses on the sketching of different viewpoints and
ideas for web pages as well as encouraging the user to write different words, as discussed
during a brainstorming activity, and then moving the words around and connecting them
using arrows to display the hierarchy of the content.

The DENIM website design tool was not used in its entirety, however, certain aspects of it
were included while redeveloping the HWU Knowledge Hub and took cognisance of the
technique of encouraging the user to identify key words during the open question interviews.

HWU Knowledge Hub
Emma Thomson

7

2.2.3 Drupal

As the HWU Knowledge Hub was already in the implementation stage at the beginning of
this evaluation project, the software being used to develop the system had already been
selected. However, if after research it was concluded that Drupal was not appropriate then the
selection of another development language would be considered. Therefore this section
investigates both the capabilities and background of Drupal, in order to gain a clear
understanding as to whether or not it can support all functionality required.

The Drupal website states Drupal is a free softwar e package that allows anyone to easily
publish, manage and organize a wide variety of content on a website (Drupal, 2006, np).
During initial investigation the reasoning behind the selection of Drupal was discussed with
Helen Ashton, the origional implementer of the HWU Knowledge Hub, to gain a clear
understanding of her reasoning. It became clear that Drupal was very flexible, and
accommodates for user groups including user permissions, which would be a key factor
required by the system. Therefore from an initial evaluation it appeared to support the
principal parameters required for the HWU Knowledge Hub.

2.3 Usability Evaluation Methods (UEMs)

As noted earlier the aim of this project was to complete implementation and thereafter
evaluation of a web portal, which had already been designed and partially implemented. The
majority of literature reviewed was therefore based around methods by which the evaluation
could be undertaken. In this section the focus is on the methods of evaluation and specific
techniques which were considered suitable for aiding in the thorough evaluation of the HWU
Knowledge Hub.
HWU Knowledge Hub
Emma Thomson

8

2.3.1 Types of Usability Evaluation Methods

Nielsen stated in 1994 that, The four basic ways o f evaluating user interfaces are
automatically (usability measures computed by running a user interface specification through
some program), empirically (usability assessed by testing the interface with real users),
formally (using exact models and formulas to calculate usability measures), and informally
(based on rules of thumb and the general skill and experience of the evaluator) (Nielsen,
1994, p413). However, it was suggested that automatic methods do not work and that it was
impractical to apply formal methods to large-scale projects.

Since Nielsens categorisation of evaluation techni ques a further study had concluded that
there were three main evaluation techniques relevant to more modern applications. These are
outlined in the figure below.



Figure 1 - Usability Evaluation Methods

Three types of UEMs have been identified: empirica l methods, inspection methods, and
inquiry methods. Empirical methods are based on users experience with the system in
question collected in some systematical way. Inspection methods are conducted by usability
specialists  and sometimes software developers, or other professionals  who examine
usability-related aspects of a user interface without involving any user. Inquiry methods focus
HWU Knowledge Hub
Emma Thomson

9

on information about users likes, dislikes, needs and understanding of the system by talking
to them, observing, then using the system in real work, or letting them answer questions
verbally or in written form (Jacobsen, 1999, p7). During the development of the HWU
Knowledge Hub, the three types of Usability Evaluation categories proposed by Jacobsen
were considered. Below are details of the in depth investigation of each of these specific
methods in order to ensure appropriate methods were selected for inclusion during evaluation.

2.3.2 Empirical Methods

In this section specific aspects of empirical techniques which have been studied are
summarised. In addition any aspects which could be utilised during evaluation are outlined.

2.3.2.1 Usability Testing

This is sometimes referred to as the Thinking aloud method. User Testing (where typical
users are brought into a usability lab, asked to think aloud while performing some typical
tasks on a prototype, and questioned about their experience of the software) is probably the
most widely used technique in industry today (John, 1996, np).

As stated this is a very common method to use, however, due to the nature of the HWU
Knowledge Hub application it was not concluded that this method would prove advantageous.
Although there are many good aspects to this method, for example being able to observe the
system from a users point of view, it was not considered feasible to use when attempting to
gather a large spectrum of viewpoints. As feedback was collected from students from a
variety of year groups and disciplines, along with staff members both academic and
administrative, other methods appeared to be more appropriate.

HWU Knowledge Hub
Emma Thomson

10

2.3.2.2 Prototyping

During investigation it became clear that one of the main advantages of prototyping was that
A prototype allows stakeholders to interact with a n envisioned product, to gain some
experience of using it in a realistic setting, and to explore imagined users (Sharp, Rogers and
Preece, 2007, 531). Therefore this ensures the prospective users of the system will have
gained an understanding of how the system will work. Inclusion of this method will usually
flag up any major problems within the system and it is for this reason that a prototype was
developed for inclusion during initial user testing prior to the first launch.

Prototyping can be categorised into two distinct groups: Low-fidelity and high-fidelity. It has
been stated that low-fidelity is more advantageous as these prototypes are cheap to produce,
simple to create and easily modified. Marc Rettig (1994) stated that high fidelity prototyping
has too many associated inherent problems, and therefore low-fidelity prototyping was the
best option. High-fidelity prototypes are useful if trying to encourage interest in a product or
sell the product to potential clients, however, t his was not applicable for this project. Low-
fidelity prototypes will usually be throwaway prototypes meaning they will not directly
evolve into the final product.

A common method of low-fidelity prototyping is paper prototyping. This was considered
applicable to this project as, Paper prototyping i s a variation of usability testing where
representative users perform realistic tasks by interacting with a paper version of the interface
that is manipulated by a person playing computer, who doesnt explain how the interface is
intended to work (Snyder, 2003, p4). Therefore it was decided to include paper prototyping
within the evaluation of the HWU Knowledge Hub for a variety of reasons. Firstly, if you
identify a problem or design flaw it is difficult if you are reviewing a working system to
pinpoint exactly the section you are referring to. If you are working with paper prototypes, at
HWU Knowledge Hub
Emma Thomson

11

the very basic pen and paper level you can annotate the prototype, ensuring you point out
exactly where you think changes may be required.

This low fidelity method was therefore used as an initial evaluation before the basic system
was launched as, Paper prototypes seem inherently less finished or real than do electronic
prototypes. Your subjects may feel more comfortable working with paper prototypes and
criticising them than they would if the Web pages seemed to exist already (Boling and Frick,
1997, p322). This was a very valid point, as it may encourage people to be more honest with
their opinions, compared to if the system had already been developed, in which case they
might hold back on their true opinion if they could see how much hard work had already been
put into the system.

2.3.3 Inspection Methods

In this section specific aspects of inspection techniques which have been studied are
summarised. In addition any aspects which could be utilised during evaluation are outlined.

2.3.3.1 Heuristic Evaluation

Heuristic Evaluation is an informal method of usab ility analysis where a number of
evaluators are presented with an interface design and asked to comment on it (Nielsen and
Molich, 1990, p249). This is with the intention that the result will clearly show, an opinion
about what is good and bad about the interface (Ni elsen and Molich, 1990, p249). Within the
same paper Nielsen states that, Major advantages o f heuristic evaluation are:
· It is cheap.
· It is intuitive and it is easy to motivate people to do it.
· It does not require advance planning.
· It can be used early in the development process.
HWU Knowledge Hub
Emma Thomson

12

A disadvantage of the method is that it sometimes identifies usability problems without
providing suggestions for how to solve them (Niels en and Molich, 1990, p255). This would
play a vital role during the evaluation of the HWU Knowledge Hub as a major aspect of the
design is focussed on user interface and how easy it would be to navigate, with a key focus
being the ease of use.

2.3.3.2 Cognitive Walkthrough

As can be seen from the UEMs in figure 1 above, Cognitive walkthroughs (CW) are
categorised as an inspection method. The first ever version of CW was included in a paper
presented during the ACM CHI90 conference and stat ed, We derived a cognitive
walkthrough procedure for systematically evaluating features of an interface in the context of
the theory. Four people independently applied this procedure to four alternative interfaces for
which we have empirical usability data (Lewis et a l, 1990). This method is based not on
normal assumptions that the testers will already have some knowledge of systems design,
but on the way in which users learn to understand and operate a new system.

This method has been evolving for many years and has been tweaked in a number of ways by
different developers. In general the procedure used, (Jacobson, 1999, p16), is that an
evaluation team outlines the task that must be completed and gives the optimum set of
instructions to achieve this. The evaluator then tries to follow the instructions, which are set
out, resulting in questions being asked. If the questions appear to be problematic then the
content is assumed to be a problem.

2.3.4 Inquiry Methods

In this section specific aspects of inquiry techniques which have been studied are
summarised. In addition any aspects which could be utilised during evaluation are outlined.
HWU Knowledge Hub
Emma Thomson

13

2.3.4.1 User Satisfaction Questionnaire

Firstly it is essential to establish whether a census or a survey would be more suitable for this
type of evaluation. The main differences between these two is that everyone can complete a
census, however, a survey is targeted at a specific predefined group of people. During the
evaluation of the HWU Knowledge Hub surveys were used to gather information from
respondents within a target audience, as it would be of no use to this project to send out
questionnaires to people who had no connections with Heriot-Watt University.

The next step in developing the questionnaire was to ensure the questions were focussed and
to the point, to ensure that the responses would allow conclusions to be drawn as to whether
or not the project objectives had been met. After having developed the questions, they were
then arranged in a logical order, usually with the more technical questions at the end. It was
also very important to include an introduction, including simple steps for completing the
questionnaire and a thank you located at the end. As a final conclusion there was a box which
gave evaluators the option to include contact information. This allowed further contact with
participants in order to arrange an interview time suitable to discuss further the content of the
questionnaire.

2.3.4.2 Interviews

There are many different types of interviews which have been examined. The Evaluation
Cookbook (cited reference below) describes them as,
· The standardised, open ended interview
Strength: Makes sure questions are asked in the same way across a sample
population by different interviewers.
Weakness: Risk losing important, unanticipated, information.
· The guided or structured interview
HWU Knowledge Hub
Emma Thomson

14

Strength: Keeps interaction focused, covering same ground with respondent
sets, while allowing individual experiences to emerge.
Weakness: cannot divert far, or long, from agenda without losing part of the
story.
· The informal, conversational interview
Strength: allows the evaluator to respond quickly to individual differences
and situational changes.
Weakness: a great deal of time is needed to get systematic information
(McAteer, 1998, p40).

This shows how easy it would be to use interviews in a variety of different situations. After
consideration the author concluded that structured interview would be included within the
evaluation methods. This was the most suited to the requirements as with only one person
conducting the interviews there would be no need to take into consideration different
questions being asked by different interviewers. Informal interviews were not considered well
suited due to the fact that they require a larger time commitment, which would therefore make
it difficult to individually collect a valid sample.

The volunteers interviewed would have previously completed an evaluation questionnaire,
and the questions within the interview were extensions of earlier questions, enabling more
depth to be gathered from their original answers. Having previously completed the
questionnaire, this also ensured they were familiar with the system. It is clear that interviews
were well suited for this type of evaluation as it allows the developer to discuss and suggest
problems and solutions with potential users in a well-defined environment.
HWU Knowledge Hub
Emma Thomson

15

2.3.5 Summary

In this section the reasoning for the selection of specific evaluation methods over others has
been discussed. On completion it was concluded that paper prototypes, questionnaires and
interviews would be included within the evaluation of the HWU Knowledge Hub.

2.4 Social Networking Sites

On completion of initial user testing it became clear that there were no significant changes
required to the HWU Knowledge Hub, which then lead to a new focus. After much
deliberation one of the new main focal points was set as examining which additional features
potential users would find useful within the system. With the continually expanding use of
social networking sites being apparent in many peoples day-to-day lives, this was considered
a sensible starting point.

2.4.1 Features of Social Networking Sites

With the author having most personal experience with social networking site Facebook
(Facebook, 2011, np) this was where the investigation started. One of the main features
within Facebook is the like function, which allows users to display a positive opinion to a
comment and/or photograph posted by another user.

Another widely used feature of Facebook is the Events section which became one of the
main focal points of the user evaluation as, it could potentially play a vital role within the
HWU Knowledge Hub.

LinkedIn is another very common social networking site, which states that, Over 90
million professionals use LinkedIn to exchange information, ideas and opportunities
(LinkedIn, 2011, np). This is a site, which the author had no personal experience of, meaning
HWU Knowledge Hub
Emma Thomson

16

that research into the main features it uses had to be conducted. LinkedIn focuses on a
different clientele than many other social networking sites and it would appear that people
would use this site along with for example Facebook. After creating an account and
exploring the site it became clear that a very important feature was creating your own profile
and sharing personal information, and this was examined in the user evaluation. Another
feature within this site is common interest groups, which individuals can join. Although these
will not be examined initially, these are other feature, which may be looked at in the future.

After looking at specific features available on social networking sites, the next investigation
was into the role social networking sites play within business and education and whether or
not they were advantageous within these environments.

2.4.2 Social Networking sites in Education

While conducting background research a very interesting article was discovered in which
details regarding how Tampere University of Technology (TUT) were trying to include social
networking tools to aid the students learning (Silius, K., 2010, p137-143). It was very
interesting to note that TUT Circle (the social n etworking site they were trying to develop)
was created using the same Content Management Framework as the HWU Knowledge Hub,
namely Drupal.

Within this paper, there is a table, which details Aspects increasing and decreasing students
motivation for study orientated social network site usage. In the table below, the main
aspects which were applicable to the HWU Knowledge Hub are outlined:
HWU Knowledge Hub
Emma Thomson

17


Motivating
Non-motivating

Basic Idea and Targets Centralization of
information
Improvement of
Communication
Nothing new
Similarity to another
service available
Usage Needs Social Interaction
Enriched user profile
Irrelevant features
Lack of interactivity
Functionalities Groups
Discussions
Resource Sharing
Managing Events
Complex widgets
Irrelevant games
Usability Easy to use
Clear and simplified user
interface
Too complex structure
Content cannot be found
easily
Technical sustainability and
flexibility
Fast and stable system
Fully Functional
User feels like a test user
Unstable system

Figure 2 - Aspects Increasing and Decreasing Students' Motivation for Study Oriented
Social Network Site Usage
1


Although above there has only been selected aspects of the study outlined, it provided a
clearer understanding of the views of potential student users. The points outlined above were
very similar to those taken into consideration when developing the HWU Knowledge Hub.

In summary this paper, clearly demonstrates throughout that they feel the inclusion of social
networking sites within university life would be advantageous, however, this may not be the
case during younger years of education, for example high school, where it may prove to be
more of a hindrance than a help. Although this system has a different focus to the HWU
Knowledge Hub, some of the functionality is the same and therefore the expertise gained
while they were producing the TUT Circle was applicable here.



1
Adapted From: Silius, K., et al., 2010, p139
HWU Knowledge Hub
Emma Thomson

18

2.4.3 Social Networking sites in Business

At the moment there are many debates around the inclusion of social networking sites within
the business environment and with no clear outcome it is up to each individual company to
decide upon how they wish to handle the situation.

Fortino and Nayak present interesting arguments to cover both sides of this ongoing debate as
they state that organizations are hard pressed to be proactive and deploy these technologies
to improve business processes, increase productivity and maintain business competiveness.
The executives are also concerned in regulating the use of these services from a security and
an IT governance perspective (Fortino, A., & Nayak, A., 2010, p1). As can be identified
from this, the main disadvantage appears to be focussed around security with concerns about
the increased risk of threats associated with the inclusion of social networking sites within a
business environment. For this reason many companies will feel that this threat outweighs
the advantages.

However, on the other hand, it seems with social networking sites becoming so predominant
in everyday life, they can also be viewed as a bus iness opportunity to be approached
cautiously but optimistically, that offer new avenues for extending sales, marketing,
recruitment, research, and technical support, which complement traditional working
practices (Wilson, 2009, p55). Although these adva ntages do not directly extend to affect the
HWU Knowledge Hub, areas such as research and te chnical support could be included
within the system, depending on the content created by users.

2.4.4 Summary

Within the HWU Knowledge Hub, the security concerns discussed above were not something
which need to be considered, as the target audience is on a much smaller scale. However, as
HWU Knowledge Hub
Emma Thomson

19

stated above the advantages outlined were very interesting and support the inclusion of
features from social networking sites within the HWU Knowledge Hub.

Overall, although the HWU Knowledge Hub was never intended to become a social
networking site, there are clear similarities between the two. This therefore made it interesting
to investigate the use of social networking sites in both an educational and business
environment, as this highlights the main advantages and disadvantages involved. Having
examined the findings, the inclusion of specific features adapted from social networking sites,
does not appear to pose a threat to the functionality and security of the HWU Knowledge
Hub.
HWU Knowledge Hub
Emma Thomson

20

Chapter 3  Approach and Implementation

3.1 Project Methodology

In order to fully develop and evaluate the HWU Knowledge Hub, several different
approaches were included, which are briefly outline below. A varied spectrum of reviewers,
both staff and students, from all years and disciplines, were included to ensure valid
evaluation test data was gathered. This web portal can be used by anyone who has a HW e-
mail address and therefore took into consideration access to the system by not only lecturers
and students but also other members of the HW staff.

Initially an evaluation of the system was conducted by the author, having never seen the
system before, and from this initial changes were proposed which could be made. Paper
prototypes were then created in order to gather initial feedback from a variety of different
users. These were distributed before the initial launch of the system and provided confidence
that there were no fatal errors in the system before going ahead with the initial launch.
Evaluators were provided with a paper document, which included screen shots of the HWU
Knowledge Hub along with appropriate questions, used to focus their attention to specific
aspects of the system. This was considered as a suitable method of evaluation, as it allowed
the testers to express clearly their thoughts and opinions about specific parts of the system.
This approach also encouraged evaluators to be more open with their opinions and feedback
as they viewed the system as being in the early stages of development.

When the system was initially being launched a questionnaire was provided to gather
feedback from the sample user group. This included questions covering qualitative and
quantitative evaluation to allow both numerical/ statistical data as well as more in-depth
textual feedback to be gathered. After analysing the data, any required changes to the system
were proposed and then implemented.
HWU Knowledge Hub
Emma Thomson

21


Initially, there was going to be a final launch of the system early in 2011. This launch was to
be aimed at all HW users, with the intention of including a link on VISION. An updated
questionnaire would be provided, including the option to attend a short interview, in order to
evaluate if the changes implemented had been effective. An evaluation test group of users
would be selected, to include both testers who evaluated the initial launch and new testers to
ensure different viewpoints as represented. Finally the data gathered from all evaluations
would be compared and conclusions derived about, for example, the successfulness,
usefulness and ease of use of the HWU Knowledge Hub. Recommendations for follow-on
areas of research would also be proposed.

However, as on completion of the first launch of the HWU Knowledge Hub it was deemed
that there were no significant changes required, the focal point of this project altered slightly.
In order to assess whether or not the system was accessible from other Heriot-Watt university
campuses, the initial launch questionnaire was completed by participants from both the
Scottish Borders and Dubai Campuses. To examine the most effective way to introduce
students to the system, an expert interview was conducted as well as future additional features
questionnaires being carried out.

3.2 Risk Analysis

In order to be prepared for any risks which may have disrupted the planned development of
the HWU Knowledge Hub, the following risks were considered. On completion of this
project it was interesting to reflect back on the risks initially identified and to determine to
what extent any of these materialised.

HWU Knowledge Hub
Emma Thomson

22

3.2.1 Risks

The risks that were considered are detailed in the table below.
ID Risk Probability Effect
1 Not having enough people to effectively evaluate the system High Serious
2 People Misusing the System Low Tolerable
3 The server not being able to handle high volumes Low Tolerable
4 Missing the first launch deadline Medium Tolerable
5 Missing the second launch deadline Low Serious
6 Key staff being ill Low Serious
7 Loss of data Low Insignificant
8 Delay In transfer to new server Medium Tolerable
9 Difficulty in adapting to using new software - Drupal Medium Tolerable

3.2.2 Mitigating Methods


3.2.2.1 Risk ID - 1

Not having enough volunteers to complete the evaluation aspect of the system would have
resulted in the sample size not being statistically relevant in order to draw meaningful
conclusions. To deter this from occurring assistance was sought from both the first year and
second year PG Cap lecturers. Both very kindly agreed to ask the new staff members to try
the system and to complete the short questionnaire on both launches of the system. The option
of asking a lecturer of a first year computing module if it would be possible for them to set
this as a short task at the end of one of their classes was also considered, however, this was
not implemented. There was also the personal approach which took advantage of enrolling the
help of peers and associates in university clubs, for example both the Mens and Womens
Basketball club members.
HWU Knowledge Hub
Emma Thomson

23


In conclusion, this risk did not come to fruition and enough people completed all evaluations
which were carried out. On completion of the first launch, while collecting feedback from
participants, this risk was apparent as it was difficult to recruit as many potential users as had
been planned for. However, a sufficient sample user group was evaluated and during latter
evaluations more users than anticipated participated.

3.2.2.2 Risk ID - 2

There was considered to be a very low probability risk of people misusing the system as posts
onto the system could not be made anonymously, therefore hopefully deterring any misuse.
However, to mitigate this risk a precautionary measure of including a notification button at
the end of each page was implemented which would allow users to draw attention to any
unsuitable content, therefore bringing it to the attention of the administrators.

In conclusion, there has been no evidence of people misusing the system, however, with the
notification measure in place, once the system is being utilised within Heriot-Watt University,
this would help monitor this.

3.2.2.3 Risk ID - 3

An initial investigation, along with IT staff into the available data bandwidth of the HW
MACS server indicated, that this should not be a problem. If problems did start to occur, one
mitigation will be to limit the number of concurrent users.

In conclusion the server has been able to handle the volumes of traffic that it has been
subjected to and therefore provides confidence that this risk should not materialise.

HWU Knowledge Hub
Emma Thomson

24

3.2.2.4 Risk ID - 4

Missing the first launch deadline would have been inconvenient, however, manageable as in
the overall project planning adequate time was given to collect feedback and also to analyse
the data. This allowed alterations to the allocation of time to accommodate this risk therefore
limiting the potential impact on tasks planned for later in the project.

In conclusion, this first launch date was successfully achieved on schedule and therefore there
was no negative impact on the development of the system.

3.2.2.5 Risk ID - 5

Missing the second launch deadline was considered to be a much more serious risk, as this
date was closer to the final project deadline. However, adequate time was allowed in the plan
for the collection of feedback and analysing it, as long as the agreed system changes did not
fall too far behind. In addition the range and scale of changes proposed to be implemented in
the second release would be graded into 2 categories for inclusion. Those in the lesser
category may not be included as they would be seen to be nice touches, rather than important/
key changes.

In conclusion, this risk didnt come to fruition as the project direction changed and the second
launch and evaluation was not considered necessary at this time.

3.2.2.6 Risk ID - 6

The two key staff members involved in developing the HWU Knowledge Hub were Fiona
Grant and Helen Ashton. Unless serious illness was to affect either staff member, this would
not have a great impact on the development of the system. In addition once administrative
HWU Knowledge Hub
Emma Thomson

25

rights were granted to the author during Semester one, this mitigated the importance of Helen
Ashton being the only person who can implement changes.

In conclusion, nobody fell ill during the development period, which therefore did not affect
the work completed in any way.

3.2.2.7 Risk ID - 7

After installation on the MACS server, the Drupal executable code will be backed up
regularly. Loss of data would have caused slight problems but as it will be made up of posts
from many sources, it would always be possible to repopulate it with examples fairly quickly,
thereby maintaining the integrity of the data.

In conclusion, the system performance has been stable to-date. Once the system is in place
within the Heriot-Watt University environment, this will be monitored and data will be
regularly backed up.

3.2.2.8 Risk ID - 8

A delay in transferring the system onto an appropriate server from which it could be launched
would incur a delay as to when the system could be first launched. This would cause minor
problems; however, as long as these were resolved quickly, no serious and lasting effects
would be incurred. To prepare for such an eventuality Drupal was installed on the MACS
server, so it was set up and ready when the files were ready to be exported and transferred.

In conclusion this risk materialized and was adequately mitigated with assistance from Iain
McCrone, MACS Computer Officer. This caused a slight delay in planned activities, however
nothing which seriously limited the development and evaluation of the system.
HWU Knowledge Hub
Emma Thomson

26

3.2.2.9 Risk ID - 9

Having never before used Drupal, investigation was conducted into the functionality,
applicability and limitations of this software to ensure appropriate background knowledge
was acquired, for when administrative rights to the HWU Knowledge Hub were granted
during Semester one.

In conclusion, there were no problems identified with adapting to the use of the Drupal
software. After exploring the capabilities within the Drupal software it appears that all
identified aims of the system and future additional features may be implemented using the
available functionality of the Drupal software.

3.3 Project Implementation

For graphical representation of the project plan, please see the Gantt chart, attached in
Appendix 1. As the focus of the project changed, a revised Gantt chart was developed to
include new activities planned; this is attached in Appendix 2.

During Semester one, the initial task was to analyse the current system to gain an
understanding of the current stage in development and implementation of the HWU
Knowledge Hub. Following this, a list of changes were proposed to Fiona Grant (The HWU
Knowledge Hub concept was her idea). These changes were then agreed by Fiona and
implemented into the system.

The first type of external evaluation included was paper prototyping. These were used to
ensure that there were no fatal errors which would need to be addressed before the initial
launch. It also helped to gain an understanding of peoples views on the system and also to
HWU Knowledge Hub
Emma Thomson

27

judge peoples interest in the idea. These evaluati ons were distributed, collected and analysed
prior to the initial launch of the system.

The system was initially on a server to which no administrative rights could be given,
therefore the changes were proposed to Helen Ashton, (who initially created the HWU
Knowledge Hub). To allow Helen time to implement the desired changes, time was spent
setting up, as a parallel activity a Drupal account within the MACS server to allow the system
to be transferred once the changes had been implemented.

During this time, while awaiting the changes to be implemented, all documentation due for
deliverable one was finalised. The desired changes were implemented and the system
transferred to the MACS server. The initial system was launched along with an evaluation
questionnaire, week ending 19
th
November. Feedback was collected before the break for the
Christmas holidays and analysed before the beginning of Semester two.

The aim was to have changes to propose, based on the feedback, by the end of the first week
of Semester two. These changes were to be agreed by Fiona Grant (as the initial idea of the
system was hers). This second implementation of changes was undertaken by the author, as
once the system was on the MACS server, administrative rights were granted. The original
plan was to re-launch the system by the end of January 2011 and an updated questionnaire
would be distributed with it. The aim was to include both users who tested the initial system
and new users to allow for a wide spectrum of views to be collected. This would have allowed
feedback received from this questionnaire to be analysed and then compared to the results
from both the initial questionnaire and also the paper prototype responses.

However, as the participants of the initial launch did not highlight any significant changes
required, this resulted in a change to the planned activities. Instead of re-launching the
HWU Knowledge Hub
Emma Thomson

28

system, the focus changed to examining how accessible the system was not only at the
Edinburgh Campus but also the other Heriot-Watt campuses, as well as investigation into
additional features which may be implemented. Following this the focus was then on
completing all documentation required in time for the dissertation deadline, of 1
st
April 2011.

Throughout the two Semesters background reading into related areas was be undertaken, for
example different user evaluation methods. For this reason there was no specific time set
aside for this task.

Once all the documentation for the dissertation was completed, work on producing a poster,
commenced, for display on the 6
th
May 2011. This clearly outlined the project focus and
outcomes.
HWU Knowledge Hub
Emma Thomson

29

Chapter 4  Evaluation


4.1 Paper Prototype

After in-depth research it became clear that a paper prototype should be part of the
preliminary evaluation, as it enabled a small group of users to test the system prior to any
interaction with the final computerized system. The results were aimed at identifying any
problems that would need to be addressed prior to the launch of the system. The paper
prototype document created for this section of the evaluation can be found in Appendix 3.

4.1.1 Design

The first task was to produce a clear and concise set of step-by-step instructions using paper
prompts to enable users to complete the common task of adding content to the HWU
Knowledge Hub. This task was selected as the trial task, as the system will only succeed if
users found it easy to add interesting content. If there was nothing significant for users to
view, they would be discouraged from returning to the site, therefore it was considered
essential that users could complete this task; as it would encourage future use of the system.

This step-by-step instruction document also included, annotated screen shots and descriptions
of key aspects of the system, to aid users as they completed the specified task.

Throughout the document along with each section, associated questions were included to
promote specific focused responses. As well as this, participants were instructed before
beginning the task that they were able to annotate the document at any point they desired,
allowing them to annotate screen shots and be precise about their opinion. Also, the fact these
were anonymous, encouraged participants to be completely honest.

HWU Knowledge Hub
Emma Thomson

30

4.1.2 Conducting the Experiment

For this section of the evaluation the sample group was quite small, however, the main
priority was to include staff and students from different years and disciplines. In total there
were 10 participants, made up of both academic and non-academic staff members, and
students from various disciplines and years. Each participant received the same document and
was given a time restriction of 5 days to complete it, irrespective of whether or not they were
a staff member or student.

4.1.3 Results

During this preliminary evaluation stage the main focus was to ensure there were no fatal
errors, which would need to be addressed prior to the initial launch. The feedback received
can be seen in full in Appendix 4. Overall it was very positive with only a few minor points
which needed to be addressed.

Figure 3 - Screenshot of the Staff View within the HWU Knowledge Hub
HWU Knowledge Hub
Emma Thomson

31


Figure 4 - Screenshot of the Student View within the HWU Knowledge Hub

One of the main design focuses of the evaluation was to investigate which screen view, if any,
(See Figures 3 and 4 above) users of the system preferred. The results were close with 60 %
of participants reporting that they preferred the students view compared to 40% who preferred
the staff view. However, more interestingly, when asked if the two views, staff and student,
should be the same style and layout or different, 80% of participants voted to keep them
different. Based on these results, it was concluded that the layout and styles of the staff and
student views were adequate for their purpose. It was interesting to note that overall, the
majority of staff participants voted for the staff view, whilst the majority of students voted for
the student vote, thus showing the design of these sections, after the initial evaluation,
appeared to be adequate.

The taxonomies (content criteria) were another key area which produced interesting results.
Questions in this section focused on whether or not the taxonomies were accurate and also if
any vital areas had not been covered. In addition it was also queried whether or not it was
HWU Knowledge Hub
Emma Thomson

32

appropriate for the staff and students taxonomies to be the same. It was concluded, stated by
100% of participants, that it was more appropriate for the student and staff criteria to be
slightly different, as different content applied to each. The results also provided suggestions
for new taxonomies for both the staff and students section, which were considered and
included where appropriate.

Another section, which could be improved, was in the create content section of the site,
where the main task was focused. It was concluded by 90% of participants that the form used
to add content was easy to complete, however problems arose before this stage, during the
initial selection of which type of content people wished to add. It was suggested that a tab
layout may be more appropriate, or the addition of an example to ensure clarity. In particular
the external link option caused confusion, as par ticipants reported that they were unsure
about what was implied by this option.

4.1.4 Conclusions

On completion of this preliminary evaluation, the results appeared very positive, with no fatal
errors appearing. Areas which required attention were highlighted as well as specific aspects
which could easily be modified. This therefore led to the conclusion that there was no reason
the initial launch of the HWU Knowledge Hub could not proceed as scheduled.

On reflection of this section of the evaluation, with the experience which has been gained
while producing and executing the initial paper prototype evaluation, a more precise and
detailed paper prototype could have been developed. However, despite some sections missing
content, due to it being a preliminary evaluation, the results collected were encouraging and
provided the assurance required to progress to the initial launch stage.

HWU Knowledge Hub
Emma Thomson

33

4.2 Initial Launch & Questionnaire

Originally is was concluded that the most suitable path for developing the HWU Knowledge
Hub was to complete an initial launch, to allow for changes corresponding to user feedback as
collected, followed by a final launch, with more detailed user evaluations. Therefore the most
applicable user evaluation method, to collect large amounts of data, appeared to be through
the use of questionnaires. This method allowed both qualitative and quantitative data to be
collected, which would then impact the changes made before the final launch. Prior to the
initial launch, example content was developed and included within the system, to ensure
potential users experience was as realistic as possible. The staff example content created can
be seen in Appendix 5 and student example content can be seen in Appendix 6.

4.2.1 Design

In order to easily distribute the questionnaire to a large group of prospective users, the most
applicable method was to make use of an online, pre-existing survey tool. This survey tool
made provision for varied types of questions, with different user input to be displayed
efficiently, to allow for large amounts of user input. The questions developed focused mainly
on aspects like, how simple it was to navigate the system, users initial reactions to the ideas
and whether the participant would reuse/recommend the system.

The initial launch questionnaire mainly asked for quantitative data, however users also had
opportunities to express their opinions using qualitative data in some questions. The
questionnaire was kept as precise as possible to ensure participants time was not being
wasted and to avoid repetition within the questions. The questionnaire as used, can be seen in
full, displayed in document format in Appendix 7.
HWU Knowledge Hub
Emma Thomson

34

4.2.2 Conducting the Experiment

The method selected for distributing the questionnaire using an online tool and then via e-
mail meant little interaction with the participants. However, one advantage of this method
meant it allowed users to spend time interacting with the HWU Knowledge Hub before they
completed the questionnaire, ensuring they were not pressured into answer the questions
quickly. The questionnaire was circulated to both staff (academic and non-academic) and
students to aim for an even distribution of participants.

4.2.3 Results

The main results are detailed below, however, these can be seen in full, in Appendix 8.

The responses received from staff and student participants during the initial questionnaire was
in the majority students, with only 15% of responders being staff members. During future
evaluations, ensuring that there would be a more even representation from staff and students
would be imperative. However, it was encouraging to gather opinions from a wide scope of
students from multiple degree specifications and years, including first year students through
to post graduate students.

Participants were then asked to display whether they felt the HWU Knowledge Hub was
Well Organised, Easy To Use, Stimulating, Us eful, to which the majority of
participants reported that they either Strongly Agree or Agree. This question enabled a
general view of participants opinions to the entire system to be collected, which then allowed
for more specific aspects of the system to be examined. The responses collected after this
question, showed participants were very enthusiastic about the system and how it had been
designed.

HWU Knowledge Hub
Emma Thomson

35

To further examine how Easy to use the system was, users were asked, How simple was it
to navigate the site?. This question provided more encouraging results as 75% of respondents
stated that they either Strongly Agree or Agree that it was simple to navigate.

One of the main focal points of this questionnaire was to examine, whether or not participants
were satisfied with the taxonomies (content categories) and also how simple it was to add
content. This had also been a main focus when completing the paper prototype, however, it
was an essential aspect of whether or not the system would be a success and therefore
required further examination. 79% of participants reported that they either Strongly Agree
or Agree that it was Easy to add content (see f igure 5 below), which therefore leads to the
conclusion that the form used when adding content is adequate and straightforward to use.

Figure 5 - Representation of responses collected regarding how easy participants found it
to 'add content' during the initial launch and evaluation - Edinburgh Campus

It was also positive to see that 88% of participants stated that they either Strongly Agree or
Agree that the taxonomies were appropriate. This question included the option to leave
qualitative data, where participants mainly stated that as they used the system more, if they
discovered any taxonomies were missing, this would be a more appropriate time for them to
highlight this. As in the paper prototype, participants were asked to consider which view they
HWU Knowledge Hub
Emma Thomson

36

preferred, the staff or student one. The results produced were similar to that of the initial user
evaluation with the student view having the slightly higher majority.

A key concern expressed by some participants of the initial questionnaire was that they felt
users of the HWU Knowledge Hub should have to log in before they were able to access any
of the content. Although this is a valid point to consider, it would not comply with the aims
set out during the initial development of the system. The HWU Knowledge Hub aims to be
easily accessible to every staff member or student within Heriot-Watt University, therefore by
forcing people to log-in to view content, this could deter them from using the system.
However, as a means to discourage any misuse of the system, users must log-in prior to
adding content.

4.2.4 Conclusions

On completion of the initial launch of the HWU Knowledge Hub with the responses gathered
and analysed, an unexpected problem arose. The problem being that with the results collected,
there appeared to be no specific areas of concern within the HWU Knowledge Hub that
required critical attention. This does not mean to say that the system is flawless, however, it
did not produce grounds for the method of evaluation which was originally planned, which
would have resulted in another launch of the system. It was felt that as participants of both the
initial and final launch would see no drastic changes, there was no significant benefit to be
gained from continuing with the initially planned activity of a final launch and evaluation.

This therefore led the focus to change from evaluating and improving the system to focusing
on how accessible the system would be from the different Heriot-Watt University Campuses,
investigating the most effective way to introduce students to the system and also enquiring as
to which additional features people would find most appropriate.

HWU Knowledge Hub
Emma Thomson

37

4.3 Resulting Changes

Having never previously worked within the Drupal environment one key task of the project
was getting to know its interface and functionality before any changes could be implemented.
While transferring the system from one domain to another this encouraged the exploration of
specific aspects of Drupal, however in order to implement changes more knowledge was
considered necessary.

On completion of the user evaluation, as stated above, there were no real improvements
required for the final launch of the system. As this was a very unexpected result, this caused a
change in both the direction of the system and the project plan. This was a very positive
outcome as it provided encouragement that the system was ready to be launched and therefore
only minimal changes were required.

However, some participants had commented on the

wording of labels within the form for
adding content, stating that these were confusing. These were changed prior to the initial
launch and later feedback received from both the Scottish Borders and Dubai Campuses
indicated no negative comments towards the form. In future evaluations, this is something,
which could be examined more closely.

In addition, one point discussed on many occasions was the permissions available to different
membership levels. It was agreed that to encourage use of the system, users did not have to
log in to view content, which meant an anonymous user should be able to view and search for
content. However the permissions were initially set only to allow an authenticated user to use
the advanced search function. This seemed to oppose what had been agreed and was therefore
modified to allow anonymous users to view and search for content. In order to add content
users would still log in, as previously discussed.

HWU Knowledge Hub
Emma Thomson

38

Although no significant changes were made, hopefully these minor improvements will benefit
the users of the HWU Knowledge Hub once it has been fully released for use.

HWU Knowledge Hub
Emma Thomson

39

Chapter 5  Current and Future Accessibility of the System


5.1 VISION

One of the aims derived in the initial stages of the HWU Knowledge Hub was for it to
become part of VISION, therefore allowing it to be easily accessible to staff and students.
Currently at Heriot-Watt University VISION is the main resource for communication
(excluding e-mail) and is used for announcements and module material along with
information from the sports union. Therefore in order to fully utilize the resources available
from the HWU Knowledge Hub, its appearance on VISION would have encouraged people to
use it.

In order for this aim to become reality there would have had to be changes made to the HWU
Knowledge Hub. For example, the logo on the home page of the HWU Knowledge Hub
would almost certainly have had to be changed to the University shield; however, it is
extremely difficult to be granted permission to use the University logo. Another factor, which
would reduce the likelihood of the transfer to VISION, was that there may be no one in place
to maintain the system in future years. Although the system does not require a lot of
maintenance this would still be seen as a limiting factor.

After much deliberation the decision has been taken that it was not a sensible option at this
time to implement the HWU Knowledge Hub onto VISION. This was mainly due to the fact
that VISION shall be replaced by a new system, starting September 2011, therefore it did not
seem worthwhile to try and incorporate the HWU Knowledge Hub into VISION for such a
short period of time. In the future it may be possible for the HWU Knowledge Hub to be
implemented into the new communication system.
HWU Knowledge Hub
Emma Thomson

40

5.2 Praxis - Expert Interview

During the initial user evaluation many participants, especially the final year students,
commented that if this system had been introduced to them earlier in their university career
then they felt they would have been able to use it to its full potential. This then prompted the
thought that introducing the system through a first year module could be a plausible solution.
This would initially be piloted only for students within the MACs environment; however, it
would have the potential to be extended to all disciplines, if there was an applicable first year
module within other departments.

In order to gain a better understanding of whether or not this introduction to the system would
be possible, an expert interview was conducted with the lecturer who is in charge of the first
year module, Praxis.

5.2.1 Design

While creating the expert interview questions, it was important to ensure that the questions
flowed well and followed on in a suitable manner from the previous question. Another focus
was to allow for points where the conversation could stray from the line of questions, if it was
deemed suitable, which would accommodate for extension questions which may become
apparent during the interview. This was acceptable as in this instance there was only one
expert interview conducted, therefore providing fewer restrictions. If there was to be multiple
interviews conducted and the results compared, the interview structure would have had to be
static with little room for additional questions.

HWU Knowledge Hub
Emma Thomson

41

5.2.2 Conducting the Interview

The interview took place in the lecturers office, as it was a neutral environment. The
questions had been pre-printed, see Appendix 9, with large text boxes to allow for notes to be
recorded.

5.2.3 Results

The interview transcript can be seen in full in Appendix 10.

The main focus of the interview was to establish whether or not the lecturer thought it was
applicable to introduce the HWU Knowledge Hub to students through the first year Praxis
module. Overall, he stated that he could see no reason as to why this would not be an
effective way to introduce students to the system. He commented that in his opinion, the best
way to demonstrate the system to students would be by having a short demonstration of the
system, including how they could access and add material, including emphasis on where and
why the system might be useful to the students.

It was also discussed that ensuring staff were fully aware of the system and it benefits would
be advantageous as they could then remind and encourage pupils to make use of this resource.
Overall, he claimed the aim should be that the system would become part of ordinary
thinking among staff.

5.2.4 Conclusions

On completion of this user evaluation method, from the results above, it was clear that in
future years, it would be advantageous to introduce students to the HWU Knowledge Hub, via
HWU Knowledge Hub
Emma Thomson

42

first year modules, for example Praxis. This could then also lead to investigating whether
there are similar modules within other disciplines.

5.3 Different Campuses Evaluation

As well as considering how accessible the HWU Knowledge Hub was for students within the
Edinburgh Campus, it was also important to review what students and staff from other Heriot-
Watt University Campuses thought about the system. As the HWU Knowledge Hub was
created in Edinburgh, it was interesting to investigate the opinions of both the staff and
students from the Dubai and Scottish Borders Campuses, to ensure the system would be
applicable across all Heriot-Watts Campuses.

5.3.1 Design

One major decision to be made was whether or not to use the same questionnaire as was used
with the staff and students within the Edinburgh campus during the initial launch. There were
clear benefits of creating a new questionnaire, one of which being that it would allow for a
more defined focal point. Also as lessons were learned after the initial questionnaire, it
became clear which questions would benefit from participants being able to expand on their
answers, therefore this new knowledge could have been incorporated into the development of
a new questionnaire.

Despite the advantages of creating a new questionnaire, this would greatly reduce the ability
to compare and contrast the responses from the three campuses. Therefore as the aim of this
section of user evaluation was to consider different opinions from staff and students in all the
Heriot-Watt campuses the final decision was to use the initial questionnaire which had been
distributed in Edinburgh.

HWU Knowledge Hub
Emma Thomson

43

One addition however, was to allow the participants to be able to state which campus they
were from as this had to be taken into consideration when collaborating the results.

5.3.2 Conducting the Experiment

In order to distribute the questionnaire to both staff and students who would be willing to
participate with this investigation, e-mail contact was made with lecturers in both the Dubai
and Scottish Borders campuses. With their assistance an updated version of the initial launch
email, which staff and students at the Edinburgh campus received, was distributed. This
provided them with the main aim of the system, information about where to locate the system
and also the online survey link to the questionnaire, which they were asked to complete.

5.3.3 Results  Dubai Campus

Due to time constraints, there was only a two week window in which people were able to
complete the questionnaire. Despite this fact, the number of participants was encouraging and
the representation from the Dubai Campus was considered statistically representative, with a
mixture of Staff both academic and non-academic as well as students.

The main results are summarised below, however, these can be seen in full, in Appendix 11.

When participants were asked to reflect on their initial reactions of the HWU Knowledge
Hub, there was a positive reaction to how Well Organised and Useful the system was,
however, there was a more negative response to how Easy to Use the system was and also
how Stimulating it was.

However, when asked specifically about how clear the content was, 100% of users reported
that they either Strongly Agree or Agree that t he content was clear. Similarly when
HWU Knowledge Hub
Emma Thomson

44

questioned about how simple the site was to navigate again 100% of participants stated that
they either Strongly Agree or Agree. This was e ncouraging after the initial reactions being
very mixed.

One area, which produced very interesting results, was How easy it was to add content. The
majority of participants, see figure 6, reported that they found it easy to add content to the
system, however, one participant stated that they  Disagree with this statement, see figure 6
below. The reason given behind this opinion was, T oo many different boxes, would need a
lot of thought, which is something which was not e xpressed by any participant from the
Edinburgh Campus. As this is the first negative opinion towards this section of the HWU
Knowledge Hub, no changes will be implemented, however, if during future research this
opinion was to be backed by future participants, this section could be revisited and adapted.


Figure 6 - Representation of responses collected regarding how easy participants found it
to 'add content' during the initial launch and evaluation - Dubai Campus

The main positive point to note from results gathered is that 100% of the participants
questioned stated that they thought that they would use the HWU Knowledge Hub again at
some point in the future. This is reassuring as despite the criticisms previously stated the
participants could see the system as a tool they would make use of.
HWU Knowledge Hub
Emma Thomson

45

During this questionnaire a key focal point was whether or not the content of the drop down
menus was appropriate. In the Dubai results with 100% of participants stating that the either