Usability Testing Report

goldbashedΤεχνίτη Νοημοσύνη και Ρομποτική

15 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

85 εμφανίσεις


Usability Testing Report

for Savvy Languages


Xi Bi, Tong Gao, Nicolas Hung,
Caroline Mossing
, and Kevin Soong

Word Count:
3722

Professor Mark Newman,
SI

622


Evaluation of Systems and Services




Table of Contents

Executive Summary
................................
................................
................................
.....

3

Introduction

................................
................................
................................
...............

3

Methods

................................
................................
................................
.....................

4

Findings and Recommendations

................................
................................
..................

7

Finding 1: Limited system feedback in recording and lesson results.

................................
................

7

Finding 2: Limited instructions limit user understanding of system features.

...............................

8

Finding 3: Limited controls and navigation options reduce efficiency of use.

.............................

11

Finding 4: System

does not match real
-
world experience.

................................
................................
..

14

Discussion:

................................
................................
................................
................

17

Conclusion

................................
................................
................................
................

18

Appendices

................................
................................
................................
...............

19

Appendix A: Recruitment Email

................................
................................
................................
......................

19

Appendix B: Pre Test Questionnaire

................................
................................
................................
..............

20

Appendix C: Pre and Post Test Terminology Definition

................................
................................
........

21

Appendix D: Logging Form

................................
................................
................................
................................

23

U01

................................
................................
................................
................................
................................
....................

24

U02

................................
................................
................................
................................
................................
....................

25

U03

................................
................................
................................
................................
................................
....................

27

U04

................................
................................
................................
................................
................................
....................

29

U05

................................
................................
................................
................................
................................
....................

31

Appendix E: Consent Form

................................
................................
................................
................................

33

Appendix F: Moderator Script

................................
................................
................................
..........................

35

Appendix G: Post Test Debrief Questions

................................
................................
................................
....

37

U01

................................
................................
................................
................................
................................
....................

38

U02

................................
................................
................................
................................
................................
....................

38

U03

................................
................................
................................
................................
................................
....................

38

U04

................................
................................
................................
................................
................................
....................

39

U05

................................
................................
................................
................................
................................
....................

39

Appendix H: Consolidated User Comments

................................
................................
................................

41




Executive Summary

The purpose of this report is

to share with our client, Savvy Languages, the process
and outcomes of our team’s research using the method of usability testing for Savvy
Languages’ web interface. Usability testing aims to simulate a first
-
time user’s
experience of a product as closely
as possible in order to reveal any difficulties a
user might have in interacting with the product a
s well as points of excellence.
Participants were recruited using contact information gathered during survey
research for the interface earlier in the semest
er.


Each team member served as a moderator for one usability test, working from the
same script. Each usability test was attended also by a note
-
taker, who recorded
observations and comments and assisted the moderator. After each test, the
moderator and n
ote
-
taker synthesized their results and
recorded them in shared
documents. After all tests were completed, the entire team came together to
synthesize findings and generate recommendations.


Key findings include:



Limited system feedback in recording and
lesson performance causes user
confusion and delays in progress.



Users don’t utilize all system features because they are not aware of their
existence.



Users have severely limited navigation capabilities and do not see how to
complete desired tasks.



Users

notice discrepancies between situations in the interface and situations
they are likely to encounter in real life.


Based on these findings, we generated a set of recommendations to improve user
experience within the interface.


Key recommendations inclu
de:



Provide more visible status indicators and fine
-
grained performance
feedback.



Add initial tutorials or instruction to make system features obvious to users.



Provide tools for the user to identify activities within the lesson and move
around between th
em.



Make user roles clear in lesson activities, and use unambiguous and
appropriate imagery.



Introduction

Savvy is a web
-
based langu
age learning system
providing situational learning
experiences. The immersive and situational lessons provide users the o
pportunity to
practice
a

language in a more realistic scenario than is offered in more traditional
language learning programs. This unique approach
aims

to bridge the gap between
traditional academic language study and real
-
world language usage.


Since the

system is in the development process,
designers


focus is on

the
situational lesson
interface. By
ensuring a satisfying learning experience
,
designers
will increase the likelihood of users’

adopt
ing

the system into their language learning

routine, thereby

increasing

its market share.


In this project we are focusing on identifying the usability problems that exist within
the current lesson’s flow. The flow includes how users find and start the lesson after
entering the website, interact with the system dur
ing the lesson and how feedback is
gained from the system. By identifying the issues in the

flow, we can compare the
users’ understanding of the system to the existing system design and
provide useful
recommendations to
designers for improving the system’s

usability
.


To test for usa
bility issues, our team conducted usability test
s to
develop an
overview of potential

pitfalls in

the system. The strength of this

technique is
its
similarity to real
-
world situations. The think
-
aloud technique used by testers
combined with their inexperience using the system provides a comprehensive view
of system strengths and weaknesses.


Methods


Overview


Our team conducted f
ive usability tests
between April 6th and 10th. Moderators pre
-
screened

participant
s
to identify
their proficiency in Spanish
. T
he participant’s
language proficiency

determined which tasks they would perform. The moderator
and note
-
taker administered follow
-
up

questions

and debrief sessions to better
understand

the tester’s thought process. The whole
group met

after all interviews
were completed,
shared notes for each

task and
identified
common findings
,

grouped
them by theme, and formulated

recommendations.


Recruiting


Using contact information from
survey participants earlier in the semester,
we se
nt
out an

email
requesting usability testers

(Appendix A)
. It contained

a link to a
Doodle.com
poll for scheduling

sessions. The email list contained a broad range of
participants from college students to retirees. The age range coincides with the
target p
opulation specified by the client. Also, to encourage participation, a $10
Starbucks gift

card was offered to all teste
r
s.


Furthermore, to reduce bias in the study, we asked participan
ts to bring their
own
laptop to simulate their
normal computer
experience
.


Summary of testers


User

Age

Gender

(M/F)

Native

Language

Learned

Spanish?

Web
-
Based

Language Learning

Experience

(1 = None, 5 = Much
)

01

60+

M

English

Yes

1

02

30
-
39

F

English

No

3

03

50
-
59

F

English

No

1

04

30
-
39

F

English

Yes

3

05

20
-
29

F

Chinese

No

2


Physical Setup and Roles


Tests were performed in

meeting rooms around North

Quad
,

which allowed
minimum distraction while performing the tasks.


Two testers were present in each test, one note taker and one moderator. The
moderator

was in charge of informing the tester about details of the study, monitor
the tasks given and ask relevant questions to understand the tester’s thought
process. The note

taker was in charge of noting every observation, comment and
response from the tester
.


Apart from their principal roles, both testers monitored each other’s performance to
avoid biased questions and acquire valuable information.


Testing Script


Testers were introduced briefly to the scope of the study and were
given a consent
form
(appendix E
) to sign. To make sure details of the study were
clear and
testers
received the same information
, the moderator r
ead through a script (appendix F
) and
gave opportunities for testers to ask questions.


The moderator introduced

the purpose of th
e s
tudy
,
emphasizing to the tester that

the study was to analyze the system and not
user
performance. This remark was
repeated throughout the study to ensure that the tester would think aloud and not
hold back any thoughts.
Moderators made sure that the te
ster determined the end of
each task.


Pretest


Testers were introduced to the Homepage and given a few minutes to familia
rize
with the layout and use

of words. Then,

a pretest was given (Appendix B
)

asking for
general demographic

information, level
of
Spa
nish

competence and experience with
web
-
based

learning tools. The last page asked testers to describe, from what they
have seen so far in the homepage, what the terms “My Path”, “Achievem
ent”,
“Leaderboard” and “5000 XP
” meant.


Tasks


Tasks varied based
on

the tester
’s

Spanish
proficiency
. Testers
with

Spanish
experience

complete
d

two lessons (
Task 2 and 2B) while others

completed
the
introductory one (Task 2). Fu
rthermore, each task was

timed.


Task 1: Find the “Greeting” Lesson


Task 2: Finish “Greeting
” Lesson


Task 2B: Finish “Clothing” Lesson

Task 3: Find “Directions” Lesson in Path


Post Test and Debrief


Upon completion of the tasks, participants were asked to re
-
define the terms asked
during the pre
-
test to determine if their understanding

of each
had

changed.


Later, the moderator aske
d a set of questions (Appendix G
) about their overall
perception of the system and followed up with more specific ones
based on

observations during the test. Testers were repeatedly advised to not hold back
comments so we could explore the reas
ons behind their actions
.


Data Collection


After each testing session, pre
-
test questionnaires, notes an
d testing times were

tran
scribed i
nto a table (Appendix D
). Both note taker and moderator complemented
each other’s memory of the events and aided in the completion of the document.



Evaluation


After all tests were completed, the team met and presented the notes by task. For
each task,
the lesson was displayed on a screen to reinforce memory and visually
explain the tester’s actions. Common problems were consolid
ated in a Google Doc
(Appendix H
), sorted into categorie
s and frequency of appearance. The team
determined severity ratings for

each category
considering frequency, impact on

learning flow and client priorities. Lastly,
the team created
recommendations
,

keeping in mind tester comments and observations from previous heuristic
evaluations.



Severity Rating Scale

0 = don’t agree
that this is a usability problem

1 = cosmetic problem

2 = minor usability problem

3 = major usability problem; important to fix

4 = usability catastrophe; imperative to fix


Findings

and Recommendations


Finding
1:
Limited system feedback in recording and
lesson results.


Severity: 4


Recording

Each lesson includes a variety of question and answer formats
, in
cluding multiple
choice
, text
input

and
audio
recording.
Users had no trouble with multiple choice
and

text
input, but

all users had trouble using the
recording function.



Figure 1
.
Users did not notice microphone icon turn red at the base when recording.


When

users
attempted

to complete the recording part of the Greeting lesson, all
found it difficult to determine whether the system was recording
their speech. One
u
ser commented

that they
“got very confused
-

the

change was subtle
.” The
lack of
obvious system status
caused significant delay in progress through the lesson. Each
user made multiple attempts to use the recording function.


Hint on inco
rrect response

After
they finish each section of the lesson, the testers used the “Check” button
to
check their answer
s
. The system provide
s

audio
-
visual

feedback
for right and

wrong
answers. However,
the feedback does not indicate specific portions of the

answer as
wrong


only the answer as a whole.



Figure 2
. Limited feedback on incorrect response


After entering an incorrect answer, 4 of our 5 testers find it difficult to rework their
answer without additional hints or feedback on about which part is wrong. For
example, U5 reported “I does not tell me the meaning
, yet it asks me to translate”
then t
ried clicking comment button, saying that it might

provide more f
eedback
about
her answer.
Repeatedly

getting
the
wrong answer without
helpful feedback

can
caus
e frustration in users. In U1’s ca
se, this continuous failure

stopped him from
completing the ov
erall task.


Recommendation 1


Create more obvious button icons, and provide more detail
ed

feedback on
answer submission.


To improve the quality of the inter
action within the lesson, we suggest these
changes
:



Provide a more obvious recording status
indicator. Considering using colors
with higher contrast to make user recognize the status of the system more
easily.



Provide more detailed feedback whenever user submitted a wrong answer.
The information may include hint to the answer or more information
about
which part is incorrect.



If the user keep
s

getting the wrong answer, consider lower
ing the difficulty of
the question, changing

it to another format, or offering to display the correct
answer.


Finding
2: Limited instructions
limit user

understanding

of
system
features.


Severity: 4


Instruction

In the lesson, there is a primary instruction on each page that tell
s the user how to
complete the exercise (e.g. type in the box, select the correct choice), but limited
indication of how to answer the questi
on (e.g. answer this question, select Andy’s
voiced response). In

the text input field of 360
view,

U1, U2, and U3 all
were unsure

whether to answer the question, or translate what is prompted.
This lack of
instruction causes confusion.


This lack of ins
truction is especially prominent when the
manner

of interaction is not
obvious. In task 2B, there is a page that only has a conversation playback, without
any anticipated answer from user. However, since the interface is identical to the
initial questions
where user has to input their response. 60% of our testers hesitate
d

on the page
, explaining to moderators that

they expect
ed

to answer a question
.



(a)







































































(b)

Figure 3
. Side
-
by
-
side of

two lesson pages (a) recording page (b) playback
-
only
page. Testers hesitate before leaving (b), not sure if t
here will be questions
.



Another key feature in the system is the 360 discovery page, where users can use
their mouse to explore the environment

in a panorama way. However, due to lack of
specific description,
none of our testers used the highlighted navigation tools to
explore the scene
.



Figure 4
. Low discoverability:
Testers
did not
use 360 functionality
.


Help button

The help button is a single rounded button that
is always present

at the bottom left
of
the screen
during the lesson. However
,

the
button is not functional
. Therefore, it
produce
s no system feedback
.


Figure 5
. Help button


When first encountering confusion about the system,
four

participants

clicked on the
help button. They clicked multiple times

before determining

that the button is not
functioning.
U1

ask
ed ”Why

is the help button

not

working?”

expressing the need of
in si
tu
ation

help information.


Recommendation 2

Add more specific instructions, and help information on each page of the
lesson.


In order to ensure an overall satisfying interaction within the lesson, we suggest that
the system should:



Provide more detailed a
nd specific instruction about how to interact with the
interface, and how question
s

should

be answered. The instructions can be
shown
in a popup format to prevent information overload
.



For the different types of lesson page, cre
ate stronger identification
to

help
user underst
and which page is playback only and which

requires user input.
Considering using different color scheme
s

to differentiate the pages.



The implementation of the help function should be prioritized to reduce the
frustration user receives w
hen they encountered confusion in lesson.


F
inding 3: L
imited controls a
nd navigation options reduce

efficiency of
use.


Severity: 2


User flow

The current system design provide
s a

progression of different page
s

in a lesson, but
does not

well

support direc
t replay of the conversation pages in a lesson.
Each
persona’s statements can be replayed, but only by clicking on the user pictures in
the conversation. In task 2, two

testers reported that they found the conversation
was too fast, and they would like to
replay. They tr
ied to click

on the conversation
text box, but nothing happened.



Figure 6
. Conversation page
is difficult to replay.


The

progression from page to page in a

lesson should be more flexible,
supporting
both mouse and keyboard access. The
current design is very mouse
-
dependent
-

people hav
e to use their mouse to click

the check button to validate their answer,
and then the continue button to progress to the next page
. Two

of our tester
s

expressed the desire

to immediately submit the answer
without
using

the mouse
,
instead
using the Enter or R
eturn key

on their keyboard
.


Navigation

Withi
n the lesson,
the
user can only continue to
the
next page in the lesson, without
having the option

to go back to previous page or

jump to

a

specific page. On each
page of
the
lesson, there is a dot indicator that
shows lesson progress
, but without
a
descri
ption, not every user understood

its meaning. Some users ignore
d

its
presence, and U1 thought of it as the progress of the conversation pla
yback. Some
testers
tried to click

on the dots to jump to previous questions, but it does not trigger
any action in the system.



Figure 7
. The page indicator is not obvious and
does not provide any functionality
.


Recommendation 3

Provide different optio
ns to navigation and controls like jumping between
pages, conversation replay and mouse and keyboard control flow to facilitate
different user flow with the system.





For each page in a lesson, provide
a
more prominent hint abo
ut the page
indicator, or add

a

tutorial about its meaning.



Make dots in the indicator clickable, providing faster access to a specific
page. When hovering over the dots, a thumbnail preview of the respective
lesson can be shown for user to know which part of the lesson they want to
j
ump to
.



Provide
a
replay button on the conversation pages to make sure people can
listen to the conversation as many times as they would like to. Currently,
conversations can be replayed by click
ing the face, but many users
overlooked this function
. Furth
ermore, the conversation bubbles should also
be scrollable for users to take time and read the sentences.



Make the primary buttons like “check” and “continue” keyboard
executable
,
making sure people can use keyboard shortcuts, such as pressing “Enter”, to
continue the lesson.


Finding 4:
System does not match real
-
world experience.


Severity: 2


Lesson progress

In current version of the system, the
lesson progress bar does not reflect the user’s
actual lesson progress
. After

the user completes

the lesson,
the progress bar under
the lesson remain
s the same
. This behavior is m
is
align
ed

with expectation.
Users
tried to make sense of the path as it was, but were unable to gain a clear
understanding because of this lack of responsiveness.
Two of our testers thin
k that
this mismatch of expectation might prevent them from continuing other lesson
s
.



Figure 8
. Lesson progress does not change with progress of lesson


Alternative answer

Participants reported feeling

restricted
in

correct answer
s
. In the Greeting
lesson
,

U1
entered a semantically correct answer, but was unable to
achieve a response of
“Correct”
. Later
,

he changed it to the correct answer and commented that “the
system should accept
a
different type of answer with

a

similar meaning”. And U4
reported

wanting “alternative answers


‘I am doing poorly’ instead of always
answering ‘I am fine
.
’”


Mismatch with social or semantic expectation

This finding is especially important because one of the main identifying
characteristics of Savvy Languages as a pro
duct is its situational, immersive
interface.
In some of the lesson page
s
, the users recognize
d

the context of the
questi
on, but the system interaction wa
s no
t very intuitive in terms of

real
-
world
experience. First, in the conversation page, the background image
s

conflict with
users’

cultural and social expectation
s
. In task 2B, upon seeing the background of
Clothing lesson, U2 acknowledge
d

that the background is relevant to th
e topic bu
t
also noticed “t
he girl looks American o
r Canadian but speaks Spanish?”

In another
case,
a

tester found the greeting conversation mismatched with the background
image of a couple at the beach.

The conversation was a greeting one, but the couple
was not po
sitioned as if they were having that conversation.



Figure 9
. Background image is relevant to the topic but
does not align with users’

cultural/social expectation


In

the conversation response page, testers reported
being unsure of

how to identify
themselves i
n the situation. In task 2,

U3
was selecting an

answer
for the question
“How are you?”

and was unsure whether to respond

as

Andy, or
as another
participant

conversation. Only after
the answer was
validate
d did

the user decide,

“Oh, I need to click what Andy said”.




Figure 10
. Ambiguity in response:
User unsure about role

in the conversation


In the picture
-
based multiple
-
choice page, tester
s

reported having issue to identify
which word the picture is referring to. U4 commented about the picture of a jean,
where she was confused whether she should describe it as a jean or a pant, or in
other occasion the same picture might reference to belt as

well.



Figure 11
. Ambiguous semantic reference in picture: Jean or Pants or Belt



Recommendation

4


Create more intuitive mapping between real
-
world expectation and the system.
Prioritize the implementation of lesson progress, provide more realistic
i
nteraction with more relevant background image, more identified role in the
conversation, and more specific reference in the picture.


To create a more natural mapping between the user expectation and the system
interaction, we suggest:



Prioritize the impl
ementation of lesson progress. By making the progress
related to the work users done in the system, user’s investment in the system
can be further reinforced.



Select more relevant background image for conversation to improve
contextual experience in lesson
s.



Considering using the username and user picture in the conversation text box
to make users aware of t
heir role in the conversation, o
r add
a
tutorial about
what users should react in the conversation.



Add more specific description or highlight about the object referenced in
multiple choice images, reducing the semantic ambiguity.


Discussion:

Our five usability tests, conducted with users who fit the demographic profile of
Savvy Languages’ intended user

base, produced an abundance of information about
the interface, but there were a few limitations to our study.


First
, the instability of the website prevented some of the tests from being properly
administered. Since the website is still under construction, we had to call the client
whenever it was non
-
functional. Coupled with testers cancelling at the last minute
and f
requently changing appointment times or unexpectedly showing up without
confirmation,
this negatively impacted our usability tests
.
On two occasions, website
outages shortened the time we were able to dedicate to testing because of team and
testers’ schedu
les. In addition, website outages disrupt the user’s progress through
lessons. These disruptions affected the amount and quality of feedback.


Because the system is still in development, the range of tasks users could complete
was very limited, resulting
in fairly low
-
level tasks used in testing. In addition, system
features that were visible but not yet implemented (such as progress bars in ‘My
Path’) caused confusion for users but may be completely intuitive when functional.
In
these situations,
the only

recommendation that we can make is “make it work”, bu
t
“how will it be work” is

diffic
ult to imagine and outside of the scope of our research
.
We are reluctant to categorize these problems as real usability problems


the
functionality isn’t wrong; it’s j
ust not there yet. These problems
limit the scope of our
r
ecommendations

to
current functionality

only and not the system as a whole.


Since the system is currently unavailable on Internet Explorer, we were unable to
capture the exact user behavior for
tes
ters who habitually use that browser
.


Though the participants who took part in our usability testing were well distributed in
age, we had no volunteers under the age of twenty and only one under the age of
thirty. It may be useful, in future research, to

recruit participants on the younger end
of the target user age range. Our users encountered problems in implementing basic
functionality, such as scrolling, perhaps in part related to the ages of our participants.


Conclusion

Throughout our usability tes
ting process, we saw that users became confused,
were
unsure of appropriate responses, and overlooked system features because of limited
system feedback, limited user freedom, and insufficient instruction. Users frequently
misinterpreted what they were sup
posed to do in order to complete an activity or did
not see how to complete their desired actions.


In order to improve the user’s experience and correct these problems, we suggest
implementing more obvious status indicators and instructions, including an

initial
tutorial or first
-
time
-
use instructions, enabling more user freedom in navigation, and
clearly indicating user roles, as well as using imagery that is clear and appropriate to
the activity.


Our testing process may have revealed problems more relevant to older users, in
simple functionality. These difficulties should certainly not be overlooked, but future
testing should target younger users to investigate whether their experience aligns
with
that of our testers or presents different problems.


Appendices

Appendix

A: Recruitment Email


Hello!



We are the students from the
School of Information

at the
University of Michigan
who
asked you to complete a survey regarding your experience when lear
ning a new
language
.



According to our survey results, you have submitted your email and expressed interest
for attending a
usability study
with us.



The study will take from
30 to 45 minutes

and involves you navigating through a
language

learning websit
e on a laptop/macbook and sharing your opinions and thoughts
with us.



The study will take place at
North Quad
on Central Campus:

105 S State St, Ann Arbor, MI 48104



For your time, you will be rewarded a
$10 Starbucks GiftCard



If interested, please
add your info on the open time slots inside this calendar:



http://doodle.com/ieypbeqy8sa7n9at

(1)

Name

(2) Email

(3) Available time


Please note that your contact information is
hidden

to other participants.


I will send you a confirmation email afterwards.



Appendix B:
Pre Test Questionnaire

Pre
-
test Questionnaire

What is your age?




below 19




20
-

29




30
-

39




40
-

49




50
-

59




more than 59


What is your gender?




Male




Female




Prefer not to answer


What is your native language?





English





Spanish





Chinese





Other: ___________________


Have you learned spanish before?




Yes




No


Rate your prior experience with any web
-
based language learning services




1

2

3

4

5



No experience





















Lots of experience

About Savvy:


What is your first impression about the interface?


______________________________________________________________



Does the interface of Savvy look easy to use?



1

2

3

4

5



Very Difficult





















Very Easy




Appendix C:
Pre and P
ost Test Terminology Definition



Terminology: “My Path”

User

Definition

01

Pre:
What I chose to study
-

the language = spanish

Post:
History of modules completed plus where I am now,

plus modules to
complete in the future

02

Pre:
“Schooling I’ve had or courses”

Post:
Same

03

Pre:
“Categories of vocabulary”

Post:
“Options for different lessons”

04

Pre:
“What I have learned so far / lessons already completed”

Post:
“The lessons I hav
e “unlocked” and when I am in the learning process”

05

Pre:
“My learning progress”

Post:
“My path shows clear organization of courses”



Terminology: “Achievement”

User

Definition

01

Pre:
How far I have progressed in the curriculum.

Post:
Modules
completed plus scores for each module plus number of modules
repeated to get high or perfect scores plus calendar of retests to determine
retention.

02

Pre:
“Tests passed via this site by category”

Post:
Same

03

Pre:
“How far I got on each category”

Post:
“Achievement categories don’t even match the lessons”

04

Pre:
“Word groups I’ve learned or excelled at so far”

Post:
“Categories of words I have learned”

05

Pre:
“Badges for what I have learned, classes I have taken”

Post:
“Earned after finishing l
esson”



Terminology: “LeaderBoard”

User

Definition

01

Pre:
How I compare to other students using this program.

Post:
My position among other students using savvy for spanish worldwide.

02

Pre:
“Who’s ahead in this achievement of my ‘friends’”

Post:
Same

03

Pre:
“Shows levels of people”

Post:
Same

04

Pre:
“Friends / site users rankings in each language the are learning”

Post:
“Levels / scores of my “friends” on the site”

05

Pre:
“My friends’ achievement / level”

Post:

Same



Terminology: “5,000
XP”

User

Definition

01

Pre:
Very unclear, probably points needed for a passing score.

Post:
This still is unclear to me, possibly total points available.

02

Pre:
“Points needed to complete this program”

Post:
“Points received / gained”

03

Pre:
“Don’t
quite get”

Post:
Same

04

Pre:
“Points earned via activities like games, quizzes”

Post:
“How many points I earned in this section so far. Or is this how many I need
in total to get to next level?”

05

Pre:
“The more courses / exercises you have taken, the
more points you get”

Post:
“Points earned after finishing lesson”




Appendix D:
Logging Form






U01


Tester ID

1

Language
Level

Beginner

PRETEST

NOTES

U01: "The interface looks clean"

TASK 1

Find "Greetings" Lesson

Success?

yes

Time

15 min

NOTES

U01:
Hesitated about “Continue” button. “I haven’t started yet, what does continue
means”.

U01: Hesitate when clicking on the “My Path” button. “I don’t know what this path
means”.

U01: suggest that when slider reached the end, it should start from the beginnin
g.

U01: does not like the idea of leaderboard nor competition.



TASK 2A

Finish "Greetings" Lesson

Success?

yes

Time

44 min.

NOTES

U01: “Greetings should be in user’s language”

U01: “The indicator is not an intuitive mapping”, U01 thought it was
about the
conversation bubbles.

U01: “I don’t want to leave comment, so I click the continue button”

U01: (In 360) “I don’t know that these controls are about”.

U01: clicked for help button for instruction but nothing happened.

U01: “Why the help button is

not working?”

U01: “If I got the answer right, I go to continue”.

U01: go from red dot to red dot easily.

U01: “the system should accept different type of answer with similar meaning”

U01: Maybe show correct answer for reinforcement.

U01: mentioned that i
n the test mode, there should not be any hint about the text.

U01: mentioned whether the record button is recording is not obvious.

U01: the instruction says: “Repeat the word” but does not provide highlight about
which word should be spoken.

U01: kept get
ting wrong answer. After 7 attempts, U01 is annoyed and exit the
stop the task.



TASK 3

In "My Path", find "Directions" lesson

Success?

yes

Time

2 min.

NOTES

U01: look around the recommended lessons and could not found the lesson

U01: disliked the wording of “Path”, U01 commented, “Path has double meaning,
path forward and path I came from.”

U01: go to my path scroll and found the lesson.



U02

Tester ID

2

Language
Level

Medium

PRETEST

NOTES

U02: "It is very colorful"

TASK 1

Find "Greetings" Lesson

Success?

yes

Time

2 min.

NOTES

U02: Continuous clicking on “Greetings” swiping banners, nothing happens

U02: Went o My Path, Greetings was “hard to find”



TASK 2A

Finish "Greetings" Lesson

Success?

yes

Time

13 min.

NOTES

U02: After correct, automatically clicked “Continue”

U02: Kept clicking on “My comments” to redo problem because it refreshed it

U02: After checking of answer

“It doesn’t tell me to continue, button should be inside the notification box.”

U02: Multip
le problems with IE. Multiple functions not working.

U02: After “Me llamo Andy”, nothing happens, U02 continued with lesson

U02: Discovered “Recheck” button will recheck input. Started using this instead of
“Comment” button

U02: Dots in 360 view not workin
g on IE

U02: “The girl looks American or Canadian but speaks Spanish?”

U02: “Why would I report a problem?”

U02: “What if I see Andy next her engaging in the conversation?”

U02: Microphone association red = activated / deactivated not clear.

U02: In 360 st
reet conversation, U02 progressed through the pop
-
up box, not by
dragging to the next button.

U02: Survey should not be in English, too hard for english learners

U02: “A test in the end of the lesson would force me to go there”



TASK 2B

Locate "Clothing" Lesson and Finsh it

Success?

yes

Time

11 min.

NOTES

U02: “I am a visual learner, seeing them acting it out as they talk helps me learn”


U02: Answers “Need different combinations of same thing”


Answer should be right, but was wrong when checking


U02: “I don’t know what it means, I need to know what it is supposed to
mean”


Regarding incorrect notification without notice of what actually is
incorrect


U02: “If it is pronunciation problem, need to overlook”




U02: “If the error kept happening, it would be frustrating”


U02: Confusion if repeating “los zapatos” or “zapatos”


U02: 360 view dropdown options are “fun”


U02: First s
elected the speaker button in sweater, then clicked the picture.


U02: “Last multiple choice option is obvious”


U02: “La mujer lleva zapatos rojos”
-

Clicked multiple times on slow button


Replayed multiple times to answer qu
estion


U02: Is assuming that repetition only asks to repeat Andy’s phrase

TASK 3

In "My Path", find "Directions" lesson

Success?

yes

Time

10 sec.




U03

Tester ID

3

Language
Level

Medium

PRETEST

NOTES

U03: “Should be able

to click and get list of words”


U03: “I can go back and forth”


U03: “I don’t really think I should touch you” (About the “touch me” banner)


U03: “I assume the 51 is percent, but the 1000 and 5000 does not really add up”



TASK 1

Find "Greetings" Lesson

Success?

yes

Time

2 min.

NOTES

U03: Uses swiping, “I don’t really know how to get in, it should click to work”.


U03: Get to “Greetings” through “My Path”


“Not quite the way I expected it”



TASK
2A

Finish "Greetings" Lesson

Success?

yes

Time

4 min.

NOTES

U03: “I have to click check or just Enter”

U03: “I did something wrong”
-

Scared after the first full picture pops up

U03: Picks right answer from what Megan said (trying to answers her)

U03:
“Oh, I need to click what Andy said”

U03: Group View
-

“What am I supposed to do?”

U03: Group View
-

“I’m going to look at the picture for a minute”

U03: Continues

U03: “That wasn’t very much, but it seems to be done”



TASK 2B

Locate "Clothing" Lesson and Finsh it

Success?

yes

Time

11 min.

NOTES

U03: Goes to My Path from Greeting Lesson’s end window


U03: “These don’t have clothing”


U03: Scrolls down and finds lesson


U03: When incorrect and clicking continue
-

“It didn’t tell me what it really
was”


U03: “It would help me learn if I saw the answer”


U03: Discovers hovering over words pops up dialog box


U03: “I can cheat!”


U03: “The
n I will always be right!”


U03: 360 store view
-

“I guess I’m looking at pictures”


U03: Discovers the orange buttons


U03: “Before [referring to dots on people], I thought they were funny badges”




U03: After answering, U03 se
arches for dots in the same screen (no
scrolling)


U03: Keeps using hover feature to answer all questions


U03: “Anyways, he is wearing a sweater, I don’t know what ‘buscando’
means”


U03: Pauses for a second, “Oh! continue”


U0
3: Question asks to type in Spanish, U03 types in English


U03: Zapatos rojos exercise
-

Too difficult

“Doesn’t tell me anything about wrong answers”

U03: Repeats the audio many times and to recognize words.

U03: When Audio in Spanish, U03 is typin
g English

U03: “Pop
-
up should give me the right answer”

U03: “Which word should I repeat” (asking word vs. sentence)

U03: “Am I supposed to be the clerk?”

U03: “Am I supposed to respond the clerk?”

U03: “I just don’t know...”

U03: Click Andy
-

“Clerk voice

again!” (Female voice in Andy)

U03: Record “Gracias”
-

(one word vs whole sentence)

TASK 3

In "My Path", find "Directions" lesson

Success?

yes

Time

10 sec

NOTES

U03: Found out scroll down after a few second to
find the indicated lesson




U04

Tester ID

4

Language
Level

Medium

PRETEST

NOTES

U04: "Very clean and modern"

TASK 1

Find "Greetings" Lesson

Success?

yes

Time

5 sec.

NOTES

U04: Found Greetings in the Recommended Skills box

TASK 2A

Finish "Greetings"

Lesson

Success?

yes

Time

10 min.

NOTES

U04: Hesitated to click continue on initial lesson
-

wanted to hear conversation
over again b/c it was too fast

U04: Had two consecutive errors, analyzed answer and corrected mistake (without
hints)

U04: Expected
automatic continue upon right answer

U04: Got confused when no answer was expected from her
-

just dialogue on
screen

U04: Wants to be notified that she’s doing a new kind of lesson

U04: Repeated dialogue
-

sometimes both speakers, sometimes only Andy

U04:

Asked to translate a sentence, tried to answer it instead. Answered 4
different ways, never realized that she was supposed to be translating it.

U04: Liked the bell as positive reinforcement of correct answer

U04: Cannot enable microphone use within mini
-
lesson in 360 lesson

U04: Got lost in mini
-
lesson, went back to “my path”

U04: Wants info to pop up when she hovers over ‘level’ and ‘achievements’

U04: Closed mini
-
lesson box, realized she hadn’t finished lesson yet.



TASK 2B

Locate "Clothing" Lesson and Finsh it

Success?

yes

Time

9 min.

NOTES

U04: Returns to my path

U04: Finds clothing

U04: Thinks it’s partially complete b/c of the progress bar

U04: Discovers translation function, doesn’t abuse it

U04: Repeats audio multiple

times instead of hovering over word

U04: Wants ability to repeat only one word instead of whole sentence

U04: Doesn’t look around in 360

U04: Answers all dots in panorama

U04: Likes combination of typing and drop
-
down menu
-

good for different learning
st
yles and goals

U04: Would be nice to have a draggable bar for slow down function
-

specify
speed



U04: Only repeats what Andy says (post
-

zapatos rojos sentence)

U04: Repeats words to self before recording, compares her recording and the
pronunciation

U04:
Not obvious when recording button was recording
-

got very confused
-

change was subtle, hard to tell.

TASK 3

In "My Path", find "Directions" lesson

Success?

yes

Time

5 sec.

NOTES

U04: Scroll down to Directions lesson



U05

Tester

ID

5

Language
Level

Beginner

PRETEST

NOTES

U05: “I did not expect this interface, it looks very pretty”

U05: “When trying to learn Japanese and Korean, websites would be text heavy”

U05: Commenting on Recommended Skills


“I do not know if these offer language for greeting or personal skills for greeting”


“It took me a second to relate the skill to teaching languages”

U05: “It looks more like a portfolio or photo sharing website”



TASK 1

Find
"Greetings" Lesson

Success?

yes

Time

5 sec.

NOTES

U05: Located Greetings from Recommended Skills

U05: Located Savvy icon to go back to homepage



TASK 2A

Finish "Greetings" Lesson

Success?

yes

Time

37 min.

NOTES

U05: Assumes 2 people are talking to
each other

U05: “Is there a way to find English translation?”


Clicks question mark for hints on translation

U05: Starts to guess the answers

U05: Finds out about word translation, clicks to translate

U05: “The background is great but is not helpful

since I don’t know what they are
talking about”

U05: “I know “hola” because friends told me before”

U05: “Why do I need to comment if I guessed it right?”

U05: “If I got it wrong, I still don’t know what I would need to comment”

U05: “I does not tell me
the meaning, yet it asks me to translate”

“I am wild guessing and frustrated”

U05: “Is comment to complain?”

U05: After finding complain button


“So comment is not about complaining”

U05: “If I learn with someone, they can leave a comment about why

the answer is
wrong”

U05: Finds out hovering can trigger translation, not necessary to click

U05: “I did not think the words were clickable or hover”

U05: “Now this (website) makes much more sense”

U05: “It assumes basic Spanish understanding. For people

with partial
understanding, kind of guess sentence, if not right, hover to see hint and correct
answer”

U05: “For me, I have no experience at all, so it is hard”



U05: “Since I had right answers, why not go to the next question? Continue is on
the right,
hard to notice and click”

U05: “Pointless to stay at this page”

U05: “Notification should have button to move on”

U05: After accidentally clicking continue twice


“How to move back?”

U05: Using the progress bar to go back


“I thought the dots

were clickable... unless I go to homepage and redo”

U05: “It does not remember where I was previously”

U05: Keeps guessing and hovering to check for answers

U05: “Providing hint prior to question makes much more sense”

U05: “After clicking “hola”, convers
ation box should continue, not continue to get
to next step”

U05: “Actually having to listen to Andy and choose right answer makes more
sense”

U05: “Oh, it changed topic from previous”

U05: “Did I get points for guessing correctly?”

U05: “Hate to pretend

I am a guy, I am female. Why my name is Andy?”

U05: Group view


“Had to notice dots, background is too colorful”

U05: Is disappointed at multiple choice response after clicking on person.

U05: “I just recognize (words) by looking, I can’t say it or

pronounce it”

U05: “All about reading, not even pronounce, speak or write”

U05: “If I read a text, do I know its meaning?”

U05: “Buttons on bottom hint on rotate”

U05: Used record button correctly

U05: “Can’t find a way to evaluate my answer”

U05: “Is the
re a way to escape the rest of the questions?”

U05: “2 buttons in recording is confusing, maybe have 1 button, click, record, click
to hear”

U05: “Previous exercise did not practice speaking, now I do with the dots”

U05: “I don’t know of options to repeat

what the person said”

U05: “Fun, enjoyable experience”

U05: “Need more of click person for conversation type”

U05: “360 not necessary, illustrate conversation with other people enough”

U05: “The correct answers (ending window) are about guess, not record
ing”

TASK 3

In "My Path", find "Directions" lesson

Success?

yes

Time

5 sec.

NOTES

U05: Found through accessing my path from Geetings lessons ending window



Appendix E: Consent

Form

Consent to Participate in Research

Identification of Investigators & Purpose of Study

You are being asked to participate in a research study conducted by Xi Bi, Tong
Gao, Nicolas Hung, Caroline Mossing, and Kevin Soong from the University of
Michiga
n.

The purpose of this study is to identify areas of the Savvy Languages web
interface that may be improved, concentrating specifically on in
-
lesson instruction
and design as well as the design of the interface. This study will contribute to the
researche
rs’ completion of their project for SI 622: Evaluation of Systems and
Services.

Research Procedures

Should you decide to participate in this research study, you will be asked to sign this
consent form once all your questions have been answered to your sati
sfaction. This
study consists of a usability test that will be administered to individual participants in
North Quad. You will be asked to complete a series of tasks using the Savvy
Languages web interface for the purpose of identifying usability issues an
d possible
solutions.




Length of Participation

Participation in this study will require no more than 60 minutes of your time.

Risks

No more than minimal risks from your involvement in this study.

Benefits

Though no direct benefit is expected on your part, your participation in this study will
help

improve an innovative, free language learning system.

Costs to Study Participants Resulting from Participation

There are no financial costs associated with this

study.

Payments to Study Participants

You will receive a $10 Starbucks gift card as compensation.



Confidentiality

The results of this research will be presented to the professor and students of SI 622
and the development team at Savvy Languages.

The re
sults of this project will be
coded in such a way that the respondent’s identity will not be attached to the final
form of this study.

The researcher retains the right to use and publish non
-
identifiable data.

While individual responses are confidential,

aggregate data will be
presented representing averages or generalizations about the responses as a
whole.

All data will be stored in a secure location accessible only to the
researcher.

Upon completion of the study, all information that matches up indiv
idual
respondents with their answers will be destroyed.

Participation & Withdrawal

Your participation is entirely voluntary.

You are free to choose not to
participate.

Should you choose to participate, you can withdraw at any time without
consequences o
f any kind.

Questions about the Study

If you have questions or concerns during the time of your participation in this study,
or after its completion or you would like to receive a copy of the final aggregate
results of this study, please contact:



Kevin S
oong

School of Information

University of Michigan

kjsoong@umich.edu



Giving of Consent

I have read this consent form and I understand what is being requested of me as a
participant in this study.

I freely consent to participate.

I have been given
satisf
actory answers to my questions.

The investigator provided me with a copy of
this form.

I certify that I am at least 18 years of age.



______________________________________


Name of Participant (Printed)

______________________________________

______________

Name of Participant (Signed)
































Date



______________________________________

______________

Name of Researcher (Signed)































Date



______________________________________

______________

Name of Researcher (Signed)































Date





Appendix F: Moderator Script

Savvy Languages Usability Test


Hello, I’m __________, and this is my colleague _____________. I will be
conducting this study, and ______ will be taking
notes. Thank you for agreeing to
participate in this usability study for Savvy Languages. I’ll be reading from a script to
ensure that the test is conducted in the same way for each participant.



CONSENT:


To begin, I’d like you to read over this
consent

form

and fill it out.


Participant fills out Consent Form.



INTRODUCTION:


I’ll begin by explaining the purpose of this study. We would like to understand more
about how users interact with the existing system and where problems exist for us to
know where changes can be made for improvement. The entire process should take
no more

than an hour.


Please understand the study is to test the design of the website and not how
you perform.

We would like to understand how you think and therefore change the
website design to complement your thought process.


I will ask you to perform a se
ries of tasks using the Savvy Languages web interface
and ______ and I will observe and take notes about your process.


During the session, we want you to “
think aloud
” about what you are doing and
guiding us through the your thought process.
The more you

tell us about what you
are thinking throughout the process, the better we will be able to formulate our
recommendations for developers at Savvy Languages.


Feel free to comment on any aspects of the system or the experience you
particularly like or disli
ke.
We did not design the website and therefore won't take
any comments personally; we just want to know how to make it better.



The information you give us will be recorded

anonymously

and combined with data
from our other participants to inform recommendations for further interface
development.


You may ask me questions at any time; however, I will not provide hints unless
you are completely stumped.



We want your experience to be as

close as possible to an independent use of
the site. _____ and I may occasionally ask clarifying questions throughout the
process.


Do you have any questions at this time?



PRE
-
TEST:


Please take a look at the homepage and familiarize with it
-

we won’
t begin the tasks
yet, but we’d like you to fill out this initial questionnaire based on your first
impression.


Participant takes Pre
-
Test questionnaire.


Thank you.

Now, we’ll begin with the tasks.


TASKS:


Task 1: Find the “Greeting” Lesson

To begin,
I’d like you to locate the “Greetings” lesson.



Make sure participant says “done” when he/she is done



Task 2: Finish “Greeting” Lesson

Now that you’ve found the Greeting lesson, go through it and answer the
questions as well as you can. Don’t worry abou
t whether you get them right or
wrong. Just remember to think aloud as you’re doing this.


Make sure participant says “done” when he/she is done



Task 2B: Finish “Clothing” Lesson
(for advanced learners)

That was great! Now I’d like you to find the less
on on Clothing and go
through it, answering the questions as well as you can. Once again,
don’t worry about whether your answers are right or wrong.






Make sure participant says “done” when he/she is done


Task 3: Find “Directions” Lesson in Path

Please

navigate to the Path and locate the lesson about
asking for
directions
.


Make sure participant says “done” when he/she is done

Appendix G: Post Test Debrief
Questions

Post
-
Test:


1.

What were your thoughts on the specific tasks you carried out today?



What
made it difficult or easy?



Finding “Greetings” Lesson




Finishing “Greetings” Lesson




Finishing “Clothing” Lesson




Finding “Directions” Lesson


4.

Which was the most difficult task, and why?


5.

Ask About user comments during Usability Test


6.

Do you have
other

comments about today’s usability test?


Debrief:


1.

What else do you think should be on the
home page
?


2.

How was your
experience

using the system?


3.

What was your
overall impression

of the Savvy system?


4.

What
features

of the site were vague or
confusing

to you, if any?


5.

Do you have any
suggested changes

or additional features in mind for
Savvy Languages?


6.

How would you describe this product in a couple of sentences?


7.

Does this product seem useful to you? Why?


8.

Is this a product you would use again? Why?


9.

Is this a product you would recommend to others? Why?





U01

U01 think that Savvy has good transparency.

U01 rated the system with a score of 85.
-
10 for there is no help when getting wrong
answers.
-
3 for unclear features like 5,000 XP.
-
2 for the
ambiguity about the path
metaphor.

U01 likes the virtual reality scene and the red dots.

U01 suggest incorporating feedback in the speaking lessons. “is this what you said?
[audio playback]”

U01 suggest the the voice can be converted into text, reducing th
e work of typing.

U01: How long does it take to get sufficient language skill for a purpose? How far can
this program take you?

U01: How much it will cost is a very important factor in choosing language service, will
it be cheaper than Rosetta Stone?


U02

U02: “Interactive is important for learning, the experience of searching for something”

U02: “Would be nice to click on the homepage pictures, and provide more categories”

U02: “I don’t know where the “Continue” button (in homepage) will take
me”

U02: “How am I connected to the leaderboard? Why is it important to me? Otherwise I
would not care about it.”

U02: “I assume the achievements are the lessons I passed”

U02: “People are impatient and would click around a lot”

U02: “Interactives makes me

get involved and not bored. It is repetitive in a another
way”

U02: Subjects are good, Idioms, Phrases, Holidays, cultural differences

U02: “Path organization makes sense, I should be able to skip it if I wanted”

U02: “If I were to go grocery shopping, I
should be able to skip to food”

U02: Activities pertaining to normal life, to use it daily.


U03

U03: “I don’t see back button”
-

To go back to homepage

U03: Keeps scrolling up to find a “Back button”

U03: “I don’t know how to get back”

U03:
Draggin banners is pretty.

U03: “What is the difference between My Path lessons and the banner ones?”

U03: After lesson ends, goes to my path, but originally wanted to go to “My Home”

U03: “Give me a way to see the right answers”

U03: Finds the hover fun
ction, but only later into the tasks.

U03: “It is nice to hear conversations again (clicking on Andy’s face), didn’t catch that in
the first lesson”

U03: “Distracted by pictures in the banner”

U03: “Most difficult thing was to find lessons in My Path

U03:

“Is there a way to get out from the lesson?”

U03: “My Path is good to keep track which ones are done”

U03: Did not see Recommended Lessons at first. Tends not to scroll down.

U03: Don’t know what the left bar is for.

U03: “Achievement categories don’t e
ven match the lessons”

U03: Exploring the interactions is engaging

U03: Badges (dots on people) “Weird yellow things”

U03: Group view


Supposed to translate but U03 was answering the questions

U03: “Are the dots part of the outfit?”

U03: Word vs s
entence conflict in directions


U04

U04: Jeans and Pants
-

same word? Picture of jeans = vocab ‘pantalones’

U04: microphone issue repeated
-

website should check for compatibility before lesson
(avoid microphone permissions issue)

U04: recording issue repeated

U04: Do you have to complete all lessons on whatever level to get to next level?
(unlocking?) (can you bypass lessons and get to the next level anyway?)

U04: Leaderboard
-

Can you send messages? Where is my inbox?

U04: Found c
onnect to Twitter, FB, what about Google Plus?

U04: Will banner change as lessons are completed? (suggestion) Show a lesson that is
not yet accessible
-

to unlock this lesson, complete lesson x.

U04: Clothing store 360 view
-

picture very busy, didn’t kno
w if she’d clicked all the
buttons

U04: Greetings lesson
-

wanted alternative answers
-

“I am doing poorly”

U04: Wants direct competition
-

showing levels and points is not going to be sufficient
to motivate her

U04: Mentioned desire to customize answer ty
pe
-

type it in or pick from drop
-
down

U04: fun, visually attractive

U04: likes chunks (small lessons)
-

can do on a 15 minute break, on a bus, etc

U04: would use again

U04: would recommend


U05

U05: “It is hard to find a free and
suitable to schedule, online (tool to learn)”

U05: Referring to other services, they are all “Test
-
based”

U05: “Hard to decide where to begin, I have no background, too many tests and
postings”

U05: About good aspects of Savvy
-

“Grading, test, audio, teac
hing pronunciation,
writing and reading, background to illustrate situation”

U05: My Path


“Steps, going through tree to develop skills. Shows how far top and bottom”

U05: “Tells your goals clearly”

U05: “Which lessons to take first”

U05: “Based on
interest, whichever I feel interested I can take”

U05: Regarding other systems
-

“Don’t know where to begin”

U05: “How many points I got in individual levels”

U05: “Only one path, if multiple paths, multiple branches. You can chose. I don’t have
as much ti
me.”

U05: “Compare trees between people”

U05: “The goal is the hardest lesson”

U05: Not sure if taking all lessons on same level is required to advance

U05: “Finishing a branch makes me proud”

U05: “Need to illustrate difficulties and requirements, somethi
ng to expect”

U05: “I don’t know what “verb 1” means, are topic and verbs related?”

U05: “Maybe separate them by trees”

U05: “For conversation, a simple background would be enough”

U05: Still does not understand skills and leaderboard

U05: “I finished but
did not get points or achievements, a little bit disappointing”

U05: “I should be the first one, look at my points!”

U05: “Compare to first and see the courses taken by the first one, reflect my process”

U05: “This week, month, overall, not always first”

U
05: “Path shows the progress but not clear”

U05: “Did not expect to see a tree”

U05: “Shows what I learned and what I could learn, which courses I can select next”

U05: “Provides path where I could go next, very clear”

U05: Finds continue button on homepag
e. “Did not recognize it before”

U05: “I had a good experience, different, others are text heavy not pretty and has
bunch of text”

U05: “My path shows clear organization of courses”

U05: “Good illustrate conversation”

U05: “360 not too important, can use a

single image with the same effect”

U05: “Talking to somebody without talking to anyone”

U05: “Hint not clear, after check, don’t know what to do next”

U05: “picture background too colorful, hard to find dot”

U05: “Provide a way to go back to previous page
, no way to go back”

U05: “More interesting if someone actually talking to me, even if I just follow script”

U05: “Proactive with friends, need conversation with actual people”

U05: “Picture has to be less distracting, makes me confuse of what to do next”

U05: “Provide steps and tell you where to go. Hint to go somewhere.”

U05: About learning curve

“Too many errors, too slow, if provide reason to use website”

U05: “If I make progress after short period, yes, I will recommend”



Appendix H: Consolidated User Comments

navigation



window within window:
U2,U3,U4



slider doesn’t link to lesson:
U1,U3



getting back from Path is unclear
-

look for ‘home’ or ‘back’ button:
U3,U4

visibility



recording icons:
U1, U4, U2, U5, U3



recommended skills doesn’t show on laptop screen(user didn’t notice that
they have to scroll down to find lessons)
U3



continue button in strange place
-

causes pause:
U4, U5



function of dots unclear
U3

user flow and flexibility



instead of “check,” enter.
U3, U4



conversation too fast
-

want to click boxes to repeat sentences
U4, U5



partial playback
-

a word not just the whole sentences
U4



realistic answer options
-

not just always answering “I am fine, and you?”
U4

instruction



360 view:

-

lack of instructi
ons
-

did not explore
U4, U2, U3



user’s role in conversation unclear:
-

tried to answer Megan & Andy instead of
enter what Andy said

U2,U3, U4



translate or answer:
U1, U2, U3



confused when nothing was expected of her (only dialogue displayed on
screen)
U4
, U5



help button:

U1, U2, U3, U5



did not figure out that there is hint when hovering at the words at first:
U5, U4



did not notice the dots on people are clickable
U3

terminology



meaning of “my path” unclear:
U1, U3
-

“my path” understood to be lessons
the u
ser has completed (past)

o

will new lessons be unlocked as more basic ones are finished? the
lines connecting lessons imply this
U4



“Achievements” unclear:
U1, U3

real
-
world mapping



experience points/progress
-

math doesn’t make sense:
U4, U5



pictures shoul
d correspond with conversation:
U2, U3



jean or pants
-

same image can be describe in different ways
U4

feedback



hint on wrong answer/want feedback about what is wrong:

U1, U2, U3, U4

o

lots

of errors are frustrating when you can’t figure out what you did
wrong
U4



reinforcement on right answer:
U1