Title: Evaluating a Video Enhanced Virtual Reality ... - Do2Learn

quartercircleAI and Robotics

Nov 14, 2013 (4 years and 6 months ago)


Video Enhanced VR for
Teaching Restaurant

Skills to

Children with Autism


While Virtual reality has been used for various training applications, few
studies have measured its effectiveness in teaching social interactions. This
research used

web delivered gaming

technology to create

worlds where children with Autism Spectrum Disorder



with avatars

restaurant social skills

of similar real world situations

within the virtual sequences

reinforced the lessons and aided in generalization
a known
problem for individuals with ASD. Before training, two unknown
restaurant social skills were identified for each of five children with ASD,
aged 7 to 16.
All children who completed VR training correctly
performed two new social skills in a virtual restaurant. In post
training real world restaurants, these children exhibited at least
one new appropriate social interaction and 75% demonstrated
2 new social skills.

Learning Steps for Autism

Define a common social skill

eating in a restaurant

Divide social skill into
discreet steps

(1) Wait to be seated, (2) Answer hostess questions, (3) Follow hostess to table, (4) Wait
for server and read menu, (4) Order food, (5) Table conversation

Define measurable

actions in each step

Ex: Step 1 includes determining if there is a hostess or you seat yourself, waiting in the
appropriate place, recognizing the hostess, and waiting for others already in line

behind each step

Theory of Mind

Show a range of appropriate

for each step

Use non

of steps with avatars in VR


of real people in same step within VR lesson

Reward or correct

at each step based on child’s actions

Vary scene/avatar actions to
teach variations

user control

of steps for different learning abilities

Vary avatar responses to
avoid patterns

and teach concepts

Program Design

Text with video explanation of each step

What to do (Question/How to Respond)

Explain the social action or exchange

Explain the meaning behind the actions

See it!
See a video of real people doing the step

Oops… What to do if something goes wrong

Show possible variations of ‘What to do’

Virtual world practice in pizza restaurant

Three windows on screen:

Interactive VR world with avatars

Question/answer and video/correction window

Displays responses child can pick

If child picks correct response, shows video

If child picks wrong response, corrects/repeats step

What to do/Why? Guidance

player practice with other people over web

Program Details

Player moves in virtual worlds with mouse or keyboard directional keys

Avatar program tracks and verbally responds to different actions of the player

Avatars in virtual world include player, a friend the player is eating with, a hostess, a
waitress, and other patrons at different tables

Visual cues. Example: Colors visually separate window functions; Error/reward
statements in red, questions with choices in blue, inactive window in black

Videos at each step are repeatable with button control

Avatar spoken words have both text displayed at window bottom and audio

Print out of step information available for guidance if child prefers hard copy

Text with video explanation program was developed with Flash

VR practice world was created and delivered over web with Wild Tangent Platform

player VR practice was created and delivered over web with Adobe Atmosphere

VR actions are controlled with JAVA programs

Graphics used Wild Tangent world creation toolkit, Atmosphere world creation toolkit,
3d StudioMax, Poser, and Photoshop

Video production was done in a real a restaurant during lunch hour. Waitress, hostess,
and friend were scripted actors. The regular restaurant clientele was part of the filming
background. Fourteen action and seventeen avatar phrase variations were used

Video processing was done with Premier

Study Design

Six children ages 7 to 16 with ASD chosen

All in special education programs and recommended by therapists.

Child 01(Age 11,
Male, Asian)
Child 02(Age 7, Male, White)
Child 03(Age 9, Female, White)
04(Age 16, Male, Black)
Child 05(Age 9, Male, White)
Child 06(12, Female, Black).

training testing in real restaurants on six skill steps

Individual testing was done in multiple lunch or dinner visits. Restaurants varied and
depended on parent recommendation. Interactions were not scripted. Actions and
latency were recorded by 2 observers. Correct response required that 90% of all parts of
a step be done correctly.

Child had to display inappropriate behavior at more than
chance level for 2 or more steps to be included in study.

Five children practiced correct social actions on computer

Training was approximately one hour, two to three times over a 2 week period. Training
first involved a treasure hunt game to practice navigation in virtual space. All children
mastered control in no more than two plays. Each child then completed the Text, VR
Pizza world, and web training programs. After training, each child assessed his
knowledge with a written questionnaire. All children came back approximately one
week later and repeated the three training programs and questionnaire.

training testing in real restaurants on six skills

training testing was repeated in the same restaurants and data collected as before.
Appropriate skill responses varied between visits because hostess, waitress, patrons, and
other details varied, creating different required social interactions. Two restaurant visits
were done per child with an average of 3
day separation between visits.


4 children completed study

Child 03 knew all skills in pre
tests. Child 06 was removed because of family conflicts

All learned 2 new skills in virtual worlds at 100% accuracy

Four subjects mastered all 6 skill steps to 100% accuracy on all computer training

All did 1 new skill in real world with 90% accuracy

75% did 2 new skills in real world with 90% accuracy

Skill retention was strongest when the restaurant visit was close in time frame to
computer practice. Child 05 did a third computer training between restaurant visits after
missing Step 1 in first post
training and subsequently did all skills correctly

Mixed results for one child who generalized only 1 skill

Child 04 was severely autistic (CARS 34) and missed all skills in pre
training. In first
training, he demonstrated 100% accuracy in 5 of 6 steps. In second restaurant visit,
music and the unexpected presence of teacher at next table distracted child. Computer
refresher and retest could not be completed before child moved to another state


Real world actions did not always match discreet step divisions. Example: child might
order correctly but stare at ceiling rather than waitress (was given 50% accuracy).
Observers required that each action be completely acceptable by normal social standards

Graphs of results and details of pilot study available from