AAC for Elderly People with Hearing, Speech or Memory Loss due to Aging

matchmoaningAI and Robotics

Nov 17, 2013 (3 years and 8 months ago)

69 views

AAC for Elderly People with Hearing,
Speech or Memory Loss due to
Aging

Lawrence Chung, Shinyoung Lim,

Yoosun Chung, Rutvij Y. Mehta, Amit B. Chembra


25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

1

© 2010

Acknowledgements



‘SymbolStix Inc.’



Students that participated in the prototype implementation


Abdullah Azzouni, Bharat Paturi, Manoj Kumar, Michael Fashola, Nisha Mathew,
Rakesh Sanapala, Rohit Sharma, Sandeep Sistla , Sritej Vepa, Vamshi Bandaru,
Vidya Vijayan , Vinay Kumar



For their valuable comments & feedback (Research group)


Ebenezer Oladimeji, Sam Supakkul and Tom Hill (a special thanks for the Nexus!)
















25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

2

© 2010


Outline


Introduction


Goal


Motivation


User Needs of AAC Service


Design of a prototype system


Our models


Olive* , H.O.P.E


Features and Demo


Enhancements


Project in a Nutshell


Conclusion

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

3

© 2010

Introduction


‘Science Blog’


Reduced Hearing more common
and hearing aids inadequate for elderly


AAC focused on Autism,
rarely

on elderly people


Treatment focused on medical/surgical procedure
not rehabilitative
approaches


Communication barriers of elderly not researched
and existing measures not tailored to their needs


Current technology


No concept exploration


Population of the elderly to be
significant

soon


Need for an assistive system ever more than before

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

4

© 2010

Goal




Identify


Communication difficulty is a problem


Understand user
needs

for an assistive system


Eradicate the need for
bulky
, costly instruments


Develop a communication system to overcome


Hearing loss, Memory loss, Speech Disorder,
Weak Vision


Use existing hardware (e.g. Cell phones)
-

cut cost


System that can be ‘personalized’ for user needs


A ‘mobile’, ‘assistive’ and ‘cost
-
effective’ system
that helps reduce the user’s communication
barriers

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

5

A
S
-
I
S

T
O
-
B
E

© 2010

Communication difficulty
-

A problem ?


Elderly are concerned over perception of others


Lack of communication decreases their
self reliance


Due to communication barriers, they limit their interactions


Lose confidence and remain confined


Fear getting embarrassed, so do not communicate


Give up their interests and live a neglected life


Psychological

problems may arise


Problems and Behavior can be identified in day to day events
or scenarios


Such problems are
preventable



just have to fill the gaps





25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

6

© 2010

Motivation (The Why question)


Scenario (Family Dinner Outing)


-

Elderly

parents


-

Loss of hearing, Speech impairment and Memory loss


-

Not ready to
publicly

accept

their age
-
related disorders


-

Do not welcome use of external medical aids


-

Human perception, memory and communication are


visual not textual


-

We function best when all senses
complement

each other


-

Similar scenarios repeat in other forms every day



25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

7

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Son

Old parents

Son
: Mom and Dad, Get
Ready. We are going to
the Chinese restaurant
we went last time for a
family get together.


Parents
:
Sure

Entire family gets in the
car and leave for the
restaurant

8


A visual scenario (Family Dinner Outing)

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Mom
: Son, where are
we going ?


Son
:
I told you


the
Chinese restaurant



In the car


On the way

9

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Mom
: Son, where are
we going ?


Son
:
(In an irritated
tone)
I have already told
you


the Chinese
restaurant !!





In the car


On the way ( A few miles later …)

Son
:
(Realizing)
Mom


Are you wearing your
hearing aid ?


Mom
:
(Does not have it
on)
Hmm.. Yes..
Ah..Here I have it….






LOSS OF HEARING !

10

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Mom
:
(To Dad)


If
Steve comes and
bows, then
acknowledge him. Last
time he met, you did
not talk nor gave a
smile to him


Dad
:
(STEVE ??? Has a
clueless expression)
OK





In the car


( Almost near the restaurant …)

MEMORY
LOSS !!

11

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Mom
:
(To Dad)


Why
did you not greet
Steve ? I told you to
greet him !!!


Dad
:
(Embarrassed)

I
could not recognize
him !





Outside the restaurant …

MEMORY
LOSS !!

Steve walks and
greets Dad.

Dad fails to recognize
him !!!

Steve


at
present

Mom
:
You remember
Steve


our daughter’s
son ! Remember how
cute he was when he
was a small boy !


Dad
:
(Apparently still
clueless


trying to
recollect)
Ah..OK






Steve as a boy

12

© 2010

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

Grandson
: Dad


What
did Grandma say? I
cannot understand
what she murmurs….


Son
:
Mom


Are you
wearing your
dentures?





Inside the restaurant …

Unclear/Incomprehensible

Speech !!

Mom instructs her
grandson to eat
carefully without
spilling

Grandson

13

© 2010

User Needs of AAC Service



Categories of Difficulties/Communication Barriers for Elderly



-

Unclear

Utterance

, often leading the elderly to repeat and


hard for the listeners to comprehend



-

Incomplete/Incomprehensible speech
: Speech often times


nonsensical, difficult to express thoughts



-

Listening Inability
: Hard of hearing especially mobile


communications (Hearing devices


voice
-
to
-
text


translation could be introduced)



25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

14

© 2010

User Needs of AAC Service



-

Comprehension Inability
: Cannot grasp or understand what


others have to say (Cognitive impairment)



-

Communication issues due to memory loss
: Cannot


remember names of people, places, activities, topic



-

Communication in emergency
: Should be able to make an


emergency call (say, 911), especially those having medical


problems or living alone (Voice to text/Voice to image relay)

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

15

© 2010

Design of the prototype system



Target 3 language acquisition mechanisms by icons/photos



1) See what you cannot hear


-

Overcome loss of hearing



2) Show what you cannot speak


-

Overcome unclear speech (Text
-
to
-
speech)



3) See what you cannot remember


-

Overcome memory loss







25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

16

© 2010

Design of the prototype system


Key technical concepts of the system:


-

Vocabulary

: To show visual images, for words heard and to


to be spoken, transformation to address complex scenarios



-

Navigation
: Identify the scenario/context and allow quick


and easy navigation based on activities of daily living



-

Learning
: Detecting and building patterns of frequent


vocabulary use in daily living


System adaptability.

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

17

© 2010

Design of the prototype system


Development Focus:


Ontology
: Philosophical study of existence (source: Wikipedia)


Things, relationships and constraints on them


Focus towards ontology
-
driven context awareness in User
Interface and services


Vocabulary : 5 basic entities


TEXT, ICONS, SOUND


SPEECH & OBJECT RECOGNITION


Target the important sensory activities of SPEECH,


HEARING, VISION and THOUGHT process


All
activities

available to the user based on the 5 entities



25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

18

© 2010

OLIVE


The first attempt

O.L.I.V.E


Older Living Independently


Started as a independent study project


AAC research for the elderly


first step


Implementation on
Samsung Instinct
Phone


Work on
‘Search’
and

‘Teach’
functions

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

19

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Research ongoing since 2009


Built on
Android

Platform


G1, Nexus Phone


Increased the vocabulary from


Search

and

Teach

to the


5 ontological primitives

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

20

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Main categories


-

Greetings


-

Daily Living Activities


-

Emergency situations


-

Talk (Build sentences)


-

Real World Scenarios


-

Active Mind (Games)

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

21

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Menu categories


Speech recognition used in


-

Tagging

images (assigning a


name to your images for a photo


album


to be extended for all


images in the system)


-

Retrieving

an image based on


voice/text input

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

22

© 2010

HOPE

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

23

Search


Greetings

© 2010

HOPE

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

24

Emergency


Daily Living Activities

Playback Feature

© 2010

HOPE

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

25

All by Yourself ?


Nothing to talk ?


No problem…..


Special added feature :


Active Mind (Memory games)

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Prominent Points:


-

Not feature oriented but
scenario

oriented


-

Use
contextual

mechanism to minimize


number of clicks


-

Speech recognition to make it not only AAC


but also
assistive


-

Activities revolve around 5 vocabulary entities


-

Add on game for Active Mind, can be used when not


actively communicating



25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

26

© 2010

HOPE (Demo)

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

27

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Enhancements:


-

Add more scenarios to reduce
effort

of the end user in


creating statements


-

Extend the tag/recognize feature to associate a name with


any image of the system instead of the photo album


-

Add more user configuration options like zooming of icons


-

Add more
vocabulary

primitives along the 5 organizational


entities


-

Cater application to a larger target user population

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

28

© 2010

HOPE


H.O.P.E


Helping Older People Easily


Enhancements:


-

Explore the ‘Contextual’ improvement


-

Use the ‘minimum clicks’ approach


-

A sample contextual mechanism for Daily Living Activities


inspired from the vocabulary of the
Maya

civilization


Revolves around the activity with only relevant contextual


icons surrounding it


Final sentence is the combination of the activity and the


context

icons e.g. Take Bath + ( Cold or Hot Water)

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

29

© 2010

Project in a Nutshell



Feasibility Study involved identifying communication


barriers and user needs



Developed service requirements and design concepts for


our prototypical system


Still difficult to draw solid tangible requirements from user
needs due to complexity and individuality in communication


barriers


On going project incremental


evolutionary

and
agile

methodology


Many issues


what to represent, how to represent in the


best effective manner.


Growth depends on more learning, discovery


25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

30

© 2010

Conclusion


Population of elderly to grow


so is the need and urgency


for countering communication difficulties


Issue has been studied, user needs and concerns identified


Design of the system based on 5 types of vocabulary


Tiny AAC software implemented


H.O.P.E


More investigation on contextual vocabulary and
representing core vocabulary


Develop a ‘minimum clicks


maximum vocabulary’
approach


Extend system by adding features for a larger audience


Make a change


Bridge the gap due to communication
barriers

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

31

© 2010

Questions ?

25
th

Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010

32

© 2010

Contact:

Rutvij

Mehta

Email : rym071000@utdallas.edu