Natural Language processing and AAC: Current advances at the interface between technology and communication. technology and communication.

Natural Language processing and AAC:
Current advances at the interface between
technologyandcommunication.
technology

and

communication.
Annalu Waller and Alan McGregor
Uii f Dd Sld
U
n
i
vers
i
ty
 
o
f D
un
d
ee,
 S
cot
l
an
d
Brian Roark and Melanie Fried‐Oken
Oregon
 
䡥慬瑨 and Science
 
啮楶敲獩瑹U USA
Kathy McCoy
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 1
University of Delaware, USA
Session Outline
•Who are we?
•
IntroductiontoNaturalLanguageProcessing
•
Introduction

to

Natural

Language

Processing
•What does NLP mean to us?
•
ExamplesofNLPinAAC
•
Examples

of

NLP

in

AAC
•Implications of NLP on AAC
ItdiSLPAT
•
I
n
t
ro
d
uc
i
ng
SLPAT
•Q&A
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 2
Who are we?
•Annalu Waller :
University of Dundee, Scotland
•
Brian
Roark
:
OregonHealthandScienceUniversity
•
Brian

Roark

:

Oregon

Health

and

Science

University
,
USA
•Melanie Frie
d
-Oken:
Ore
g
on Health and Science
g
University, USA
•Kathy McCoy :
University of Delaware, USA
•Alan McGregor :
University of Dundee, Scotland
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 3
AAC Today…
Speech Generating Devices (SGDs) have great 
p
otential for those with severe s
p
eech
 
慮a
/
潲
 
浯瑯m
 
pp
/
業灡楲浥湴献
䉵B 
嬱[
 
䉵B 
嬱[
…
 
AAC devices remain difficult to use

䍩ti t i bl % f 

C
潭浵o
i
捡
瑩
潮
 
牡
t
敳
 
牥浡
i
n
 b
e
l
ow
 
㄰
% 
o
f 
湡瑵牡n speech 
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 4
[1] Beukelman, D.R., Mirenda, P. (2005). Augmentative and Alternative Communication: Management of
Severe Communication Disorders in Children and Adults
. Baltimore, Paul H. Brookes Publishing Co.
WhiNlL
Wh
at
i
s
N
atura
l

L
anguage
Processing?
Processing?
•Field of computer science focused on giving
co
m
pute
r
s

t
h
e

ab
ili
ty

to

p
r
ocess
h
u
m
a
n l
a
n
guage
coputesteabtytopocessuaaguage
•Dream: if we can build language processing
abilities into AAC devices
,
the
y
ma
y
be easier
,

,yy,
more efficient, more satisfying
•Natural Language Processing is about knowledge
about language and about algorithms for using that
knowledge
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 5
Knowledge needed to understand
ddl
an
d
pro
d
uce
l
anguage
•
Phonetics

and

phonology:
h
ow

wo
r
ds

a
r
e
r
e
l
ated

to

sou
n
ds

t
h
at

Phonetics

and

phonology:
how

words

are

related

to

sounds

that

realize them; what sounds can go together to make words
•Morphology:how words are constructed from more basic meaning
units
units
•Syntax:how words can be put together to form correct utterances
•Lexical semantics:what words mean
•Compositional semantics: how word meanings combine to form
larger meanings
•
P
ra
g
matics:how situation affects inter
p
retation of utterance
g
p
•Discourse structure:how preceding utterances affects processing of
next utterance
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 6
NlLPi
N
atura
l

L
anguage
P
rocess
i
ng
Methods
Methods
•Hand-written “rules” capture allowable
seque
n
ces/co
m
b
in
at
i
o
n
s
sequeces/cobatos
•Statistical (machine learning) methods made
p
ossible b
y
lar
g
e cor
p
ora and electronic resources
pygp
–N-gram models
–Language models
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 7
WhiNlL
Wh
at
i
s
N
atura
l

L
anguage
Processing?
Processing?
•Given some human language input (speech, text),
syste
m
p
r
oduces

so
m
e

output

o
f
use
systepoducessoeoutputo
use
•Exampleapplications widely in use:
•MachineTranslation
(
text-to-text
)
(
)
•Automatic speech recognition(speech-to-text)
•Optical Character recognition (image-to-text)
•Information extraction & data mining from text
•Well-known systems: Google translate;Siri; Watson
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 8
Uses of NLP
•Given my 14-year-old’s text messages:
w
nt
2
evan
s
hs
4
dnnr
cul8r
w
nt
2

evan
s
hs
4

dnnr
cul8r
•Can we “translate” this; or “recognize” this?
•
IfwecantranslatefromChinesetoEnglish
•
If

we

can

translate

from

Chinese

to

English
,
maybe we can automatically translate this
•
Ifwecanrecognizespokenlanguagemaybe
•
If

we

can

recognize

spoken

language
,
maybe

we can recognize this type of input
(Infact
lotsofphonologicalcontent)
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 9
(In

fact
,
lots

of

phonological

content)
AutomaticNLP systems
•Most systems based on statistics of actual use
•
Trainingdata:textwithtranslations;transcribed
•
Training

data:

text

with

translations;

transcribed

speech
•
Systemsmake
“
bestguess
”
ofstatisticalmodels
Systems

make

best

guess

of

statistical

models
•Often very useful, but can be wrong
•
ThinkofGoogletranslateor
Siri
•
Think

of

Google

translate

or

Siri
•Systems can be very complex but still seem dumb
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 10
WhatdoesNaturalLanguage
What

does

Natural

Language

Processin
g
mean to us?
g
•The rehabilitation engineer: A way to make
AAC more intuitive and natural to use
•The computer scientist: The means to make
intelligent AAC systems –reducing some of the
load from the user and enabling better
communication.
•The computational linguist: Links to other kinds
of human language technology; AAC offers new
applicationsrequiringnovelNLPtechniques
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 11
applications

requiring

novel

NLP

techniques
.
WhatdoesNaturalLanguage
What

does

Natural

Language

Processing mean to us?
•The speech-language pathologist: Language
supportduringassessment,treatment,anduseof
support

during

assessment,

treatment,

and

use

of

SGDs for people with CCN
•The AAC user: Allows the device to s
p
eed u
p
the
pp
communication for the AAC user. Word
prediction allows conversation to be faster so I can
p
articipate more. It would be really nice for it to
be able to know what I am thinking.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 12
ElfNlL
E
ar
l
y uses o
f

N
atura
l

L
anguage
ProcessinginAAC
Processing

in

AAC
•Word prediction –syntactic knowledge, n-gram
m
ode
lin
g
odeg
•Semantic phrase retrieval
•
Expandingtelegraphicinput
–
syntaxlexical
Expanding

telegraphic

input

syntax
,
lexical

semantics
•
Utterance
-
basedsystems
–
typicalconversational
Utterance
based

systems

typical

conversational

patterns, script/goals etc…
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 13
UiifDlM
U
n
i
vers
i
ty o
f

D
e
l
aware:
M
ore
IntelligentWordPredication
Intelligent

Word

Predication
•Word Predication is used a lot in current AAC
app
li
cat
i
o
n
s,

but
i
t
h
as

gotte
n
a

bad
n
a
m
e
appcatos,buttasgotteabadae
–Interface issues
–Need to specify/switch word predication dictionaries
–Context switching required
•Exasperated by inappropriate predictions
Uifllldiil
•
U
se a var
i
ety o
f
rea
ll
y
l
arge corpora an
d
stat
i
st
i
ca
l

language processing
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 14
Prediction Exam
p
le
p
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 15
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 16
Can we do better??
•Intuitively all possible words do not occur with
equa
l lik
e
l
y
h
ood
du
rin
g

a

co
n
ve
r
sat
i
o
n
.
equa
eyood
dugacovesato.
•The topic of the conversation affects the words
that will occur.
–E.g., when talking about baseball: ball, bases, pitcher,
bat, triple….
–
How often do these same words occur in your second
language acquisition class?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 17
Topic Modeling Approach
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 18
Topic Identification
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 19
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 20
Topic Similarity Scores for Above
Cti
C
onversa
ti
on
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 21
Topic Application
•How do we use those similarity scores?
•
Essentiallyweightthecontributionofeachtopic
•
Essentially

weight

the

contribution

of

each

topic

by the amount of similarity that topic has with the
current conversation.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 22
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 23
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 24
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 25
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 26
STlkdU
S
cene
T
a
lk
er an
d

U
tterance-
BasedSystem:Pragmatics
Based

System:

Pragmatics
•Joint work with University of Delaware, Jan
B
ed
r
os
i
a
n
(Weste
rn Mi
c
hi
ga
n
U
ni
ve
r
s
i
ty),
Lin
da

edosa(WestecgaUvesty),da
Hoag (Kansas State University)
•S
y
stem Recommendations based on findin
g
s from
yg
a series of experiments looking at pragmatic
mismatches between prestored utterances and
current discourse situation
•Looked at attitudes of an unfamiliar
ilil
did
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 27
conversat
i
ona
l
partner
i
n a goa
l
-
di
recte
d

transactional situation
Example Screen
Clear
4
4
5
5
6
6
7
7
1
1
2
2
3
3
Nachosplease.
appetizer
soup/salad
entree
dessert
4
4
5
5
6
6
7
7
1
1
2
2
3
3
I ‘d like an
I ‘d like an
appetizer, appetizer,
pleaseplease
What would
What would
you you
recommendrecommend
I’d like a
I’d like a
salad, salad,
pleaseplease
What’s your
What’s your
soup of the soup of the
day?day?
What are
What are
today’s today’s
specials?specials?
I‘d like
I‘d like
dessert, dessert,
pleaseplease
No
No pepper
pepper,
,
pleaseplease
?
What’s
What’s in in
that?that?
Nachos
Nachos
pleaseplease
Garden
Garden
salad salad
PleasePlease
I’ll
I’ll have that have that
I’ll have that
I’ll have that
Crème
Crème
bruleebrulee
What are
What are
my my
choices?choices?
We’re going
We’re going
to split to split thatthat
OK we’ll
OK we’ll
have thathave that
Bleu cheese
Bleu cheese
on the sideon the side
No thanks
No thanks
Fries
Fries
What would
What would
you you
recommendrecommend
I’ll
I’ll order order
from the from the
menumenu
Extra
Extra
cheesecheese
No thanks
No thanks
No onion
No onion
Clam
Clam
chowder chowder
pleaseplease
I’ll have
I’ll have thatthat
The steak
The steak
medium medium
rarerare
assistance
quickfires
waiter
smalltalk
AAC Research at Dundee
The aim of our research group is to utilise NLPto support language and 
conversational development by:
Ebli l l 
橫 hi
•
E
na
bli
ng
 l
anguage
 
p
l
慹
 
–
j
o
k
敳e
 
p
h
潮
i
捳
•Providing automatic access to  appropriate vocabulary
•
Scaffolding communication skills
Scaffolding communication skills
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 29
Examples of NLP & AAC
•STANDUP ‐Enabling language play
•How was School Today? –narrative generation
•Sound  Prediction
 
–
䥳捡I
•etc
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 30
STANDUP: Pun Generation –Rolf Black Dr Graeme Ritchie, Aberdeen
Dr Helen Pain, Edinburgh
What do you call a 
What do you call a 
spicy missile?
A hot shot
A hot shot
.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 31
Language Exploration: Joke Generation
Whdll
synonym
k
?
Wh
at
d
o you ca
ll
a
strangemar
k
et
?
Abizarrebazaar!
homophone
synonym
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 32
The SCAN Project –Ha Trinh PhDshi
p
funded b
y
 SICSA
/
Dundee Universit
y
•Aims to develop a sound‐based communication aid for 
n
o
n
spea
kin
g
 
peop
l
e
 wh
o
 
e
x
pe
ri
e
n
ce
 li
te
r
acy
 
d
iffi
cu
l
t
i
es
p
y/y
ospeagpeopeoepeeceteacydcutes
•Using a sound‐to‐speech approach to enable users to 
generate novel words and sentences in interactive 
generate novel words and sentences in interactive 
conversation
•
Employing 
NLP 
瑯 improve the usability and accessibility 
•
䕭灬潹楮E 
NLP 
瑯 improve the usability and accessibility 
潦 the proposed system
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 33
How was School Today? -Rolf BlackDigital Economy (RCUK) Funded; 
“Storytelling puts an equal emphasis 
on the feelings and audience 
Partners: Aberdeen, CS, Perth Education
on the feelings and audience 
楮癯汶敭敮i as on the structure of the 
story; and regards narrative 
develo
p
ment as a social 
p
rocess which
 
pp
扥杩湳 in infancy and is scaffoldedby 
adults” [Grove, 2009]
The “How was School toda
y
...?”
 
y
灲潶楤敳 the child with automatically 
generated narrative utterances based 
on sensor and other data.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 34
Photos:Prototype user testing at Corseford School (left) 
and interface screenshot.
Interaction Data
RFID sensors track the child’s 
interactions with:
•teaching and other staff;
dfid
•peers an
d
 
f
r
i
en
d
s;
•objects such as teaching tools.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 35
Location Data
Hall
Classroom
Sensors on doorways detect the 
location of the child.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 36
User Modelling
The timetable provides 
information about time, 
activity, interaction and 
location.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 37
Additional Data
Voice recordings can be added 
to the database to provide 
additional information that
 
捡湮潴 be detected by sensors.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 38
Generating Narratives
The system then generates a 
narrative with several events. 
E t it f  b 
E
very
 
敶en
t 
捯湳
i
s
t
s
 
o
f 
a
 
湵n
b
敲
 
潦 messages.
This supports interactive story‐
tlli  d t  
t
e
lli
ng
 
慳
 
潰灯獥
d t
o
 
a
 
浯湯汯杵m  story.
Event: Music Class
2 messages generated
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 39
Brain Computer Interface Project
•Rapid Serial Visual Presentation (RSVP™)
–
Linearscanning,switchactivationviaP300ERP
Linear

scanning,

switch

activation

via

P300

ERP
–Important role for the language model
•Use in ERP detection, similar to use in speech recognition
•Use to decide sequencing (scan likely symbols earlier)
•Use to decide to stop scanning earlier, speeding text entry
•Use for word com
p
letion and
p
rediction
pp
•Use for graceful recovery from spelling errors
•As with other scanning methods, speed is needed
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 40
–Language model is providing some of that
Implications of NLP on AAC?
•Clinical Perspective
•
EndUserPerspective
•
End

User

Perspective
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 41
IliifNLPAAC
I
mp
li
cat
i
ons o
f

NLP
on
AAC
:
ClinicalPerspective
Clinical

Perspective
Assessment challenges:
•
CanNLPassisttheclinicianevaluate(speech
•
Can

NLP

assist

the

clinician

evaluate

(speech
,
language and writing) communication skills in
b
oth testin
g
and natural settin
g
s?
gg
•Can NLP assist the clinician to identify an end
user’s language level? There are quantitative and
qualitative changes in language behaviorsover
time. Yet SLPs are not aware of these changes
hliSGD
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 42
w
h
en eva
l
uat
i
ng
SGD
use.
•Changes in representation levels: As a child
becomes literate, she moves from picture or
blbdttthhCNLP
sym
b
o
l

b
ase
d
sys
t
ems
t
o or
th
ograp
h
y.
C
an
NLP

assist with assessment of a child’s literacy level
overtime
andmakeadjustmentstotheSGDs?
over

time

and

make

adjustments

to

the

SGDs?
•The adult with language impairments secondary to
TBIoraphasiaisregainingskills;canNLPassist
TBI

or

aphasia

is

regaining

skills;

can

NLP

assist

with the identification of these changes and adjust
language presentation in SGDs?
•Can NLP predict if an adult is experiencing early
degenerative language disease?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 43
TreatmentChallengesfromthe
Treatment

Challenges

from

the

SLP Pers
p
ective:
p
•SGD use requires considerable working memory.
Can NLP li
g
hten the co
g
nitive load?
gg
•When the user moves from word generation to
word prediction, she must change her cognitive
set. Can NLP help with attentionaldemands of
prediction/completion during message generation?
•Organization of vocabulary and creation of an
optimized layout is still unstandardized. Can NLP
ktkfhlithit
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 44
k
eep
t
rac
k
o
f

h
ow peop
l
e are us
i
ng
th
e
i
r sys
t
em,
and optimize layout and organization?
•
Rightnow,ausermustlearnlanguage,concept
Right

now,

a

user

must

learn

language,

concept

organization and production, as well as device
operations and strategic demands. Can NLP help
with operation, strategic and linguistic competence
for interaction?
•Can NLP create rules for knowledge organization
based on context: user’s linguistic skills; partners’
killdlltid
s
kill
s an
d

l
anguage use;
l
oca
ti
on an
d

environmental cues for message generation and
interaction
interaction
.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 45
IliifNLPAAC
I
mp
li
cat
i
ons o
f

NLP
on
AAC
:
EndUserPerspective
End

User

Perspective
•Include AAC users in entire development process
–
des
i
g
n
,

p
l
a
nnin
g,

eva
l
uat
i
o
n
desg,pag,evauato
•Developers and users might have different
p
ur
p
oses
pp
•Important to me
–
Accessibilit
y
with limited dexterit
y
yy
–Accessibility in ALL conditions
•My Dream: read my brain and translate my
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 46
thoughts into speech!
Introducing SLPAT
•Speech and Language Processing for Assistive
T
ec
hn
o
l
og
i
es

(S
LPAT
)

www.s
l
pat.o
r
g
ecooges(S)
www.spat.og
•Three workshops over past three years
•
LA2010;Edinburgh2011;Montreal2012
LA

2010;

Edinburgh

2011;

Montreal

2012
•Another planned for 2013 (at Interspeech)
•
Forthcomingspecialissue
•
Forthcoming

special

issue
•Recently formed SIG of ACL
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 47
Introducing SLPAT
•Range of topics in workshops and special issue
•
Generationofutterancesandnarrativesfromdata
Generation

of

utterances

and

narratives

from

data
•Scanning systems
•Processing of dysarthricspeech
•Text simplification
•Word-, utterance-or phoneme-based predictive
systems
•Communication and Alzheimer’s disease
•
Signlanguagerecognitionand
translation
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 48
•
Sign

language

recognition

and

translation
Questions/Comments?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 49