Natural Language processing and AAC:
Current advances at the interface between
technologyandcommunication.
technology
and
communication.
Annalu Waller and Alan McGregor
Uii f Dd Sld
U
n
i
vers
i
ty
o
f D
un
d
ee,
S
cot
l
an
d
Brian Roark and Melanie Fried‐Oken
Oregon
䡥慬瑨 and Science
啮楶敲獩瑹U USA
Kathy McCoy
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 1
University of Delaware, USA
Session Outline
•Who are we?
•
IntroductiontoNaturalLanguageProcessing
•
Introduction
to
Natural
Language
Processing
•What does NLP mean to us?
•
ExamplesofNLPinAAC
•
Examples
of
NLP
in
AAC
•Implications of NLP on AAC
ItdiSLPAT
•
I
n
t
ro
d
uc
i
ng
SLPAT
•Q&A
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 2
Who are we?
•Annalu Waller :
University of Dundee, Scotland
•
Brian
Roark
:
OregonHealthandScienceUniversity
•
Brian
Roark
:
Oregon
Health
and
Science
University
,
USA
•Melanie Frie
d
-Oken:
Ore
g
on Health and Science
g
University, USA
•Kathy McCoy :
University of Delaware, USA
•Alan McGregor :
University of Dundee, Scotland
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 3
AAC Today…
Speech Generating Devices (SGDs) have great
p
otential for those with severe s
p
eech
慮a
/
潲
浯瑯m
pp
/
業灡楲浥湴献
䉵B
嬱[
䉵B
嬱[
AAC devices remain difficult to use
䍩ti t i bl % f
C
潭浵o
i
捡
瑩
潮
牡
t
敳
牥浡
i
n
b
e
l
ow
%
o
f
湡瑵牡n speech
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 4
[1] Beukelman, D.R., Mirenda, P. (2005). Augmentative and Alternative Communication: Management of
Severe Communication Disorders in Children and Adults
. Baltimore, Paul H. Brookes Publishing Co.
WhiNlL
Wh
at
i
s
N
atura
l
L
anguage
Processing?
Processing?
•Field of computer science focused on giving
co
m
pute
r
s
t
h
e
ab
ili
ty
to
p
r
ocess
h
u
m
a
n l
a
n
guage
coputesteabtytopocessuaaguage
•Dream: if we can build language processing
abilities into AAC devices
,
the
y
ma
y
be easier
,
,yy,
more efficient, more satisfying
•Natural Language Processing is about knowledge
about language and about algorithms for using that
knowledge
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 5
Knowledge needed to understand
ddl
an
d
pro
d
uce
l
anguage
•
Phonetics
and
phonology:
h
ow
wo
r
ds
a
r
e
r
e
l
ated
to
sou
n
ds
t
h
at
Phonetics
and
phonology:
how
words
are
related
to
sounds
that
realize them; what sounds can go together to make words
•Morphology:how words are constructed from more basic meaning
units
units
•Syntax:how words can be put together to form correct utterances
•Lexical semantics:what words mean
•Compositional semantics: how word meanings combine to form
larger meanings
•
P
ra
g
matics:how situation affects inter
p
retation of utterance
g
p
•Discourse structure:how preceding utterances affects processing of
next utterance
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 6
NlLPi
N
atura
l
L
anguage
P
rocess
i
ng
Methods
Methods
•Hand-written “rules” capture allowable
seque
n
ces/co
m
b
in
at
i
o
n
s
sequeces/cobatos
•Statistical (machine learning) methods made
p
ossible b
y
lar
g
e cor
p
ora and electronic resources
pygp
–N-gram models
–Language models
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 7
WhiNlL
Wh
at
i
s
N
atura
l
L
anguage
Processing?
Processing?
•Given some human language input (speech, text),
syste
m
p
r
oduces
so
m
e
output
o
f
use
systepoducessoeoutputo
use
•Exampleapplications widely in use:
•MachineTranslation
(
text-to-text
)
(
)
•Automatic speech recognition(speech-to-text)
•Optical Character recognition (image-to-text)
•Information extraction & data mining from text
•Well-known systems: Google translate;Siri; Watson
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 8
Uses of NLP
•Given my 14-year-old’s text messages:
w
nt
2
evan
s
hs
4
dnnr
cul8r
w
nt
2
evan
s
hs
4
dnnr
cul8r
•Can we “translate” this; or “recognize” this?
•
IfwecantranslatefromChinesetoEnglish
•
If
we
can
translate
from
Chinese
to
English
,
maybe we can automatically translate this
•
Ifwecanrecognizespokenlanguagemaybe
•
If
we
can
recognize
spoken
language
,
maybe
we can recognize this type of input
(Infact
lotsofphonologicalcontent)
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 9
(In
fact
,
lots
of
phonological
content)
AutomaticNLP systems
•Most systems based on statistics of actual use
•
Trainingdata:textwithtranslations;transcribed
•
Training
data:
text
with
translations;
transcribed
speech
•
Systemsmake
“
bestguess
”
ofstatisticalmodels
Systems
make
best
guess
of
statistical
models
•Often very useful, but can be wrong
•
ThinkofGoogletranslateor
Siri
•
Think
of
Google
translate
or
Siri
•Systems can be very complex but still seem dumb
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 10
WhatdoesNaturalLanguage
What
does
Natural
Language
Processin
g
mean to us?
g
•The rehabilitation engineer: A way to make
AAC more intuitive and natural to use
•The computer scientist: The means to make
intelligent AAC systems –reducing some of the
load from the user and enabling better
communication.
•The computational linguist: Links to other kinds
of human language technology; AAC offers new
applicationsrequiringnovelNLPtechniques
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 11
applications
requiring
novel
NLP
techniques
.
WhatdoesNaturalLanguage
What
does
Natural
Language
Processing mean to us?
•The speech-language pathologist: Language
supportduringassessment,treatment,anduseof
support
during
assessment,
treatment,
and
use
of
SGDs for people with CCN
•The AAC user: Allows the device to s
p
eed u
p
the
pp
communication for the AAC user. Word
prediction allows conversation to be faster so I can
p
articipate more. It would be really nice for it to
be able to know what I am thinking.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 12
ElfNlL
E
ar
l
y uses o
f
N
atura
l
L
anguage
ProcessinginAAC
Processing
in
AAC
•Word prediction –syntactic knowledge, n-gram
m
ode
lin
g
odeg
•Semantic phrase retrieval
•
Expandingtelegraphicinput
–
syntaxlexical
Expanding
telegraphic
input
syntax
,
lexical
semantics
•
Utterance
-
basedsystems
–
typicalconversational
Utterance
based
systems
typical
conversational
patterns, script/goals etc…
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 13
UiifDlM
U
n
i
vers
i
ty o
f
D
e
l
aware:
M
ore
IntelligentWordPredication
Intelligent
Word
Predication
•Word Predication is used a lot in current AAC
app
li
cat
i
o
n
s,
but
i
t
h
as
gotte
n
a
bad
n
a
m
e
appcatos,buttasgotteabadae
–Interface issues
–Need to specify/switch word predication dictionaries
–Context switching required
•Exasperated by inappropriate predictions
Uifllldiil
•
U
se a var
i
ety o
f
rea
ll
y
l
arge corpora an
d
stat
i
st
i
ca
l
language processing
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 14
Prediction Exam
p
le
p
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 15
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 16
Can we do better??
•Intuitively all possible words do not occur with
equa
l lik
e
l
y
h
ood
du
rin
g
a
co
n
ve
r
sat
i
o
n
.
equa
eyood
dugacovesato.
•The topic of the conversation affects the words
that will occur.
–E.g., when talking about baseball: ball, bases, pitcher,
bat, triple….
–
How often do these same words occur in your second
language acquisition class?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 17
Topic Modeling Approach
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 18
Topic Identification
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 19
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 20
Topic Similarity Scores for Above
Cti
C
onversa
ti
on
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 21
Topic Application
•How do we use those similarity scores?
•
Essentiallyweightthecontributionofeachtopic
•
Essentially
weight
the
contribution
of
each
topic
by the amount of similarity that topic has with the
current conversation.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 22
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 23
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 24
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 25
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 26
STlkdU
S
cene
T
a
lk
er an
d
U
tterance-
BasedSystem:Pragmatics
Based
System:
Pragmatics
•Joint work with University of Delaware, Jan
B
ed
r
os
i
a
n
(Weste
rn Mi
c
hi
ga
n
U
ni
ve
r
s
i
ty),
Lin
da
edosa(WestecgaUvesty),da
Hoag (Kansas State University)
•S
y
stem Recommendations based on findin
g
s from
yg
a series of experiments looking at pragmatic
mismatches between prestored utterances and
current discourse situation
•Looked at attitudes of an unfamiliar
ilil
did
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 27
conversat
i
ona
l
partner
i
n a goa
l
-
di
recte
d
transactional situation
Example Screen
Clear
4
4
5
5
6
6
7
7
1
1
2
2
3
3
Nachosplease.
appetizer
soup/salad
entree
dessert
4
4
5
5
6
6
7
7
1
1
2
2
3
3
I ‘d like an
I ‘d like an
appetizer, appetizer,
pleaseplease
What would
What would
you you
recommendrecommend
I’d like a
I’d like a
salad, salad,
pleaseplease
What’s your
What’s your
soup of the soup of the
day?day?
What are
What are
today’s today’s
specials?specials?
I‘d like
I‘d like
dessert, dessert,
pleaseplease
No
No pepper
pepper,
,
pleaseplease
?
What’s
What’s in in
that?that?
Nachos
Nachos
pleaseplease
Garden
Garden
salad salad
PleasePlease
I’ll
I’ll have that have that
I’ll have that
I’ll have that
Crème
Crème
bruleebrulee
What are
What are
my my
choices?choices?
We’re going
We’re going
to split to split thatthat
OK we’ll
OK we’ll
have thathave that
Bleu cheese
Bleu cheese
on the sideon the side
No thanks
No thanks
Fries
Fries
What would
What would
you you
recommendrecommend
I’ll
I’ll order order
from the from the
menumenu
Extra
Extra
cheesecheese
No thanks
No thanks
No onion
No onion
Clam
Clam
chowder chowder
pleaseplease
I’ll have
I’ll have thatthat
The steak
The steak
medium medium
rarerare
assistance
quickfires
waiter
smalltalk
AAC Research at Dundee
The aim of our research group is to utilise NLPto support language and
conversational development by:
Ebli l l
橫 hi
•
E
na
bli
ng
l
anguage
p
l
慹
j
o
k
敳e
p
h
潮
i
捳
•Providing automatic access to appropriate vocabulary
•
Scaffolding communication skills
Scaffolding communication skills
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 29
Examples of NLP & AAC
•STANDUP ‐Enabling language play
•How was School Today? –narrative generation
•Sound Prediction
䥳捡I
•etc
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 30
STANDUP: Pun Generation –Rolf Black Dr Graeme Ritchie, Aberdeen
Dr Helen Pain, Edinburgh
What do you call a
What do you call a
spicy missile?
A hot shot
A hot shot
.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 31
Language Exploration: Joke Generation
Whdll
synonym
k
?
Wh
at
d
o you ca
ll
a
strangemar
k
et
?
Abizarrebazaar!
homophone
synonym
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 32
The SCAN Project –Ha Trinh PhDshi
p
funded b
y
SICSA
/
Dundee Universit
y
•Aims to develop a sound‐based communication aid for
n
o
n
spea
kin
g
peop
l
e
wh
o
e
x
pe
ri
e
n
ce
li
te
r
acy
d
iffi
cu
l
t
i
es
p
y/y
ospeagpeopeoepeeceteacydcutes
•Using a sound‐to‐speech approach to enable users to
generate novel words and sentences in interactive
generate novel words and sentences in interactive
conversation
•
Employing
NLP
瑯 improve the usability and accessibility
䕭灬潹楮E
NLP
瑯 improve the usability and accessibility
潦 the proposed system
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 33
How was School Today? -Rolf BlackDigital Economy (RCUK) Funded;
“Storytelling puts an equal emphasis
on the feelings and audience
Partners: Aberdeen, CS, Perth Education
on the feelings and audience
楮癯汶敭敮i as on the structure of the
story; and regards narrative
develo
p
ment as a social
p
rocess which
pp
扥杩湳 in infancy and is scaffoldedby
adults” [Grove, 2009]
The “How was School toda
y
...?”
y
灲潶楤敳 the child with automatically
generated narrative utterances based
on sensor and other data.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 34
Photos:Prototype user testing at Corseford School (left)
and interface screenshot.
Interaction Data
RFID sensors track the child’s
interactions with:
•teaching and other staff;
dfid
•peers an
d
f
r
i
en
d
s;
•objects such as teaching tools.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 35
Location Data
Hall
Classroom
Sensors on doorways detect the
location of the child.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 36
User Modelling
The timetable provides
information about time,
activity, interaction and
location.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 37
Additional Data
Voice recordings can be added
to the database to provide
additional information that
捡湮潴 be detected by sensors.
Database
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 38
Generating Narratives
The system then generates a
narrative with several events.
E t it f b
E
very
敶en
t
捯湳
i
s
t
s
o
f
a
湵n
b
敲
潦 messages.
This supports interactive story‐
tlli d t
t
e
lli
ng
慳
潰灯獥
d t
o
a
浯湯汯杵m story.
Event: Music Class
2 messages generated
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 39
Brain Computer Interface Project
•Rapid Serial Visual Presentation (RSVP™)
–
Linearscanning,switchactivationviaP300ERP
Linear
scanning,
switch
activation
via
P300
ERP
–Important role for the language model
•Use in ERP detection, similar to use in speech recognition
•Use to decide sequencing (scan likely symbols earlier)
•Use to decide to stop scanning earlier, speeding text entry
•Use for word com
p
letion and
p
rediction
pp
•Use for graceful recovery from spelling errors
•As with other scanning methods, speed is needed
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 40
–Language model is providing some of that
Implications of NLP on AAC?
•Clinical Perspective
•
EndUserPerspective
•
End
User
Perspective
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 41
IliifNLPAAC
I
mp
li
cat
i
ons o
f
NLP
on
AAC
:
ClinicalPerspective
Clinical
Perspective
Assessment challenges:
•
CanNLPassisttheclinicianevaluate(speech
•
Can
NLP
assist
the
clinician
evaluate
(speech
,
language and writing) communication skills in
b
oth testin
g
and natural settin
g
s?
gg
•Can NLP assist the clinician to identify an end
user’s language level? There are quantitative and
qualitative changes in language behaviorsover
time. Yet SLPs are not aware of these changes
hliSGD
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 42
w
h
en eva
l
uat
i
ng
SGD
use.
•Changes in representation levels: As a child
becomes literate, she moves from picture or
blbdttthhCNLP
sym
b
o
l
b
ase
d
sys
t
ems
t
o or
th
ograp
h
y.
C
an
NLP
assist with assessment of a child’s literacy level
overtime
andmakeadjustmentstotheSGDs?
over
time
and
make
adjustments
to
the
SGDs?
•The adult with language impairments secondary to
TBIoraphasiaisregainingskills;canNLPassist
TBI
or
aphasia
is
regaining
skills;
can
NLP
assist
with the identification of these changes and adjust
language presentation in SGDs?
•Can NLP predict if an adult is experiencing early
degenerative language disease?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 43
TreatmentChallengesfromthe
Treatment
Challenges
from
the
SLP Pers
p
ective:
p
•SGD use requires considerable working memory.
Can NLP li
g
hten the co
g
nitive load?
gg
•When the user moves from word generation to
word prediction, she must change her cognitive
set. Can NLP help with attentionaldemands of
prediction/completion during message generation?
•Organization of vocabulary and creation of an
optimized layout is still unstandardized. Can NLP
ktkfhlithit
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 44
k
eep
t
rac
k
o
f
h
ow peop
l
e are us
i
ng
th
e
i
r sys
t
em,
and optimize layout and organization?
•
Rightnow,ausermustlearnlanguage,concept
Right
now,
a
user
must
learn
language,
concept
organization and production, as well as device
operations and strategic demands. Can NLP help
with operation, strategic and linguistic competence
for interaction?
•Can NLP create rules for knowledge organization
based on context: user’s linguistic skills; partners’
killdlltid
s
kill
s an
d
l
anguage use;
l
oca
ti
on an
d
environmental cues for message generation and
interaction
interaction
.
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 45
IliifNLPAAC
I
mp
li
cat
i
ons o
f
NLP
on
AAC
:
EndUserPerspective
End
User
Perspective
•Include AAC users in entire development process
–
des
i
g
n
,
p
l
a
nnin
g,
eva
l
uat
i
o
n
desg,pag,evauato
•Developers and users might have different
p
ur
p
oses
pp
•Important to me
–
Accessibilit
y
with limited dexterit
y
yy
–Accessibility in ALL conditions
•My Dream: read my brain and translate my
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 46
thoughts into speech!
Introducing SLPAT
•Speech and Language Processing for Assistive
T
ec
hn
o
l
og
i
es
(S
LPAT
)
www.s
l
pat.o
r
g
ecooges(S)
www.spat.og
•Three workshops over past three years
•
LA2010;Edinburgh2011;Montreal2012
LA
2010;
Edinburgh
2011;
Montreal
2012
•Another planned for 2013 (at Interspeech)
•
Forthcomingspecialissue
•
Forthcoming
special
issue
•Recently formed SIG of ACL
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 47
Introducing SLPAT
•Range of topics in workshops and special issue
•
Generationofutterancesandnarrativesfromdata
Generation
of
utterances
and
narratives
from
data
•Scanning systems
•Processing of dysarthricspeech
•Text simplification
•Word-, utterance-or phoneme-based predictive
systems
•Communication and Alzheimer’s disease
•
Signlanguagerecognitionand
translation
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 48
•
Sign
language
recognition
and
translation
Questions/Comments?
© Waller, Roark, Fried-Oken, McCoy, McGregor: ISAAC 2012Slide 49
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment