Content using Natural Language

spectacularscarecrowΤεχνίτη Νοημοσύνη και Ρομποτική

17 Νοε 2013 (πριν από 3 χρόνια και 1 μήνα)

42 εμφανίσεις

Kingston
University

Creating and Editing Mathematical
Content using Natural Language
Commands

A Progress Report of the TalkMaths
Project

Eckhard Pfl
ü
gel

Faculty of Science, Engineering
and Computing

Kingston University

Kingston
University

Overview


Motivation and Objective


Existing Solutions


The Technical Challenges


Our Answer: TalkMaths


Current Progress


Future Steps


Conclusion

Introduction


Our Motivation:


Not everyone can use a computer with
traditional input devices


Mathematical content is notoriously difficult to
be accessed with alternative technologies


Our Objective:


To carry out research and development
that will lead to the creation of a user
-
friendly tool/system, for accessing
mathematical content using speech

Kingston
University

Speech Recognition
Technology


Continuous speech recognition has been
around for a while


Recently, experienced tremendous
improvements and attraction in the media
(
iPhone
, Google)


Two different architectures: client
-
based,
server
-
based


Why would people want to use it?


Out of
commodity?


Out of necessity!


Kingston
University

Desktop
-
based Speech
Recognition Tools


Currently, most reliable (commercial) tool
available is Dragon NaturallySpeaking
(DNS)


An alternative is Windows Speech
Recognition (free since Vista, much
improved in Windows 7)


An open source (free) solution exists

Sphinx


Other solutions might appear in the near
future


Kingston
University

Dragon NaturallySpeaking


Popular commercial speech recognition
product


Excellent continuous speech recognition
rates


Effectively allows for hands
-
free creation
and editing of text


Additional features:


Commands for typical computer
-
based tasks


Supports creation of (application
-
specific or
global) macros


Kingston
University

Dragon NaturallySpeaking


Drawbacks


No support for mathematics (all symbols need
to be spelled out)


Does not integrate in any mathematical editor


Cannot “translate” speech input into specific
mathematical
markup

(
MathML
,
LaTeX
)


Development of more sophisticated macros
needs (expensive) SDK


Anecdotal evidence of mediocre user support


Kingston
University

Standards for Spoken
Mathematics


Motivated by different contexts:


Dictating to other human beings


Input for computer systems (parsing)


Audio output for text
-
to
-
speech systems


Probably first documented source: Chang’s booklet
[1]


Fateman

[2,3] gives fairly detailed rules for spoken
mathematics


Other approach: [4]


Raman [5] is motivated by TTS


Our contribution, based on [2,3]:
Wigmore

[6]

Kingston
University

Existing Tools and Systems


MathTalk


Collection of Dragon NaturallySpeaking macros


Only usable as input for specific computational maths
interface


Commercial product (expensive)


Limited scope


Maths Speak & Write


Uses Windows speech technology


Desktop application with GUI


Supports multi
-
modal input


Research Project


Mainly experimental relevance

Kingston
University

Related Tool


Speed


Goal: programming by voice


Speech
-
plugin

for Eclipse IDE


Uses Java interface for DNS


Speech input with keyboard and mouse


Exports spoken version of Java


This includes navigation


Also research project


Overall conclusion:
none of the existing tools are fit for
purpose.

Kingston
University

Kingston
University

Technical Challenges


Research aspect: Need tools and techniques from
natural language processing, compiler construction
and HCI


Standards for spoken maths


Need flexible and powerful grammar


Have to deal with ambiguity


Parsing Algorithms


Difficulties: incomplete or incorrect input


Editing Paradigms


Need novel strategies for speech
-
driven UIs

Kingston
University

TalkMaths Background


Use speech recognition for my own work


Frequently needed specialist tasks:
enter/modify mathematical equations


Could use Equation Editor combined with
DNS macros


More problematic if using LaTeX


Idea: write more sophisticated commands


Turned this into research project

Our Answer: TalkMaths


Web
-
based User Interface/System


Separate the application from speech
-
front
-
end


Devise special class of speech commands
(“speech templates”)


Insight: spoken mathematics can and
should be process similarly to spoken
structured content (i.e.
markup

languages,
programming code)


Kingston
University

TalkMaths UI

Kingston
University

TalkMaths.org Web UI

Kingston
University

http://talkmaths.org/editor.
php

TalkMaths.org Web UI



Future Version

Kingston
University

Kingston
University

How can TalkMaths Help with
Accessibility of Mathematics?


TalkMaths aims at people for whom it is


Difficult to use keyboard/mouse


Difficult to decipher equations on screen


TalkMaths can help by


Speaking mathematical input


Equation rendering: arbitrary big font sizes


Voice
-
activated zooming function


Editing mathematical expressions by voice


Planned features
:


Playback of formulae


Importing/exporting existing documents


Maintaining documents on server


Progress Report


Are developing speech front
-
end
prototype, using Windows speech
recognition


Still have issues with the DNS interface


About to release new version of parser
with more robust error recovery


New website design in progress

Kingston
University

Kingston
University

Future Work


Extending the range of covered mathematics


Allowing for multi
-
line equations/multiple/embedded
expressions


Potentially, allows sharing of maths input


Improved speech editing (“select and say”)


Higher robustness of code/ease of use/installation


Improved documentation

Ensuring Continuation


Ph.D. student support (full
-
time until
March 2014, part
-
time until 2017)


Industrial funding?


EU funding?


Domain/hosting not that expensive

Kingston
University

Kingston
University

Conclusion


TalkMaths

seems to be a novel
application


TalkMaths

can be very helpful for
members of Higher Education with
computer access problems


The current prototype will improve
significantly over the next years


Keep up to date:
www.TalkMaths.org


Acknowledgements


This work would have been impossible
without my colleagues and students:


James Denholm
-
Price


Gordon Hunter


(and others)


Funded by


EPSRC Doctoral Training Award


MSOR Mini Project grant

Kingston
University

References

1.
Lawrence A. Chang. Handbook for spoken mathematics (Larry's
speakeasy). Lawrence Livermore National Laboratory, University of
California, USA, 1983.

2.
R.
Fateman
. How can we speak math? University of California at
Berkeley, 2009.

3.
R.
Fateman
. 2
-
D Display of Incomplete Mathematical Expressions.
University of California at Berkeley, 2006.

4.
Cameron Elliott and Je A.
Bilmes
. Computer based mathematics using
continuous speech recognition. CHI 2007 Workshop on Striking a
C[h]
ord
: Vocal Interaction in Assistive Technologies, Games and More,
2007.

5.
T. V. Raman. Audio system for technical readings. Springer
Verlag
,
Berlin, 1998.

6.
A.
Wigmore
. Speech
-
Based Creation and Editing of Mathematical
Content. Ph.D. Thesis, Kingston University, U.K., 2011
.




Kingston
University