Table of Contents

rangesatanskingdomSoftware and s/w Development

Dec 2, 2013 (3 years and 8 months ago)

61 views

Gesture Recognizer


A
Developer’s Guide

September 2
3
nd
,

2008

Mikko Kauppila


Table of Contents

1. Introduction

................................
................................
................................
................................
...................

2

2. Eclipse Integrati
on

................................
................................
................................
................................
.........

3

2.1 Importing the Build File(s)

................................
................................
................................
.......................

3

2.2 Eliminating Warnings (and Errors)

................................
................................
................................
...........

3

2.3 Running Ant Targets

................................
................................
................................
................................

3

2.4 Importing Formatting Rules

................................
................................
................................
.....................

3

3. Motion Source Framework

................................
................................
................................
............................

4

3.1 Example: Adding a New Sensor Type

................................
................................
................................
......

4

4. Gesture Recognition Framework

................................
................................
................................
...................

6

4.1 Overview

................................
................................
................................
................................
..................

6

4.1 Feature Extraction

................................
................................
................................
................................
...

6

4.2 Hidden Markov Models

................................
................................
................................
...........................

8

5. Training

Framework

................................
................................
................................
................................
.......

9

5.1 Feature Extraction

................................
................................
................................
................................
...

9

5.2 Estimating Individual Gesture Models
................................
................................
................................
...

10

5.3 Combining Individual Gesture Models

................................
................................
................................
..

10

6. J2ME Compatibility

................................
................................
................................
................................
......

11

6.1 Bootclasspath modifications

................................
................................
................................
.................

11

6.2 Collections

................................
................................
................................
................................
.............

11

6.3 Serialization

................................
................................
................................
................................
...........

12

6.4 Logging

................................
................................
................................
................................
...................

12


1.
Introduction

In this document, we will describe how the recognizer can be modified and extended.

Even if you aren’t
planning to modif
y

and extend the recognizer, this document provides a lot of insight to the internal
workings of the r
ecognizer.

Knowledge of
Java (including generics) and
UML

class diagrams is assumed
. To get the most out of this
document, working knowledge of machine learning methods
is a must.

2.
Eclipse Integration

Extending and modifying the recognizer may require w
orking on its source code.
This can be done
conveniently using the Eclipse IDE (
http://www.eclipse.org
), as follows.

2.1 Importing the Build File(s)

First, choose
File
-
> New
-
> Project…
-
> Java
-
> Java Project from
Existing Ant Buildfile

-
> Next
. Enter the full
path of either build
-
j2se.xml or build
-
j2me.xml, depending on which one you wish to work on. (If you wish
to work on both, repeat this procedure for both buildfiles.) The project name will be set automatically

once
you enter the buildfile path.
Check the item
Link to buildfile in the file system
, then hit
Finish
.

2.2 Eliminating Warnings
(
and Errors
)

To get rid of needless compiler warnings (and even errors on J2ME side), you should do the following:



If you’re
getting warnings about missing
serialVersionUID

fields, you can do one of two
things: either disable this warning for the selected project, or disable it for the whole workspace
(i.e., for all projects by default). To disable the warning for the selected p
roject, select
Project
-
>
Properties
-
> Java Compiler
-
> Errors/Warnings
-
> Enable project specific settings
-
> Potential
programming problems
-
> Serializable class without serialVersionUID
-
> Ignore
-
> OK
. To disable the
warning for the whole workspace, s
elect
Window
-
> Preferences
-
> Java
-
> Compiler
-
>
Errors/Warnings
-
> Potential programming problems
-
> Serializable class without serialVersionUID
-
> Ignore
-
> OK
.



For both build
-
j2me.xml and build
-
j2se.xml, you need to remove the reference to JRE_LIB by
selecting
Project
-
> Properties
-
> Java Build Path
-
> JRE_LIB
-
> Remove
. For build
-
j2se.xml, you must
now (in the same view) select
Add Library…
-
> JRE System Library
-
> Finish
.
On J2ME, this change is
mandatory
: otherwise Eclipse will show needless compil
er errors.



On J2ME, you must build the project once (via ant target “build”) before all errors vanish. This is
because the first build will produce a set of low
-
level classes to
/
bin/j2me
-
boot, which are used by
Eclipse (Section 6.1).

2.
3

Running
Ant Targe
ts

To build the system directly in Eclipse, you must go to Ant view (
Window
-
> Show View
-
> Ant
), select
Add
Buildfiles…
, and then choose build
-
j2me.xml and/or build
-
j2se.xml. The buildfile target
s can then be run

straightforwardly via the Ant view. You ma
y also wish to select
Hide Internal Targets

from the Ant view to
hide extra targets that shouldn’t be run directly.

2.
4

Importing Formatting Rules

Finally,

you may wish to

import formatter rules
for the project from the file /eclipse/formatter.xml
(
Propert
ies
-
> Java Code Style
-
> Formatter
-
> Enable project specific settings
-
>
Formatter
-
> Import…).

Ant
target
checkstyle

in both build
-
j2se.xml and build
-
j2me.xml can be used to check the code standard.

3.
Motion Source Framework

A motion source is a sourc
e of motion data.
Motion data consist
s

of variables such as acceleration, angular
velocity, compass reading, and button presses on the motion sensor



however, only acceleration is
mandatory.

Package
fi.ubilife.gesture.motion

provides a framework for such
motion sources
(Figure 1).


Figure 1: Main classes in package

fi.ubilife.gesture.motion
.

The role of a
MotionSource

is to create
MotionDatum
s and pass them to its
MotionListener
s.
Notice that there

are two parallel hierarchies: one for
MotionSource

(on th
e right)
, and one for
FusionMotionSource
(on the left)
objects.

A fusion motion source fuses da
ta from multiple motion
sources
;

in gesture recognition this could be used to implement two
-
handed gesture recognition.

Each
MotionSource

is associated with a
Mo
tionSetup
, which contains
serializable metadata about
the
source
,
such as the sensor’s update frequency.

The providing of the motion data is implemented asynchronously via publish
-
subscribe pattern (
classes
MotionListener

and

FusionMotionListener
). Classes

CompositeMotionListener

and
CompositeFusionMotionListener

provide convenient implementations for
broadcasting

motion
data to multiple listeners
.

3.1 Example: Adding a New Sensor Type

If you wish to add a new motion sensor to the framework, just follow the

pattern seen in
ShakeMotionSource

and
NokiaMotionSource

in Figure 1:



your sensor class should
implement
Mo
tionSource
; and



your sensor

should
contain
a single
CompositeMotionListener

to ho
ld references to
listeners
.

In addition, you may wish to add support

to your sensor in the
desktop
-
based
sensor editor GUI (ant target
sensor
-
editor

in build
-
j2se.xml).

See class
fi.ubilife.gesture.recognizer.training.sensor.AddSensorDialog

to see how this
is implemented for
NokiaMotionSource

and
ShakeMotionSource
; then im
plement a similar
feature for your sensor.

4. Gesture Recognition Framework

In this section we will see how the recognition process works, and how you can make changes to it.

4.1 Overview

The recognition framework is provided by package
fi.ubilife.gesture
.recognizer
, as depicted
in Figure 2.

(
Notice that applications see only classes
GestureSource

and
GestureListener
.
)


Figure 2: Recognition framework.

Matcher

is an abstract class whose implementations actually do the low
-
level recognition. Matchers
produ
ce instances of class

GestureMatchInternal
, which is a subclass of
GestureMatch

containing some additional information for cross
-
validation
; a
pplications
will see them downcasted to
GestureMatch
.

GestureSourceImpl

is essentially a
mediator
: it mediates

Ges
tureMatch

objects from
Matcher
s
to
GestureListener
s, and motion data from
FusionMotionSource

to
Matcher
.
Also,
GestureSourceImpl

can perform optional plotting.

There are currently two
Matcher

implementations:
IsolatedMatcher
, which isolates gestures using
button data from motion sources, and c
lassifies each isolated gesture; and
GarbageMatcher
, which
performs continuous recognition

using a

garbage model.

4.1 Feature Extraction

Feature extraction is a process that transforms raw motion data to feature vector
s. It is these feature
vectors that are in the end passed to the statistical pattern recognition models (Section 4.2).

Feature
extraction seeks to eliminate unnecessary variability in the motion signal.


Figure 3: Feature extraction classes.

Our feature e
xtraction framework is depicted in Figure 3.

The top
-
level class,
FusionGestureFeaturizer
, consists of a single
GestureFeaturizer

for each sensor.

Essentially,
FusionGestureFeaturizer

transforms
MotionDatum

objects (
received
from motion sources,
Section 2)

to
FusionFeature

objects.

Although Figure 2 doesn’t show it in full, the
Matcher

implementations
both
contain a single
FusionGestureFeaturizer
.

Each
FusionGestureFeaturizer

is also associated with a
W
eightedEstimator<FusionFeature>
, which is used to estim
ate probability distribution functions
of features
(
Pdf<FusionFeature>
) from training data.

(
In practice, we use multivariate Gaussian
distributions.
)

There is a single default implementation of
GestureFeaturizer

called

DefaultGestureFeaturizer
. It is made

of a single
Preprocessor
, and a single
Postprocessor
.

A
Preprocessor

processes a continuous signal on
-
the
-
fly, whereas a
Postprocessor

handles data in
batches. It is important to notice that
post
-
processing is never used in continuous recognition
, because

a
continuous recognizer doesn’t know wherein the acceleration sequence the gestural batches lie in.

There are a great variety of different
Preprocessor

and
Postprocessor

implementation
s

available
in package
fi.ubilife.gesture.recognizer.features
.

Many

are

based on standard decorator
and composite design patterns.

In practice,
feature extraction is extended by
writing

new implementation
s

of
Preprocessor

and
Postprocessor
.

Section 5 explains how to get the recognizer to use your new implementations.

4.
2 Hid
den Markov Models



Figure 4: Hidden Markov Model classes.

We use Hidden Markov Models as our underlying statistical model.
Figure 4 depicts the
classes, provided
mostly by package
fi.ubilife.gesture.gm
. (The letters “gm” stand for generative model, an
ex
ample of which a Hidden Markov Model is.
)

Notice that
the

classes are generified (the “<E>” part), and
can be used
for any type

of observations

(
e.g.,
univariate, multivariate, or wrapped).

The main classes are
Hmm<E>

and
Viterbi<E>
. A
n

Hmm<E>

consists of
a transition matrix stored in
sparse format for efficiency
,

and a
Pdf<E>

(probability distribution function)

for each hidden state.

Viterbi<E>

provides
the Viterbi algorithm for finding maximum
-
likelihood state sequences.

(Notice that
Hmm<E>

and
Viterbi<E>

are just realizations of their more abstract counterparts
GenerativeModel<E>

and
PathAlgorithm<E>
.)

The classes
Bakis
,
HmmEstimator<E>
,
CombinedModel<E>
,
FillerModel<E>
, shown on the
left side of Figure 4, are utilities for building HMMs
.

T
hey
are used a
t training time only (Section 5).

5. Training Framework

As explained in Section 4, all low
-
level recognition is done by
Matcher

objects.
The ultimate objective of
training is, therefore, to produce such
Matcher

objects from training data
.

The classes invo
lved in training
are depicted in Figure 5. The number of classes may look intimidating at first, but we will go through them
one at a time to observe that each
serves

its purpose.


Figure 5: Training framework.

5.1 Feature Extraction

Starting from the top

of Figure 5
, we have
FusionGestureFeaturizerFactory
, which creates
FusionGestureFeaturizer
s given a
FusionMotionSetup
. During training, we always assume
that we have a
fixed

FusionMotionSetup
.
1

However, we can’t use a fixed featurizer, since
FusionGesture
Featurizer
s are
stateful

objects:
2

if we wish to extract features from multiple
motion signals, we must use different
FusionGestureFeaturizer

at each time


this is the
mo
tivation for using this

additional factory layer.




1

This
FusionMotionSetup

is read from the training data files; it is an error to use files recorded with different
sensor configurations.

2

Stateful means that the object’s methods

may return different results with identical arguments, depending on
previous invocations. For example, consider a lag filter that always returns the argument passed to it 10 invocations
ago; such object is evidently stateful.

The s
ingleton c
lass
TrainingSetup

c
ontains training
-
related configuration. For example, it contains a
single
FusionGestureFeaturizerFactory
, which
will

effectively
be
used throughout the
training
.

Therefore,
all feature extraction related configuration is inherently hard
-
coded to
TrainingSe
tup
.
3

If
you wish to change the feature extraction strategy, see the class
TrainingSetup
.

5.2 Estimating Individual Gesture Models

On the bottom
-
left side of Figure 5 is class
GestureEstimator
,
which estimates the HMM parameters
of individual gesture types



of a clockwise hand movement, for example


and returns the said parameters
as
Hmm<FusionFeature>

objects. That is, for each ge
sture type, there will be a single
Hmm<FusionFeature>

object

created
.

GestureEst
imator

objects are created by
TrainingSetup
;

in other words,
TrainingSetup

encapsulates all configuration related to estimating
individual gesture models

(for example, number of
HMM states used).

5.3 Combining Individual Gesture Models

Once these individual gesture models have been estimated, we must

still
combine

them into a
Matcher
.
This is done by the
MatcherConstructor

objects, seen on the bottom
-
right side of Figure 5.

There are
multiple
MatcherConstructor

implementations available: for example,
IsolatedMatcherConstructor

produces
IsolatedMatcher

objects.

TrainingSetup

provides factory methods for creating
MatcherConstructors
,
encapsulating

all parameters.

The bottom line is that
class
TrainingSetup

contains
most of the

hard
-
coded training configuration
.



3

We aren’t exactly
proud

of
such

state of affa
irs
.

However, making

all training par
ameters exter
nally configura
ble
would have been a major task.

6. J2ME Compatibility

The recognizer
is

wri
tten to run on both J2ME
and J2SE
.

4

After modifying

the recognizer, it is a good idea to
rebuild it for both platforms to ensure compatibility.

Essentially, we have tried to enforce compatibility via folders:



folder /src
-
common/ should contain only portab
le code;



folder /src
-
j2me/ may contain J2ME
-
specific code;



folder /src
-
j2se/ may contain J2SE
-
specific code.

Portable code
in /src
-
common/

may use most Java
4 and Java
5 language features
.

The supported features
are:



Generics
,

achieved using target
jsr14

(
a backward compatibility mode

in javac compiler
)
.



Varags, achieved using target jsr14.



Enhanced for loop, achieved using target jsr14.



Anno
tations, including the standard Java 5

annotation
s

(e.g.,
SuppressWarnings
)
;

achieved
using additional bootclasspath
classes (Section 6.1).



Enumerations
;

achieved using additional bootclasspath classes (Section 6.1).

The following
language
features are
not supported
:



Assertions. Instead of using
Java 4 s
tyle assertion
s
,
all portable code should
throw instances of
Asserti
onError
.

6.1
Bootclasspath modifications

Folder /src
-
j2me
-
boot/ contains
a few
re
-
implementations of classes in

system packages (
java.lang.*
)
normally not available on J2ME.

These classes are automatically
built
by build
-
j2me.xml and packaged

to /bin/j2me
-
boot/jvm14/ and
/bin/j2m
e
-
boot/jvm15/, targeted

to JVM

versions
1.4 and
1
.5
, respectively.

Many phone models refuse to install JAR files containing classes in system packages. Therefore, using
obfuscation is required. We use the Eric Lafortune’s ProGuard o
bfuscator.

Again, this is all encapsulated by
build
-
j2me.xml.

6.2 Collections

Unlike J2SE, J2ME doesn’t provide an extensive set of collection classes. Package
fi.ubilife.gesture.collections

provides a portable set of collection classes.

Please use them
in

all portable code in
/src
-
common/.




4

To be precise, by J2ME
and J2SE we actually refer to CLDC 1.1 and J2SE 5.0.

6.3 Serialization

Java provides a native serialization mechanism, but unfortunately it is not compatible over different JVM
versions
, and not even available on J2ME.

Therefore, a custom serialization mechanism
is
provide
d by
package
fi.ubilife.gesture.conte
nt
. Take a look at
some
classes implementing
ContentSerializable

to get a hunch of how it is used.

6.4 Logging

We have used Apache’s excellent log4j for logging.
A light
-
weight

implementation of log4j for J2ME is
contai
ned in /src
-
j2me/.
Therefore,
please us
e log4j
-
style logging consistently
.