[Cheng 2013]= Viewgraphs-byAngusx

soilflippantΤεχνίτη Νοημοσύνη και Ρομποτική

17 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

75 εμφανίσεις

實驗室
:

先進網路技術與服務實驗室


報告者
:
黃福銘

(Angus F.M. Huang)


NuActiv
:
Recognizing Unseen New
Activities Using Semantic Attribute
-
Based Learning

TMSG

2013.07.10

Angus F.M. Huang

Publication


MobiSys

2013




Session

8
:

Behavior

and

Activity

Recognition


NuActiv
:

Recognizing

Unseen

New

Activities

Using

Semantic

Attribute
-
Based

Learning


SocioPhone
:

Everyday

Face
-
To
-
Face

Interaction

Monitoring

Platform

Using

Multi
-
Phone

Sensor

Fusion


MoodScope
:

Building

a

Mood

Sensor

from

Smartphone

Usage

Patterns


Auditeur
:

A

Mobile
-
Cloud

Service

Platform

for

Acoustic

Event

Detection

on

Smartphones

2

Angus F.M. Huang

Authors


Heng
-
Tze

Cheng


Feng
-
Tso

Sun


Martin

Griss


Electrical

and

Computer

Engineering


Carnegie

Mellon

University



Paul

Davis


Jianguo

Li


Di

You


Applied

Research

Center


Motorola

Mobility

3

Angus F.M. Huang

Abstract


To recognize a new human activity


No such training example



Essential for


User
-
centric & Context
-
aware applications


Most methods


Only recognize known
activities

4



NuActiv


Semantic

attributes

for

representation


Two
-
layer

zero
-
shot

learning

algorithm


User

feedback

for

reinforcing

accuracy



Evaluation


10
-
exercise
-
activity

dataset

collected


Public

dataset

of

34

daily

life

activities


Achieved

up

to

79
%

accuracy

Angus F.M. Huang

Outline


INTRODUCTION


NUACTIV
SYSTEM OVERVIEW


Scenarios
and Design
Considerations


System
Architecture of
NuActiv


SYSTEM
DESIGN AND
ALGORITHMS


Feature
Extraction


Semantic
Attribute
-
Based Activity
Recognition


Active
Learning: Reinforcing
Activity Recognition Using Minimal
User
Feedback

5


EVALUATION


System Implementation


Datasets


Evaluation Methodology


Case Study I: Exercise Activities


Case Study II: Daily
-
Life Activities


Active Learning Experiments



CONCLUSION

Angus F.M. Huang

Introduction


Diversified

people

daily

activities


462

different

activities


American

Time

Use

Survey



Two

steps

of

existing

approaches


(
1
)

Collect

and

label


(
2
)

Classify


6

Angus F.M. Huang

Two
Research Questions


Q
1
.

How

to

recognize

a

previously

unseen

new

activity

class

when

we

have

no

training

data

from

users
?



Q
2
.

If

we

have

the

opportunity

to

ask

users

for

labeled

training

data,

how

to

reinforce

the

recognition

accuracy

using

minimal

help

from

users?

7

Angus F.M. Huang

Two
Inspirations


Many

human

activities

and

context

types

share

the

same

underlying

semantic

attributes


Sitting

vs
...


Having lunch in the cafeteria


Working
at a
desk



The

limit

of

supervised

learning

can

be

overcome

by

incorporating

human

knowledge


Office

working

vs
...


Motion
-
related
attributes: Sitting,
HandsOnTable
,


Sound
-
related
attributes:
PrinterSound
,
KeyboardSound
,
Conversations

8

Angus F.M. Huang

Challenges


Q1


Zero
-
shot

learning


To

learn

a

classifier

that

can

recognize

new

classes

that

have

never

appeared

in

the

training

dataset


Which

attributes

are

useful


From

Static

image

data

to

Sequential

sensor

data



Decompose

high
-
level

activities

into

combinations

of

semantic

attributes


Human

readable

term


Two
-
layer

attribute
-
based

learning

algorithm

9

Angus F.M. Huang

Challenges


Q2


Accuracy

reinforcement

by

user

feedback



Outlier
-
aware

active

learning

algorithm


Hybrid

stream/pool
-
based

sampling

scheme

10

Angus F.M. Huang

11

Activity

Recognition

Angus F.M. Huang

NuActiv

System Overview


Two

scenarios

of

activity

domain



Daily

life

activities


ex
.

ReadingAtHome

&

Driving




ReadingOnTrain


It

is

also

arguably

of

much

larger

variation

because

different

people

do

the

same

things

differently

(time,

situation,

)



Exercise

activities


Various

same

underlying

attributes

12

Angus F.M. Huang

System Architecture of
NuActiv


13

Angus F.M. Huang

System Design and Algorithms


Feature

Extraction


Semantic

Attribute
-
Based

Activity

Recognition


Active

Learning
:

Reinforcing

Activity

Recognition

using

Minimal

User

Feedback

14

Angus F.M. Huang

Feature
Extraction


The
mean and standard deviation
of sensor data in
dimension x, y, and z


Pairwise correlation
between each pair of
dimensions x, y, and z


Local slope
of sensor data in dimension x, y, and z
in using 1
st
-
order linear regression


Zero
-
crossing rate
in dimension x, y, and z

15

Angus F.M. Huang

Examples of features extracted
from acceleration data for each
exercise
activity

16

Angus F.M. Huang

Semantic Attribute
-
Based
Activity
Recognition


Activity

class

space,

y

=

{y
1
,y
2
,

,
y
k
}


Feature

space,

x

=

[X
1
,X
2
,

,
X
d
]


Classifier

function

f
:

x→y



y

=

{{y
1
,y
2
,

,
y
s
},{y
s+
1
,

y
s+u
}}

=

y
S

U

y
U


y
S
,

seen

classes


y
U
,

unseen

classes



Problem
:

How

to

recognize

an

unseen

class

y



y
U

?



Semantic

attribute

space,

a

=

[A
1
,A
2
,

,A
m
]

17

Angus F.M. Huang

Graphical representation of
semantic attribute
-
based
activity recognition

18

Angus F.M. Huang

Activity
-
Attribute Matrix


Encode

the

human

knowledge


M

x

N

matrix
;

M

activities,

N

attributes


Common
-
sense

knowledge,

domain

knowledge,

web

text

mining,

crowdsourcing

platforms


19

Angus F.M. Huang

Attribute Detection


To

train

a

set

of

attribute

detectors

so

that

we

are

able

to

infer

the

presence/absence

of

an

attribute

from

the

sensor

data

features


Training

data


what

we

need

is

one

set

of

positive

samples

and

another

set

of

negative

samples


Classifier


Support

Vector

Machine

(SVM)


After

training

phase,

we

have

a

trained

attribute

detector

for

each

attribute

specified

in

the

activity
-
attribute

matrix

20

Angus F.M. Huang

Attribute
-
Based Activity Classification


A

nearest
-
neighbor

classifier

is

used

to

recognize

the

high
-
level

activity



given

an

attribute

vector

generated

from

attribute

detectors


in

the

attribute

space



The

activity

recognizer



takes

an

attribute

vector

a

=

[
A
1
,A
2
,

,A
m
]

as

input


returns

the

closest

high
-
level

activity

y*

21

Angus F.M. Huang

Hybrid Feature/Attribute
-
Based
Activity Recognition


Transforming

low
-
level

features

to

mid
-
level

attributes

has

the

benefit

for

unseen

class

recognition


to

keep

the

advantages

of

both

feature
-
based

and

attribute
-
based


How

do

we

know

if

a

sample

belongs

to

a

seen

class

or

an

unseen

class?


a

sample

from

seen

class

-
>

similar

to

others


a

sample

from

unseen

class

-
>

like

anomaly



We

first

train

an

unseen

class

detector

using

the

one
-
class

SVM

classifier



where

only

the

positive

samples

are

given

to

the

classifier



After

using

the

unseen

class

detector,

we

then

do

a

hybrid

feature/attribute
-
based

activity

recognition

using

the

Algorithm
-
1

22

Angus F.M. Huang

23

Angus F.M. Huang

Active Learning: Reinforcing Activity
Recognition using Minimal User
F
eedback


Our

idea

is

simple
:


We

ask

a

user

for

labels

only

when

we

are

highly

uncertain

about

our

recognition

result


To

achieve

this
:


We

used

the

idea

of

uncertainty

sampling

in

the

field

of

active

learning



The

idea

of

active

learning

algorithms


a

machine

learning

algorithm

can

perform

better

with

less

training

data

if

it

is

allowed

to

choose

the

data

from

which

it

learns

24

Angus F.M. Huang


Hybrid

Sampling

Scheme


Stream
-
based

sampling


Pool
-
based

sampling



Uncertainty

Sampling

Metrics


Least

Confident


Minimum

Margin


Maximum

Entropy



But!


Outliers

get

higher

uncertainty

scores


Outliers

do

not

help

training

a

classifier



Outlier
-
Aware

Uncertainty

Sampling


To

select

samples

that

are

uncertain

but

not

outliers


Mean

similarity

between

this

sample

and

all

the

other

samples


Algorithm
-
2

25

Angus F.M. Huang

26

Angus F.M. Huang

Evaluation


System

Implementation


Nexus

S

4
G

phones

&

MotoACTV

wristwatches


Accelerometer

and

gyroscope


Android

app


JAVA,

SVM

classifier,

LibSVM

library


27

Angus F.M. Huang

28

Angus F.M. Huang


Datasets


Exercise

activity

dataset


20

subjects,

10

exercise

activities,

10

iterations,

dumbbell


phone
-
arm,

watch
-
wrist,

watch
-
hip


30

Hz

sampling

rate,

1

second

time

window

size

with

50
%

overlap


Public

dataset

on

daily
-
life

activities


A

published

dataset,

Technische

Universität

Darmstadt


34

classes,

one

subject,

7

days


Wearable

sensor
-
wrist&hip
,

100
Hz,

30

seconds

with

50
%

overlap


17

attributes


29

Angus F.M. Huang

Evaluation
Methodology


Leave
-
two
-
class
-
out

validation


Each

time

we

first

train

our

system

on

(
N
-
2
)

classes,

and

then

test

the

discriminative

capability

of

the

classifier

on

the

remaining

2

classes

that

were

"unseen"

by

the

system

during

the

training

process


True

positive

(TP),

True

negative

(TN),

False

positive

(FP),

False

negative

(FN)


Precision


the

percentage

of

times

that

a

recognition

result

made

by

the

system

is

correct


Recall


the

percentage

of

times

that

an

activity

performed

by

a

user

is

detected

by

the

system


F
1
-
score


a

integrated

measure

that

combines

both

30

Angus F.M. Huang

Unseen Activity Recognition Result


What

is

the

overall

precision/recall

of

unseen

activity

recognition

using

NuActiv
?

How

does

the

performance

vary

among

classes
?



Average

accuracy


79
%
,

for

overall

activities


80
-
90
%
,

for

five

activities

31

Angus F.M. Huang

The Impact of Number of Unseen
Classes And Comparison with Baseline


How

does

the

recognition

accuracy

change

with

the

number

of

unseen

classes
?


Baseline

approach


Random
-
guess

prediction

32

Angus F.M. Huang

Comparison of Different Attribute Detecto
rs


How

does

the

performance

vary

with

the

use

of

different

classification

algorithms

for

attribute

detectors
?


Decision

Tree

classifier,

Naive

Bayes

classifier,

and

k
-
Nearest

Neighbor

(k
-
NN)

classifier


k
-
NN

is

comparable

to

SVM


but

requires

the

storage

and

access

to

all

the

training

data


33

Angus F.M. Huang

Evaluation of The Importance of Attributes


How

to

select

attributes

based

on

their

importance

to

unseen

activity

recognition
?


Discriminability


how

well

can

an

attribute

discriminate

between

different

high
-
level

classes


ArmUp
,

ArmDown
,

ArmFwd


Detectability


how

accurately

can

we

detect

the

presence

or

absence

of

an

attribute


34

Angus F.M. Huang

Cross
-
User Activity Recognition Results


What

is

the

cross
-
user

performance,

i
.
e
.

when

the

users

in

the

training

set

are

different

from

those

in

the

testing

set?

Is

the

system

able

to

generalize

from

one

or

a

few

users

to

many

new

users
?


70
-
80
%

for

all


5

seen

users


65
%

for

1

seen


35

Angus F.M. Huang

Impact of Device Position on Attribute
Detection Accuracy


An

attribute

is

often

inherently

associated

with

a

characteristic

of

a

human

activity

or

a

motion

of

a

specific

part

of

human

body


to

understand

how

the

position

or

the

set

of

positions

at

which

the

sensors

are

placed

affects

the

attribute

detection

accuracy


The

upper

arm

sensor


achieves

better

and

stable

36

Angus F.M. Huang

Recognizing Unseen
New Daily Life Activi
ty


How

does

NuActiv

perform

on

recognizing

unseen

daily

life

activities
?


60
-
70
%

precision

and

recall

rate


Some

classes

have

High

recall

and

Low

precision,

and

vice

versa


Some

classes

do

not

have

clear

difference


Sitting
-
desk
-
activities

vs
.

Sitting
-
talking
-
on
-
phone


Low

precision

rate

37

Angus F.M. Huang

Comparison of Active Learning Algorithms


How

efficiently

can

the

system

reinforce

its

performance

using

active

learning?

How

does

the

performance

vary

with

different

active

learning

algorithms?


38

Angus F.M. Huang

Outlier
-
Aware Uncertainty Sampling Results


What

is

the

effect

of

outlier
-
aware

uncertainty

sampling

on

active

learning

algorithms
?

39

Angus F.M. Huang

Conclusion


NuActiv


Uses

semantic

attribute
-
based

learning

to

recognize

unseen

new

activity

classes

by

reusing

and

generalizing

the

attribute

model

learned

for

other

seen

activities



Outlier
-
aware

active

learning

algorithm


To

efficiently

improves

the

recognition

accuracy

of

the

system

using

minimal

user

feedback



79
%

recognition

accuracy

on

the

unseen

activity

recognition

40

Angus F.M. Huang

Angus Comments


Very depend on domain knowledge



Pity: it is unseen, not unknown



Can apply it to PLASH..


Unseen activities recognition


Unseen trip recommendation


Composite mobility
-
activity
recognition


car
-
driver / car
-
passenger, ...


Important for adaptive service provision



Important attributes evaluation for
PLASH’s
Urban Routing
&
Friend
Matchmaking



41

Angus F.M. Huang

42

Dumbbell Fly

Dumbbell Curl

Angus F.M. Huang

43

Bent
-
Over Row

Chest Press

Angus F.M. Huang

44

Daily Life Activities

Angus F.M. Huang

45