from Sensor Data

lettuceescargatoireAI and Robotics

Nov 7, 2013 (3 years and 11 months ago)

68 views

Recognizing Human Activity
from Sensor Data

Henry Kautz

University of Washington

Computer Science & Engineering

graduate students
:
Don Patterson, Lin Liao

CSE faculty
: Dieter Fox, Gaetano Borriello

UW School of Medicine
: Kurt Johnson

Intel Research
: Matthai Philipose, Tanzeem Choudhury

Converging Trends…

Pervasive sensing infrastructure

GPS enabled phones

RFID tags on all consumer products

Wireless motes

Breakthroughs in core artificial intelligence

After “AI boom” fizzled, basic science went on…

Advances in algorithms for probabilistic reasoning and
machine learning

Bayesian networks

Stochastic sampling

Last decade: 10 variables


1,000,000 variables

Healthcare crisis

Epidemic of Alzheimer’s Disease

Deinstitutionalization of the cognitively disabled

Nationwide shortage of caretaking professionals

...An Opportunity

Develop technology to

Support independent living by people
with cognitive disabilities

At home

At work

Throughout the community

Improve health care

Long term monitoring of activities of daily
living (ADL’s)

Intervention
before

a health crisis


The University of Washington
Assisted Cognition Project

Synthesis of work in

Ubiquitous computing

Artificial intelligence

Human
-
computer interaction


ACCESS

Support use of public transit

CARE

ADL monitoring and assistance


This Talk

Building models of everyday plans and
goals

From sensor data

By mining textual description

By engineering commonsense knowledge

Tracking and predicting a user’s behavior

Noisy and incomplete sensor data

Recognizing user errors

First steps toward proactive assistive technology

ACCESS

Assisted Cognition in Community, Employment, &
Support Settings


Supported by

The National Institute on Disability & Rehabilitation
Research (NIDDR)

The National Science Foundation (NSF)


Learning & Reasoning About
Transportation Routines

Task

Given a data stream from a
wearable GPS unit...

Infer

the user’s location and mode of
transportation (foot, car, bus, bike, ...)

Predict

where user will go

Detect

novel behavior

User errors?

Opportunities for learning?

Why Inference Is Not Trivial

People don’t have wheels

Systematic GPS error

We are not in the woods

Dead and semi
-
dead zones

Lots of multi
-
path propagation

Inside of vehicles

Inside of buildings

Not just location tracking

Mode, Prediction, Novelty

GPS Receivers We Used

Nokia 6600 Java Cell
Phone with Bluetooth
GPS unit

GeoStats wearable
GPS logger


Geographic Information
Systems

Bus routes and bus stops

Data source: Metro GIS


Street map

Data source: Census 2000

Tiger/line data


Architecture

Learning Engine

Inference Engine

GIS

Database



Goals




Paths



Modes



Errors

Probabilistic Reasoning

Graphical model:

Dynamic Bayesian network

Inference engine:

Rao
-
Blackwellised particle filters

Learning engine:

Expectation
-
Maximization (EM) algorithm

Graphical Model (Version 1)

Transportation Mode

Velocity

Location

Block

Position along block

At bus stop, parking lot, ...?

GPS Offset Error

GPS signal

Rao
-
Blackwellised Particle
Filtering

Inference: estimate current state
distribution given all past readings

Particle filtering

Evolve approximation to state distribution using
samples (particles)

Supports multi
-
modal distributions

Supports discrete variables (e.g.: mode)

Rao
-
Blackwellisation

Each particle includes a Kalman filter to represent
distribution over positions

Improved accuracy with fewer particles

Tracking

blue = foot

green = bus

red = car

Learning

User model = DBN parameters

Transitions between blocks

Transitions between modes

Learning: Monte
-
Carlo EM

Unlabeled data

30 days of one user, logged at 2
second intervals (when outdoors)

3
-
fold cross validation




Results

Model

Mode Prediction
Accuracy

Decision Tree

(supervised)

55%

Prior w/o bus info

60%

Prior with bus info

78%

Learned

84%

Probability of correctly
predicting the future

City Blocks

Prediction Accuracy

How can we
improve predictive
power?

Transportation Routines

B

A

Goals

work, home, friends, restaurant, doctor’s, ...

Trip segments

Home

to
Bus stop A

on
Foot

Bus stop

A

to
Bus stop B

on
Bus

Bus stop B

to
workplace

on
Foot


Work

“Learning & Inferring Transportation Routines”, Lin Liao, Dieter
Fox, & Henry Kautz,
AAAI
-
2004 Best Paper Award

Hierarchical Model

Transportation mode

x=<Location, Velocity>

GPS reading

Goal

Trip segment

x
k
-
1

z
k
-
1

z
k

x
k

m
k
-
1

m
k

t
k
-
1

t
k

g
k
-
1

g
k

Hierarchical Learning

Learn flat model

Infer goals

Locations where user is often motionless

Infer trip segment begin / end points

Locations with high mode transition probability

Infer trips segments

High
-
probability single
-
mode block transition
sequences between segment begin / end
points

Perform hierarchical EM learning

Inferring Goals

Inferring Trip Segments

Going to work


Going home


Correct goal
and route
predicted 100
blocks away

Novelty & Error Detection

Approach: model
-
selection

Run several trackers in parallel

Tracker 1: learned hierarchical model

Tracker 2: untrained flat model

Tracker 3: learned model with clamped final
goal

Estimate the likelihood of each tracker given
the observations

Detect User Errors

Untrained Trained Instantiated

Application:

Opportunity
Knocks

Demonstration (by Don
Patterson) at
AAHA
Future of Aging Services
,
Washington, DC, March,
2004

CARE

Cognitive Assistance in Real
-
world Environments


supported by the Intel Research Council


Learning & Inferring Activities
of Daily Living

Research Hypothesis

Observation: activities of daily
living involve the manipulation of
many physical objects

Cooking, cleaning, eating, personal
hygiene, exercise, hobbies, ...

Hypothesis: can recognize
activities from a time
-
sequence of
object “touches”

Such models are robust and easily
learned or engineered


Sensing Object Manipulation

RFID: Radio
-
frequency ID
tags

Small

Semi
-
passive

Durable

Cheap

Where Can We Put Tags?

How Can We Sense Them?

coming... wall
-
mounted “sparkle reader”

Example Data Stream

Making Tea

Building Models

Core ADL’s amenable to classic
knowledge engineering

Open
-
ended, fine
-
grained models:
infer from natural language texts?

Perkowitz
et al
., “Mining Models of
Human Activities from the Web”,
WWW
-
2004

Experimental Setup

Hand
-
built library of 14
ADL’s

17 test subjects

Each asked to perform
12 of the ADL’s

Data not segmented

No training on
individual test subjects


Activity

Prior Work

CARE

Accuracy/Recall

Personal Appearance



92/92

Oral Hygiene



70/78

Toileting



73/73

Washing up



100/33

Appliance Use



100/75

Use of Heating



84/78

Care of clothes and linen



100/73

Making a snack



100/78

Making a drink



75/60

Use of phone



64/64

Leisure Activity



100/79

Infant Care



100/58

Medication Taking



100/93

Housework



100/82

95/84

General Solution

Quantitative Results

Point Solution

Quantitative Results

Point Solution

Anecdotal Results

General Solution

Anecdotal Results

Pervasive Computing, Oct
-
Dec 2004

Current Directions


Affective & physiological state

agitated, calm, attentive, ...

hungry, tired, dizzy, ...

Interactions between people

Human Social Dynamics

Principled human
-
computer interaction

Decision
-
theoretic control of interventions

Why Now?

A goal of much work of AI in the 1970’s
was to create programs that could
understand the narrative of ordinary
human experience

This area pretty much disappeared

Missing probabilistic tools

Systems not able to
experience

world

Lacked focus


“understand” to what end?

Today: tools, grounding, motivation

Challenge to Nanotechnology
Community

Current sensors detect physical or
physiological state: user mental
state must be
indirectly inferred

To what can extend can
nanotechnology afford
direct
access to a person’s emotions and
intentions?