Mobile Robot Sensing

grassquantityΤεχνίτη Νοημοσύνη και Ρομποτική

15 Νοε 2013 (πριν από 3 χρόνια και 1 μήνα)

68 εμφανίσεις

Off
-
the
-
Shelf Vision
-
Based
Mobile Robot Sensing

Zhichao Chen

Advisor: Dr. Stan Birchfield

Clemson University


Vision in Robotics


A robot has to perceive its surroundings in
order to interact with it.


Vision is promising for several reasons:


Non
-
contact (passive) measurement


Low cost


Low power


Rich capturing ability

Project Objectives

Path following
:


Traverse a desired trajectory in
both indoor and outdoor environments.

1. “Qualitative vision
-
based mobile robot navigation”,
Proceedings of the
IEEE International Conference on Robotics and Automation (ICRA)
,

2006.

2. “Qualitative vision
-
based path following”,
IEEE Transactions on
Robotics
, 25(3):749
-
754, June 2009.

Person following
:

F
ollow a person in a
cluttered indoor environment.

“Person Following with a Mobile Robot Using Binocular Feature
-
Based

tracking”,
Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),

2007

Door detection
:

Build a semantic map of the
locations of doors as the robot drives down a
corridor.

“Visual detection of lintel
-
occluded doors from a single camera”,
IEEE
Computer Society Workshop on Visual Localization for Mobile Platforms
(in association with CVPR),2008
,


Motivation for Path Following


Goal:

Enable mobile robot to follow a desired
trajectory in both indoor and outdoor environments



Applications:

courier, delivery, tour guide, scout
robots



Previous approaches:


Image Jacobian
[Burschka and Hager 2001]


Homography
[Sagues and Guerrero 2005]



Homography (flat ground plane)
[Liang and Pears 2002]


Man
-
made environment
[Guerrero and Sagues 2001]


Calibrated camera
[Atiya and Hager 1993]


Stereo cameras
[Shimizu and Sato 2000]


Omni
-
directional cameras
[Adorni et al. 2003]

Our Approach to Path Following


Key intuition:

Vastly overdetermined system

(Dozens of feature points, one control decision)



Key result:

Simple control algorithm


Teach / replay approach using sparse feature points


Single, off
-
the
-
shelf camera


No calibration for camera or lens


Easy to implement (no homographies or Jacobians)

Preview of Results

milestone image

top
-
down view

current image

overview

Tracking Feature Points


Kanade
-
Lucas
-
Tomasi (KLT) feature tracker


Automatically selects features using eigenvalues of 2x2 gradient
covariance matrix





Automatically tracks features by minimizing sum of squared
differences (SSD) between consecutive image frames







Augmented with gain and bias to handle lighting changes






Open
-
source implementation
























W
d
J
I
x
d
x
d
x
2
2
2























W
d
J
I
x
d
x
d
x
2
)
2
(
2


[http://www.ces.clemson.edu/~stb/klt]

unknown displacement

gray
-
level images



W
T
Z
)
(
)
(
x
g
x
g
gradient of image

Teach
-
Replay

Teaching

Phase

start

destination

detect

features

track

features

Replay

Phase

track

features

compare

features

current feature

goal feature

initial feature

goal feature

Qualitative Decision Rule


Landmark

image plane

Feature is to the right


|u
Current
| > |u
Goal
|



“Turn right”

Feature has changed sides


sign(u
Current
)


sign(u
Goal
)



“Turn left”

No evidence

“Go straight”

feature

funnel lane

Robot at goal

u
Goal

u
Current

Feature is to the right



“Turn right”

Side change



“Turn left”

The Funnel Lane at an Angle


Landmark

image plane

Robot at goal

feature

α

α

α

funnel lane

No evidence

“Go straight”

A Simplified Example


Turn right”


Turn left”


Go straight”

Landmark

feature

Robot at goal

funnel lane

funnel lane

funnel lane

funnel lane


Go
straight”


Go
straight”


Go
straight”

The funnel Lane Created by
Multiple Feature Points

α

α

Landmark #1

Landmark #2

Landmark #3

Feature is to the right



“Turn right”

Side change



“Turn left”

No evidence

“Do not turn”

Qualitative Control Algorithm





D
C
D
C
u

sign
u

sign
and

u
u


Desired heading


Funnel constraints:

u
Goal















otherwise




0


u
u

and

0
(u

if


)
u
,
φ(u
,
u


max

u
u

and

0
(u

if


)
u
,
φ(u
,
u


min

θ
D
C
C
D
C
C
D
C
C
D
C
C
i
d


where
φ

is the signed distance between the u
C

and u
D

Incorporating Odometry






N
1
i
o
i
d
d
β)θ
(1
θ
N
1
β
θ
Desired heading


Desired heading

from odometry

Desired heading

from
i
th feature
point

N is the number of the features;

[0,1]
β

Overcoming Practical Difficulties

To deal with rough terrain:

Prior to comparison, feature coordinates are warped to

compensate for a non
-
zero roll angle about the optical axis by

applying the RANSAC algorithm.

To avoid obstacles:

The robot detects and avoids an obstacle by sonar, and the

odometry enables the robot to roughly return to the path.

Then the robot converges to the path using both odometry

and vision.

Experimental Results

Videos available at http://www.ces.clemson.edu/~stb/research/mobile_robot

milestone image

top
-
down view

current image

overview

Experimental Results

Videos available at http://www.ces.clemson.edu/~stb/research/mobile_robot

milestone image

top
-
down view

current image

overview

Experimental Results:

Rough Terrain

Experimental Results:

Avoiding an Obstacle

Experimental Results

Indoor

Outdoor

Imaging Source Firewire camera

Logitech Pro 4000 webcam

Project Objectives

Path following
:

Enable mobile robot to follow a
desired trajectory in both indoor and outdoor
environments.

1. “Qualitative vision
-
based mobile robot navigation”,
Proceedings of the

IEEE International Conference on Robotics and Automation (ICRA)
,

2006.

2. “Qualitative vision
-
based path following”,
IEEE Transactions on
Robotics
, 2009

Person following
:

Enable a mobile robot to
follow a person in a cluttered indoor environment
by vision.

“Person Following with a Mobile Robot Using Binocular Feature
-
Based

tracking”,
Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),

2007

Door detection
:

Detect doors as the robot
drives down a corridor.

“Visual detection of lintel
-
occluded doors from a single camera”,
IEEE
Computer Society Workshop on Visual Localization for Mobile Platforms
(in association with CVPR),2008



Motivation


Goal:

Enable a mobile robot to follow a person in a cluttered
indoor environment by vision.


Previous approaches:


Appearance properties: color, edges
.


[Sidenbladh et al. 1999, Tarokh and Ferrari 2003, Kwon et al. 2005]


Person has different color from background or faces camera.


Lighting changes.


Optical flow.

[Piaggio et al 1998, Chivil
ò

et al. 2004]


Drift as the person moves with out
-
of
-
plane rotation


Dense stereo and odometry.
[Beymer and Konolige 2001]


difficult to predict the movement of the robot (uneven surfaces,
slippage in the wheels).

Our approach


Algorithm:

Sparse stereo based on
Lucas
-
Kanade feature tracking.



Handles:


Dynamic backgrounds.


Out
-
of
-
plane rotation.


Similar disparity between the person and
background.


Similar color between the person and
background.

System overview


Detect 3D features of the scene


( Cont. )



Features are selected in the left image I
L

and matched in the right image I
R.

Left image

Right image

The size of each square indicates the

horizontal disparity of the feature.

System overview


Detecting Faces


The Viola
-
Jones frontal face detector is applied.


This detector is used both to initialize the system
and to enhance robustness when the person is
facing the camera.

Note: The face detector is not necessary in our system.

Overview of Removing Background

2) using the estimated motion
of the background.


3) using the estimated
motion of the person

1) using the known disparity
of the person in the
previous image frame.

Discard features for which

where is the known disparity of the person in the previous frame,

and is the disparity of a feature at time
t

.

Remove Background

Step 1: Using the known disparity

d
t
t
d
d



|
|
~
t
d
~
t
d
Original features

Foreground features in step 1

Background features

Remove Background

Step 2: Using background motion


Estimate the motion of the background by computing a

4
×

4 affine transformation matrix H between two image
frames at times t and t + 1:


(1)

1
f
1
f
i
1
t
i
t














H


Random sample consensus (RANSAC) algorithm is


used to yield dominant motion.

Foreground features with similar

disparity in step 1

Foreground features after step 2

Remove Background

Step 3: Using person motion



Similar to step 2, the motion model of the person is
calculated.


The size of the person group should be the biggest.


The centroid of the person group should be proximate to
the previous location of the person.

Foreground features after step 2

Foreground features after step 3

System overview


System overview


Experimental Results

Video

Project Objectives

Path following
:

Enable mobile robot to follow a
desired trajectory in both indoor and outdoor
environments.

1. “Qualitative vision
-
based mobile robot navigation”,
Proceedings of the

IEEE International Conference on Robotics and Automation (ICRA)
,

2006.

2. “Qualitative vision
-
based path following”,
IEEE Transactions on
Robotics
, 2009


Person following
:

Enable a mobile robot to
follow a person in a cluttered indoor environment
by vision.

“Person Following with a Mobile Robot Using Binocular Feature
-
Based

tracking”,
Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),

2007

Door detection
:

Detect doors as the robot
drives down a corridor.

“Visual detection of lintel
-
occluded doors from a single camera”,
IEEE
Computer Society Workshop on Visual Localization for Mobile Platforms
(in association with CVPR),2008


Motivation for Door Detection

Topological map

Metric map

Either way, doors are

semantically meaningful


landmarks

Previous Approaches to
Detecting Doors

fuzzy logic

[Munoz
-
Salinas et al. 2004]

color segmentation

[Rous et al. 2005]

neural network

[Cicirelli et al 2003]

Limitations:



require different colors for doors and walls



simplified environment (untextured floor, no reflections)



limited viewing angle



high computational load



assume lintel (top part) visible


Range
-
based approaches

sonar [Stoeter et al.1995], stereo [Kim et al. 1994], laser [Anguelov et al. 2004]


Vision
-
based approaches

lintel

post

What is Lintel
-
Occluded?

Lintel
-
occluded


post
-
and
-
lintel architecture


camera is low to ground


cannot point upward b/c obstacles

Our Approach

,
)
(
sign
)
(
N
1
n
n












x
h
x
n

n

,
)
(
sign
)
(
N
1
n
n












x
h
x
n

n

Assumptions:

• Both door posts are visible

• Posts appear nearly vertical

• The door is at least a certain
width

Key idea: Multiple cues are necessary for robustness (pose, lighting, …)

Video

Pairs of Vertical Lines

1.
Edges detected by Canny

2.
Line segments

detected by modified Douglas
-
Peucker algorithm

3.
Clean up (merge lines across small gaps, discard short lines)

4.
Separate vertical and non
-
vertical lines

5.
Door candidates
given by all the vertical line pairs whose spacing
is within a given range

Canny edges

detected lines

vertical lines

non
-
vertical lines

Homography

In the world

In the image











































1
1
1
'
'
33
32
31
23
22
21
13
12
11
y
x
h
h
h
h
h
h
h
h
h
y
x
H
y
x

(x’, y’)

(x, y)

Prior Model Features:

Width and Height

3.0



y
x
Principal point

(0,0)

(0,0)

(0,y)

(x, y)

(x,0)

?

?

1


CH
H
T
An Example

As the door turns, the bottom corner traces an ellipse

(projective transformation of circle is
ellipse
)

But not horizontal

Data Model (Posterior) Features

Image gradient along edges (
g
1
)

Placement of top and


bottom edges (
g
2
, g
3
)

Color (
g
4
)

texture (
g
5
)

Kick plate (
g
6
)

Vanishing point (
g
7
)

and two more…

Data Model Features (cont.)

Intensity along the line

darker

(light off)

positive

negative

brighter

(light on)

no gap

Bottom gap(g
8
)

Data Model Features (cont.)

wall

wall

door

Slim “U”

intersection line

of wall and floor

extension of

intersection line

bottom door edge

vertical door lines

ε

L
left

L
Right

floor

Concavity(g
9
)

The strong classifier

Two Methods to Detect Doors

}
1
,
1
{
,
classifier

weak

each
for

decison

hard

the

is


where
,
0
)
(
sgn
)
(
ψ
1














n
n
N
n
n
n
h
h
x
h
x

Adaboost

Weights of

features

Bayesian formulation

Training images

)
d
(
)
d
|
(
)
d
(
prior
data




I
E
Weights of

features

(yields better results)

Bayesian Formulation


)
d
(
)
d
|
(
)
|
d
(
p
I
p
I
p

door

image

Taking the log likelihood,

)
d
(
)
d
|
(
)
d
(
prior
data




I
E
)
d
(
)
d
|
(
)
|
d
(
p
I
p
I
p

door

image





data
1
data
)
d
(
)
,
d
(
N
i
i
i
f
I





prior
1
prior
)
d
(
)
d
(
N
j
j
j
g

Data model

Prior model

]
1
,
0
[
)
d
(
g

and

(d)

where
j

i
f
MCMC and DDMCMC


Markov Chain Monte Carlo (MCMC) is used here to maximize
probability to detect door
(like random walk through state space of
doors)


Data driven MCMC (DDMCMC) is used to speed up computation


doors appear more frequently at the position close to the vertical lines


the top of the door is often occluded or a horizontal line closest to the top


the bottom of the door is often close to the wall/floor boundary.


Experimental Results: Similar
or Different Door/Wall Color

Experimental Results: High
Reflection / Textured Floors

Experimental Results:
Different Viewpoints

Experimental Results:
Cluttered Environments

Results

25 different buildings

600 images:



100 training



500 testing


91.1% accuracy with

0.09 FP per image


Speed: 5 fps 1.6GHz
(unoptimized)

False Negatives and Positives

distracting reflection

concavity
and bottom
gap tests fail

strong

reflection

concavity
erroneously
detected

two vertical lines


unavailable

distracting reflection

Navigation in a Corridor


Doors were detected and tracked from frame to frame.


Fasle positives are discarded if doors were not
repeatedly detected.

Video

Conclusion


Path following


Teach
-
replay, comparing image coordinates of feature points (no
calibration)


Qualitative decision rule (no Jacobians, homographies)


Person following


Detects and matches feature points between a stereo pair of
images and between successive images.


RANSAC
-
based procedure to estimate the motion of each region


Does not require the person to wear a different color from the
background.


Door detection


Integrate
a variety of features of door


Adaboost training and DDMCMC.


Future Work


Path following


Incorporating higher
-
level scene knowledge to enable
obstacle avoidance and terrain characterization


Connecting multiple teaching paths in a graph
-
based
framework to enable autonomous navigation between
arbitrary points.


Person following


Fusing the information with additional appearance
-
based
information ( template or edges) .


Integration with EM tracking algorithm
.


Door detection


Calibrate the camera to enable pose and distance measurements
to facilitate the building of a geometric map.


Integrated into a complete navigation system that is able to drive
down a corridor and turn into a specified room