# lec04_basic_segmx

AI and Robotics

Oct 29, 2013 (4 years and 6 months ago)

114 views

The University of

Ontario

CS
4487/9587

Algorithms for Image Analysis

Segmentation (2D)

Acknowledgements: Alexei Efros, Steven Seitz

The University of

Ontario

CS 4487/9587
Algorithms for Image Analysis

Segmentation (2D)

Blobs

Need for blobs

Extracting blobs

Thresholding, region growing, mean
-
shift

Minimum spanning trees

Object extraction

Intelligent scissors (also known as
live
-
wire
)

Snakes

Region+boundary methods

Local, greedy, and global processing

Extra Reading: Sonka at.al. Ch. 5

Gonzalez and Woods, Ch. 10

The University of

Ontario

HW assignment 1

2D image segmentation

Live
-
wire (based on
Dijkstra

algorithm for shortest

paths on a graph)

Implement and experiment on
many images

Compare
with local/greedy methods

Some useful code libraries and sample images will
be posted within a couple of days

The University of

Ontario

Goal: Extract “Blobs”

What are “blobs”?

Regions of an image that are somehow coherent

Why?

Object extraction, object removal, compositing, etc.

…but are “blobs” objects?

No, not in general

The University of

Ontario

Blob’s coherence

Simplest way to define blob coherence is as
similarity in brightness or color:

The tools become blobs

The house, grass, and sky make

different blobs

The University of

Ontario

Why is this useful?

AIBO

RoboSoccer

(VelosoLab)

The University of

Ontario

Ideal Segmentation

The University of

Ontario

Result of Segmentation

The University of

Ontario

Thresholding

Basic segmentation operation:

mask(x,y) = 1 if im(x,y) > T

mask(x,y) = 0 if im(x,y) < T

T is threshold

User
-
defined

Or automatic

Same as
histogram
partitioning:

The University of

Ontario

Sometimes works well…

What are potential

Problems?

The University of

Ontario

…but more often not

The University of

Ontario

Region growing

Is this same as global threshold?

K

(initial seed(s))

p

in
K

their neighbors
q

if
|Ip
-
Iq| < T

Repeat until nothing changes

The University of

Ontario

Region growing

The University of

Ontario

What can go wrong

with region growing ?

Region growth may “leak”
through a single
week spot

in the boundary

The University of

Ontario

Region growing

See region
leaks

into sky
due to a weak boundary
between them

The University of

Ontario

Due to Pedro
Felzenszwalb

and Daniel Huttenlocher

Motivating example

This image has three
perceptually distinct regions

Difference along border between A and
B is less then differences within C

A

C

B

Q: Where would
image thresholding

fail?

Q: Where would
region growing

fail?

A: Region A would be divided in two sets and

region C will be split into a large number

of arbitrary small subsets

A: Either A and B are merged or region C is

split into many small subsets

Also, B and C are merged

The University of

Ontario

Color
-
Based Blob Segmentation

Automatic Histogram Partitioning

Given image with N colors, choose K

Each of the K colors defines a region

not necessarily contiguous

Performed by computing color histogram, looking for
modes

The University of

Ontario

Finding Modes in a Histogram

How Many Modes Are There?

Easy to see, hard to
compute

The University of

Ontario

Mean Shift [
Comaniciu & Meer]

1.
Initialize random seed, and fixed window

2.
Calculate center of gravity ‘
x
’ of the window (
the“mean
”)

3.
Translate the search window to the mean

4.
Repeat Step 2 until convergence

Iterative
Mode Search

x

o

x

x

mode

The University of

Ontario

Mean
-
Shift

The University of

Ontario

Mean
-
shift results

More Examples
:
http://www.caip.rutgers.edu/~comanici/segm_images.html

The University of

Ontario

Issues:

Although often useful, all these approaches
work only some of the time, and are considered
rather “hacky”.

Can’t even handle our tiger:

Problem is that blobs != objects!

The University of

Ontario

Extracting objects

How could this be done?

The University of

Ontario

Extracting objects

Many approaches proposed

color cues

region cues

contour cues

We will consider a few of these

Today:

Intelligent Scissors (contour
-
based)

E. N. Mortensen and W. A. Barrett,
Intelligent Scissors for Image
Composition
, in ACM Computer Graphics (SIGGRAPH `95), pp. 191
-
198,
1995

The University of

Ontario

Intelligent Scissors

The University of

Ontario

Intelligent Scissors

Approach answers a basic question

Q: how to find a path from seed to mouse that follows
object boundary as closely as possible?

A: define a path that stays as close as possible to edges

The University of

Ontario

Intelligent Scissors

Basic Idea

Define edge score for each pixel

edge pixels have low cost

Find lowest cost path from seed to mouse

seed

mouse

Questions

How to define costs?

How to find the path?

The University of

Ontario

Path Search (basic idea)

Graph Search Algorithm

Computes minimum cost path from seed to
all

other

pixels

Note: diagonal “paths”
are scaled by a factor
.

Why?

3 sets of nodes

Free

Active

Done

The University of

Ontario

Segmentation should be

Invariant
to Image Rotation

After object rotation

L

Path’s cost along the top boundary

Path’s cost along the top
-
left boundary

2

2

2

2

2

2

2

2

2

2

2

2

The University of

Ontario

can be assigned directly to graph edges

Graph

node for every pixel
p

link between every adjacent pair of pixels
e=(p,q)

cost
w(e)

Note: each

has a cost

this is a little different than the figure before where
each pixel (graph node) had a cost

p

q

Treat the image as a graph

The University of

Ontario

Defining edge costs

Treat the image as a graph

Want to hug image edges:
how to define cost of a link?

p

q

the link should follow the intensity edge

want intensity to change rapidly to the link

the cost of edge should be small when the
difference of intensity to the link is large

T

T

The University of

Ontario

Defining edge costs

p

q

First, estimate image derivative in the direction
orthogonal to the edge

Use finite differences (or cross
-
correlation filters)

1

-
1

-
1

-
1

1

1

The University of

Ontario

Defining edge costs

p

q

Second, use some penalty function
g( )

assigning low
penalty to large directional derivatives and large
penalty to small derivatives

The University of

Ontario

Defining edge costs

p

q

Finally, cost of en edge
e

should be local contrast
score adjusted by the edge length

Why?

The University of

Ontario

Defining edge costs

When computing the shortest path we approximate a
contour
C

minimizing continuous geometrically
meaningful functional

Cost of one edge

Cost of a PATH

Integral of contrast penalty along the contour

The University of

Ontario

(see Cormen et.al. “
Introduction to Algorithms
”, p.595)

Dijkstra’s shortest path algorithm

0

5

3

1

3

3

4

9

2

1.
init node costs to

, set
p

= seed point, cost(
p
) =
0

2.
expand
p

as follows:

for each of
p
’s neighbors
q

that are not expanded

set cost(
q
) = min( cost(
p
) +
c
pq
, cost(
q
) )

ALGORITHM

3 sets of nodes

Free

Active

Done

The University of

Ontario

Dijkstra’s shortest path algorithm

4

1

0

5

3

3

2

3

9

5

3

1

3

3

4

9

2

1

1

1.
init node costs to

, set
p

= seed point, cost(
p
) =
0

2.
expand
p

as follows:

for each of
p
’s neighbors
q

that are not expanded

set cost(
q
) = min( cost(
p
) +
c
pq
, cost(
q
) )

if
q
’s cost changed, make
q

point back to
p

put
q

on the ACTIVE list (if not already there)

ALGORITHM

3 sets of nodes

Free

Active

Done

The University of

Ontario

Dijkstra’s shortest path algorithm

4

1

0

5

3

3

2

3

9

5

3

1

3

3

4

9

2

1

5

2

3

3

3

2

4

1.
init node costs to

, set
p

= seed point, cost(
p
) =
0

2.
expand
p

as follows:

for each of
p
’s neighbors
q

that are not expanded

set cost(
q
) = min( cost(
p
) +
c
pq
, cost(
q
) )

if
q
’s cost changed, make
q

point back to
p

put
q

on the ACTIVE list (if not already there)

3.
set
r

= node with minimum cost on the ACTIVE list

4.
repeat Step 2 for
p

=
r

ALGORITHM

3 sets of nodes

Free

Active

Done

The University of

Ontario

Dijkstra’s shortest path algorithm

3

1

0

5

3

3

2

3

6

5

3

1

3

3

4

9

2

4

3

1

4

5

2

3

3

3

2

4

1.
init node costs to

, set
p

= seed point, cost(
p
) =
0

2.
expand
p

as follows:

for each of
p
’s neighbors
q

that are not expanded

set cost(
q
) = min( cost(
p
) +
c
pq
, cost(
q
) )

if
q
’s cost changed, make
q

point back to
p

put
q

on the ACTIVE list (if not already there)

3.
set
r

= node with minimum cost on the ACTIVE list

4.
repeat Step 2 for
p

=
r

ALGORITHM

3 sets of nodes

Free

Active

Done

The University of

Ontario

Dijkstra’s shortest path algorithm

3

1

0

5

3

3

2

3

6

5

3

1

3

3

4

9

2

4

3

1

4

5

2

3

3

3

2

4

2

1.
init node costs to

, set
p

= seed point, cost(
p
) =
0

2.
expand
p

as follows:

for each of
p
’s neighbors
q

that are not expanded

set cost(
q
) = min( cost(
p
) +
c
pq
, cost(
q
) )

if
q
’s cost changed, make
q

point back to
p

put
q

on the ACTIVE list (if not already there)

3.
set
r

= node with minimum cost on the ACTIVE list

4.
repeat Step 2 for
p

=
r

ALGORITHM

3 sets of nodes

Free

Active

Done

The University of

Ontario

Path Search (basic idea)

A

B

Dijkstra algorithm

-

processed nodes (distance to A is known)

-

active nodes (front)

-

active node with the smallest distance value

The University of

Ontario

Dijkstra’s shortest path algorithm

Properties

It computes the minimum cost path from the seed to every
node in the graph. This set of minimum paths is represented
as a
tree

Running time, with N pixels:

O(N
2
) time if you use an active list

O(N log N) if you use an active priority queue (heap)

takes < second for a typical (640x480) image

Once this tree is computed once, we can extract the optimal
path from any point to the seed in O(N) time.

it runs in real time as the mouse moves

What happens when the user specifies a new seed?

The University of

Ontario

Livewire extensions

Directed graphs

Restricted search space

Restricted domain (e.g. near
a priori

model)

Restricted backward search

Different edge weight functions

Image
-
Edge strength

Image
-
Edge Curvature

Proximity to known approximate model/boundary

Multi
-
resolution processing

The University of

Ontario

Results

http://www.cs.washington.edu/education/courses/455/03wi/projects/project1/artifacts/index.html