Normalized Cuts and
Image Segmentation
Amir Lev

Tov
IDC, Herzliya
Advanced Topics in Computer Vision
[
1
] Normalized Cuts and Image
Segmentation, Shi and Malik, IEEE Conf.
Computer Vision and Pattern Recognition,
1997
.
[
2
] Normalized Cuts and Image
Segmentation, Shi and Malik, IEEE
Transactions on pattern analysis and
machine intelligence, Vol
22
, No
8
,
2000
Main References
More References
[
3
] Weiss Y. Segmentation using eigenvectors: a
unifying view. Proceedings IEEE International
Conference on Computer Vision,
1999
.
[
4
] Ng A.Y., Jordan , M.I., and Weiss Y, On
Spectral Clustering: Analysis and an algorithm,
NIPS
2001
[
5
] Rayleigh’s Quotient, Nail Gumerov,
2003
[
6
] Wu and Leahy, an optimal graph theoretic
approach to data clustering, PAMI,
1993
Mathematical Introduction
Definition: is an
Eigen Value
of n x n
matrix A, if there exist a non

trivial vector
such that:
That vector is called
Eigen Vector
of A
corresponding to the Eigen Value
All Eigenvectors correspond to different
Eigenvalues, are mutually linearlly
independent (Orthogonal set).
Mathematical Introduction
Matrix A is called
Hermitian
if
Where A* is the
conjugate transpose
of A:
Real matrix is Hermitian
Symmetric
Let A be a Hermitian matrix.
Then
is called the
Rayleigh’s Quotient
of A.
Mathematical Introduction
For real matrices the definition becomes:
Where A is just symmetric.
Mathematical Introduction
Theorem
: Rayleigh’s Quotient gets its
minimum value at A’s minimal eigenvalue,
and the corresponding eigenvector
achieve this minimum.
Moreover: if A has n eigenvalues
then R(A,v) has n stationary points
achieved at their eigenvectors
respectively.
Mathematical Introduction
The
Generalized Rayleigh’s Quotient
is:
where B is hermitian (real

>symmetric)
and positive definite matrix
(all its eigenvalues are non

zero positives).
Minimum achieved by solving
:
Segmentation Introduction
Segmentation Introduction
Problem: Divide an image into subsets of
pixels (Segments).
Some methods:
Thresholding
Region Growing
K

means
Mean

Shift
Use of changes in color, texture etc.
Contours
Segmentation Introduction
The problem is not very well defined, for
example, how many groups are in the
picture?
4
? Maybe
3
?
2
? Or even every X
Segmentation Introduction
In order to get good Segmentation:
Low level cues like colors, texutre etc.
High level knowledge as global impression
from the picture (top

>down).
Need good similarity function
Number of segments is not known in
advance
The Graph partitioning method
Main Idea:
Model the image by a graph G=(V,E).
Assign similarity values to edges weights.
Find a cut in G of minimal value, which yield
partition of V into two subsets.
•
Matrix representation of computations.
•
Using Linear Algebra tools and Spectral Analysis
to solve the new minimization problem.
Recursively repartition the subpartitions.
Graph Modeling
The Graph G=(V,E)
Nodes:
•
Pixels
•
Some other higher level features
Edges:
•
Between every pair of nodes in V
Weights
:
•
Weight w(i,j) is function of similarity between node
i and j.
Graph Modeling
Objective
Partition the set of vertices into disjoint sets
Number of segments m is not known.
Cut:
•
Case of m=
2
, Bi

partition of V into A and B:
The Cut Value is:
•
The optimal cut is the one that minimizes its value
Minimun Cut
Wu and Leahy[
1993
]:
Use the mentioned cut criteria
Partition G into k subgraphs recursively
Minimize the maximum cut value
Produce good segmentation on some of the
images
Min Cut

The Problem
It is not the best cut !
Favors cutting small sets of isolated nodes:
Normalized Cut [Shi,Malick,
1997
]
Normalize the cut value with the volume of
the partition:
Where
Normalized Cut
Properties:
Sets with weak connections
G
et low Ncut value.
High Association within Sets
G
et low Ncut value.
But

small sets are panalized with high Ncut value.
Normalized Association
Normalized Association:
Naturally related criterions:
Computing the Optimal Cut
Given partition of nodes from V into A,B :
Let be an N=V dimensional indicator for A,
i.e
if node i is in A, and

1
otherwise
Let
be the total connection
from node i to all other nodes.
Rewrite:
Computing the Optimal Cut
Objective: Transform
into Rayleigh’s Quotient

like expression:
Matrix Representation
Let D be an N x N diagonal matrix
with d on its diagonal:
Let W be an N x N symmetrical affinity matrix with
Matrix Representation
Let k be the Ratio between degree of A to V:
Let be an N x
1
vector of all ones
Note: (
1
+x)/
2
and (
1

x)/
2
are indicators for
and
respectively
Matrix Representation
We can rewrite our expression as:
Ncut(A,B) = Ncut(x) =
Matrix Representation
It can be shown that the previous
expression is equal to:
Where
is ratio between degree of A
to degree of B
Matrix Representation
Setting new indicator y=(
1
+x)

b(
1

x)
we get
the Constraint:
Matrix Representation
Denominator:
Finding the Minimum
Putting the last two expression together we get
the Rayleigh’s quotient:
With the conditions:
Minimum achieved by finding the minimal Eigenvalue of
the system(
1
) (relaxing y to take on real values)
Corresponding Eigenvector will be in fact indicator vector
for nodes in the segment (A)
Finding the Minimum
But
–
we have two constraints:
We’ll see that the first one is satisfied:
Replacing y by
we get the standard
eigensystem (
2
):
is an eigenvector of it,with an eigenvalue of
0
.
Since the Laplaician matrix (D

W) is symmetric semi

positive definite, so that the new system
Finding the Minimum
Thus, z
0
is the smallest eigenvector of (
2
)
Also known: all the eigenvectors of
equation (
2
) are orthogonal to each other
In particular, z
1
, the second smallest
eigenvector is orthogonal to z
0
Finding the Minimum
In terms of our original system
)
1
):
Is the smallest eigenvector with
Where y
1
is the
2
nd
smallest
eigenvector of (
1
)
The
1
st
constraint is automatically satisfied:
Finding the Minimum
In Rayleigh’s Quotient, under the constraint that
z is orthogonal to the j

1
smallest eigenvectors,
the quotient is minimized by the next smallest
eigenvector zj and its minimum is the eigen
value
We get:
Consequently:
Finding the Minimum
Conclusion: the
2
nd
smallest eigenvector of (
1
)
is the real solution to our Normalized Cut
problem.
What about the
2
nd
constraint that y takes on
discrete values??
Solving the discrete problem is NP

Complete
Solution
–
approximate the continuous solution by
splitting the vector coordinates at different
thresholds, choosing the one that gives the best
NCut value.
Complexity
Wait!
What about the original graph problem ?
MinCut
–
Has Polynomial

Time algorithm
by the MaxFlow algorithm.
Impractical for images
Normalized Cut
–
NP

Complete
Need fast approximations
Complexity
Solving standard eigenvalue problem
Impractical for segmenting large number of
pixels
Special properties of our problem:
The graph often locally connected=>sparse
matrix
Only the top eigenvectors are needed
Low precision requirements
Using Lanczos eigensolver

Repartitioning
Recursively apply the above method to each of
the partitions
Subject to some “stability” criteria:
•
Create sub partitions by varying the splitting point around the
optimal value and check if Ncut value change much
Until certain Ncut threshold exceeded
Another approach: Use high order eigenvectors
Pros: more discriminative information
Cons: according to Shi&Malik, Approximation error
accumulates with every eigenvector taken
Summary of the Algorithm
1.
Given features, construct the graph
2.
Solve
for eigenvectors with the
smallest eigenvalues
3.
Use the eigenvector with the
2
nd
smallest
eigenvalue to bipartition the graph
Find the splitting point that minimizes Ncut
4.
Check stability and Ncut threshold to decide
whether
to divide the current partition.
5.
Recursively repartition the segmented parts if
necessary
Experiments
Pixels as Graph nodes
Weight Function:
X(i)
–
Spatial location of node I
F(i)
–
Feature vector based on Intensity, Color
or Texture information at node i
Experiments
Example of F(i):
F(i) =
1
, For point set segmentation
F(i) = I(i), Intensity value, for segmenting
brightness images
Experiments
Point set
Taken from [
1
]
Experiments
Synthetic image of corner
Taken from [
1
]
Experiments
Taken from [
1
]
Experiments
“Color” image
Taken from [
1
]
Experiments
Without well defined boundaries:
Taken from [
1
]
Experiments
Texture segmentation
Different orientation stripes
Taken from [
1
]
A little bit more..
Taken from [
2
]
A little bit more..
High order eigenvectors
Taken from [
2
]
High order eigenvectors
Taken from [
2
]
1
st
Vs
2
nd
Eigenvectors
Taken from [
3
]
Summery
Treat the problem as graph partitioning
The new idea:
Normalized Cut instead of Regular Cut
NCut criteria measures both:
Dissimilarity between groups
Similarity within a group
Global impression extraction of the image
Spectral Analysis in favor of segmenting images
Generalized eigenvalue system gives real
solution=>”segmenting” this data provide
clustering of the original image
Thanks!
Comments 0
Log in to post a comment