Spectral Clustering
Eyal David
Image Processing seminar
May
2008
2
Lecture Outline
Motivation
Graph overview and construction
Demo
Spectral Clustering
Demo
Cool implementations
3
A Tutorial on Spectral Clustering
\
Arik Azran
4
Spectral Clustering Example
–
2
Spirals
2
1.5
1
0.5
0
0.5
1
1.5
2
2
1.5
1
0.5
0
0.5
1
1.5
2
Dataset exhibits complex
cluster shapes
K

means performs very
poorly in this space due bias
toward dense spherical
clusters.
0.8
0.6
0.4
0.2
0
0.2
0.4
0.6
0.8
0.709
0.7085
0.708
0.7075
0.707
0.7065
0.706
In the embedded space
given by two leading
eigenvectors, clusters are
trivial to separate.
Spectral Clustering

Derek Greene
5
Lecture Outline
Motivation
Graph overview and construction
Graph demo
Spectral Clustering
Spectral Clustering demo
Cool implementation
6
Matthias Hein and Ulrike von Luxburg August
2007
7
Matthias Hein and Ulrike von Luxburg August
2007
8
Matthias Hein and Ulrike von Luxburg August
2007
9
Matthias Hein and Ulrike von Luxburg August
2007
10
Matthias Hein and Ulrike von Luxburg August
2007
11
Matthias Hein and Ulrike von Luxburg August
2007
12
Matthias Hein and Ulrike von Luxburg August
2007
13
Matthias Hein and Ulrike von Luxburg August
2007
14
Matthias Hein and Ulrike von Luxburg August
2007
15
Matthias Hein and Ulrike von Luxburg August
2007
16
Matthias Hein and Ulrike von Luxburg August
2007
17
Matthias Hein and Ulrike von Luxburg August
2007
Demo
(Live example)
19
Lecture Outline
Motivation
Graph overview and construction
Demo
Spectral Clustering
Demo
Cool implementations
20
Matthias Hein and Ulrike von Luxburg August
2007
21
Matthias Hein and Ulrike von Luxburg August
2007
22
Matthias Hein and Ulrike von Luxburg August
2007
23
Matthias Hein and Ulrike von Luxburg August
2007
24
Matthias Hein and Ulrike von Luxburg August
2007
25
Matthias Hein and Ulrike von Luxburg August
2007
26
Matthias Hein and Ulrike von Luxburg August
2007
27
Matthias Hein and Ulrike von Luxburg August
2007
28
Matthias Hein and Ulrike von Luxburg August
2007
29
Eigenvectors & Eigenvalues
30
Matthias Hein and Ulrike von Luxburg August
2007
31
Matthias Hein and Ulrike von Luxburg August
2007
Demo
(Live example)
33
Spectral Clustering Algorithm
Ng, Jordan, and Weiss
Motivation
Given a set of points
We would like to cluster them into k
subsets
1
,...,
l
n
S s s R
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
34
Algorithm
Form the affinity matrix
Define
if
Scaling parameter chosen by user
Define D a diagonal matrix whose
(i,i) element is the sum of A’s row i
nxn
W R
i j
0
ii
W
2 2
 /2
i j
s s
ij
W e
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
35
Algorithm
Form the matrix
Find
, the k largest eigenvectors of
L
These form the the columns of the new
matrix X
Note: have reduced dimension from nxn to nxk
1/2 1/2
L D WD
1 2
,,...,
k
x x x
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
36
Algorithm
Form the matrix Y
Renormalize each of X’s rows to have unit length
Y
Treat each row of Y as a point in
Cluster into k clusters via K

means
2 2
/( )
ij ij ij
j
Y X X
k
R
nxk
R
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
37
Algorithm
Final Cluster Assignment
Assign point to cluster j iff row i of Y was
assigned to cluster j
i
s
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
38
Why?
If we eventually use K

means, why not just
apply K

means to the original data?
This method allows us to cluster non

convex
regions
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
39
Some Examples
40
Ng et al On Spectral clustering: analysis and algorithm
41
Ng et al On Spectral clustering: analysis and algorithm
42
Ng et al On Spectral clustering: analysis and algorithm
43
Ng et al On Spectral clustering: analysis and algorithm
44
Ng et al On Spectral clustering: analysis and algorithm
45
Ng et al On Spectral clustering: analysis and algorithm
46
Ng et al On Spectral clustering: analysis and algorithm
47
Ng et al On Spectral clustering: analysis and algorithm
48
User’s Prerogative
Affinity matrix construction
Choice of scaling factor
Realistically, search over and pick value that
gives the tightest clusters
Choice of k, the number of clusters
Choice of clustering method
2
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm
49
0
5
10
15
20
25
30
35
40
45
50
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
K
Eigenvalue
Largest eigenvalues
of Cisi/Medline data
λ
1
λ
2
How to select
k
?
Eigengap
: the difference between two consecutive eigenvalues.
Most stable clustering is generally given by the value
k
that
maximises the expression
1
k k k
Choose
k=
2
1
2
max
k
Spectral Clustering

Derek Greene
50
Matthias Hein and Ulrike von Luxburg August
2007
Recap
–
The bottom line
51
Summary
Spectral clustering can help us in hard
clustering problems
The technique is simple to understand
The solution comes from solving a simple
algebra problem which is not hard to
implement
Great care should be taken in choosing the
“starting conditions”
The End
Comments 0
Log in to post a comment