spectral clustering

savagelizardAI and Robotics

Nov 25, 2013 (3 years and 10 months ago)

153 views

Spectral Clustering

Eyal David

Image Processing seminar

May
2008

2

Lecture Outline


Motivation


Graph overview and construction


Demo


Spectral Clustering


Demo


Cool implementations


3

A Tutorial on Spectral Clustering
\
Arik Azran

4

Spectral Clustering Example


2
Spirals

-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Dataset exhibits complex
cluster shapes



K
-
means performs very
poorly in this space due bias
toward dense spherical
clusters.

-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
-0.709
-0.7085
-0.708
-0.7075
-0.707
-0.7065
-0.706
In the embedded space
given by two leading
eigenvectors, clusters are
trivial to separate.

Spectral Clustering
-

Derek Greene

5

Lecture Outline


Motivation


Graph overview and construction


Graph demo


Spectral Clustering


Spectral Clustering demo


Cool implementation


6

Matthias Hein and Ulrike von Luxburg August
2007

7

Matthias Hein and Ulrike von Luxburg August
2007

8

Matthias Hein and Ulrike von Luxburg August
2007

9

Matthias Hein and Ulrike von Luxburg August
2007

10

Matthias Hein and Ulrike von Luxburg August
2007

11

Matthias Hein and Ulrike von Luxburg August
2007

12

Matthias Hein and Ulrike von Luxburg August
2007

13

Matthias Hein and Ulrike von Luxburg August
2007

14

Matthias Hein and Ulrike von Luxburg August
2007

15

Matthias Hein and Ulrike von Luxburg August
2007

16

Matthias Hein and Ulrike von Luxburg August
2007

17

Matthias Hein and Ulrike von Luxburg August
2007

Demo

(Live example)

19

Lecture Outline


Motivation


Graph overview and construction


Demo


Spectral Clustering


Demo


Cool implementations


20

Matthias Hein and Ulrike von Luxburg August
2007

21

Matthias Hein and Ulrike von Luxburg August
2007

22

Matthias Hein and Ulrike von Luxburg August
2007

23

Matthias Hein and Ulrike von Luxburg August
2007

24

Matthias Hein and Ulrike von Luxburg August
2007

25

Matthias Hein and Ulrike von Luxburg August
2007

26

Matthias Hein and Ulrike von Luxburg August
2007

27

Matthias Hein and Ulrike von Luxburg August
2007

28

Matthias Hein and Ulrike von Luxburg August
2007

29

Eigenvectors & Eigenvalues

30

Matthias Hein and Ulrike von Luxburg August
2007

31

Matthias Hein and Ulrike von Luxburg August
2007

Demo

(Live example)

33

Spectral Clustering Algorithm
Ng, Jordan, and Weiss


Motivation


Given a set of points





We would like to cluster them into k
subsets








1
,...,
l
n
S s s R
 
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

34

Algorithm


Form the affinity matrix


Define




if



Scaling parameter chosen by user


Define D a diagonal matrix whose


(i,i) element is the sum of A’s row i



nxn
W R

i j

0
ii
W

2 2
|| ||/2
i j
s s
ij
W e




Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

35

Algorithm


Form the matrix







Find



, the k largest eigenvectors of
L


These form the the columns of the new
matrix X


Note: have reduced dimension from nxn to nxk


1/2 1/2
L D WD
 

1 2
,,...,
k
x x x
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

36

Algorithm


Form the matrix Y


Renormalize each of X’s rows to have unit length





Y



Treat each row of Y as a point in


Cluster into k clusters via K
-
means


2 2
/( )
ij ij ij
j
Y X X


k
R
nxk
R

Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

37

Algorithm


Final Cluster Assignment


Assign point to cluster j iff row i of Y was
assigned to cluster j

i
s
Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

38

Why?


If we eventually use K
-
means, why not just
apply K
-
means to the original data?




This method allows us to cluster non
-
convex
regions


Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

39


Some Examples

40

Ng et al On Spectral clustering: analysis and algorithm
41

Ng et al On Spectral clustering: analysis and algorithm
42

Ng et al On Spectral clustering: analysis and algorithm
43

Ng et al On Spectral clustering: analysis and algorithm
44

Ng et al On Spectral clustering: analysis and algorithm
45

Ng et al On Spectral clustering: analysis and algorithm
46

Ng et al On Spectral clustering: analysis and algorithm
47

Ng et al On Spectral clustering: analysis and algorithm
48

User’s Prerogative


Affinity matrix construction


Choice of scaling factor


Realistically, search over and pick value that
gives the tightest clusters


Choice of k, the number of clusters


Choice of clustering method

2

Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry
based on Ng et al On Spectral clustering: analysis and algorithm

49

0
5
10
15
20
25
30
35
40
45
50
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
K
Eigenvalue
Largest eigenvalues

of Cisi/Medline data

λ
1

λ
2

How to select
k
?


Eigengap
: the difference between two consecutive eigenvalues.


Most stable clustering is generally given by the value
k

that
maximises the expression

1
k k k
 

  


Choose
k=
2

1
2
max





k
Spectral Clustering
-

Derek Greene

50

Matthias Hein and Ulrike von Luxburg August
2007

Recap


The bottom line

51

Summary


Spectral clustering can help us in hard
clustering problems


The technique is simple to understand


The solution comes from solving a simple
algebra problem which is not hard to
implement


Great care should be taken in choosing the
“starting conditions”

The End