Modelbased clustering of highdimensional data:
an overview and some recent advances
Charles BOUVEYRON
Laboratoire SAMM,EA 4543
Université Paris 1 PanthéonSorbonne
This presentation is based on several works
jointly done with S.Girard & G.Celeux
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 1/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 2/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 3/64
Introduction
Clustering has become a recurring problem:
it usually occurs in all applications for which a partition is
necessary (interpretation,decision,...),
but modern data are very often highdimensional (p large),
and the number of observations is sometimes small as well
(n p).
Example:segmentation of hyperspectral images
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 4/64
The classiﬁcation problem
The classiﬁcation problem consists in:
organizing the observations x
1
,...,x
n
∈ R
p
into K classes,
i.e.associating the labels z
1
,...,z
n
∈ {1,...,K} to the data.
Supervised approach:complete dataset (x
1
,z
1
),...,(x
n
,z
n
)
−4 −3 −2 −1 0 1 2 3 4
−4
−3
−2
−1
0
1
2
3
4
5
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
Nonsupervised approach:only the observations x
1
,...,x
n
−3 −2 −1 0 1 2 3 4
−4
−3
−2
−1
0
1
2
3
−3 −2 −1 0 1 2 3 4
−4
−3
−2
−1
0
1
2
3
−2 −1 0 1 2 3 4
−3
−2
−1
0
1
2
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 5/64
Probabilistic classiﬁcation and the MAP rule
The classical probabilistic framework assumes that:
the observations x
1
,...,x
n
are independant realizations of a
random vector X ∈ X
p
,
the labels z
1
,...,z
n
are independant realizations of a random
variable Z ∈ {1,...,K},
where z
i
= k indicates that x
i
belongs to the kth class.
The classiﬁcation aims to build a decision rule δ:
δ:X
p
→ {1,...,K},
x → z.
The optimal rule δ
∗
is the one which assigns x to the class with the
highest posterior probability (called the MAP rule):
δ
∗
(x) = argmax
k=1,...,K
P(Z = kX = x).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 6/64
Probabilistic classiﬁcation and the MAP rule
The classical probabilistic framework assumes that:
the observations x
1
,...,x
n
are independant realizations of a
random vector X ∈ X
p
,
the labels z
1
,...,z
n
are independant realizations of a random
variable Z ∈ {1,...,K},
where z
i
= k indicates that x
i
belongs to the kth class.
The classiﬁcation aims to build a decision rule δ:
δ:X
p
→ {1,...,K},
x → z.
The optimal rule δ
∗
is the one which assigns x to the class with the
highest posterior probability (called the MAP rule):
δ
∗
(x) = argmax
k=1,...,K
P(Z = kX = x).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 6/64
Generative and discriminative approaches
The diﬀerence between both approaches:
the way they estimate the posterior probability P(ZX)
which is used in the MAP decision rule.
Discriminative methods:
they directly model the posterior probability P(ZX),
by building a boundary between the classes.
Generative methods:
they ﬁrst model the joint distribution P(X,Z),
and then deduce the posterior probability using the
Bayes’ rule:
P(ZX) =
P(X,Z)
P(X)
∝ P(Z)P(XZ).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 7/64
Generative and discriminative approaches
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
Fig.Discriminative (left) and generative (right) methods.
Discriminative methods:
logistic regression (it models log(f
1
(x)/f
2
(x))),
Support Vector Machines (SVM),decision trees,...
Generative methods:
mainly,modelbased classiﬁcation methods,
but it exists also nonparametric methods.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 8/64
The mixture model
The mixture model:
the observations x
1
,...,x
n
are assumed to be independant
realizations of a random vector X ∈ X
p
with a density:
f(x) =
K
k=1
π
k
f(x,θ
k
),
K is the number of classes,
π
k
are the mixture proportions,
f(x,θ
k
) is a probability density with its parameters θ
k
.
The Gaussian mixture model:
among all mixture models,the Gaussian mixture model is
certainly the most used in the classiﬁcation context,
in this case,f(x,θ
k
) is the Gaussian density N(µ
k
,Σ
k
)
with θ
k
= {µ
k
,Σ
k
}.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 9/64
The mixture model
The MAP decision rule becomes in the mixture model framework:
δ
∗
(x) = argmax
k=1,...,K
P(Z = kX = x),
= argmax
k=1,...,K
P(Z = k)P(X = xZ = k),
= argmin
k=1,...,K
H
k
(x),
where H
k
is deﬁned by H
k
(x) = −2 log(π
k
f(x,θ
k
)).
The building of the decision rule consists in:
1 estimate the parameters θ
k
of the mixture model,
2 calculate the value of H
k
(x) for each new observation x.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 10/64
Gaussian mixtures for classiﬁcation
Gaussian model FullGMM (QDA in discrimination):
H
k
(x) = (x −µ
k
)
t
Σ
−1
k
(x −µ
k
) +log(det Σ
k
) −2 log(π
k
) +C
st
.
Gaussian model ComGMM which assumes that ∀k,Σ
k
= Σ (LDA
in discrimination):
H
k
(x) = µ
t
k
Σ
−1
µ
k
−2µ
t
k
Σ
−1
x −2 log(π
k
) +C
st
.
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
−3 −2 −1 0 1 2 3
−4
−3
−2
−1
0
1
2
3
4
Fig.Decision boundaries for FullGMM (left) and ComGMM (right).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 11/64
The curse of dimensionality
The curse of dimensionality:
this term was ﬁrst used by R.Bellman in the introduction of
his book “Dynamic programming” in 1957:
All [problems due to high dimension] may be subsumed under
the heading “the curse of dimensionality”.Since this is a
curse,[...],there is no need to feel discouraged about the
possibility of obtaining signiﬁcant results despite it.
he used this term to talk about the diﬃculties to ﬁnd an
optimum in a highdimensional space using an exhaustive
search,
in order to promotate dynamic approaches in programming.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 12/64
The curse of dimensionality
In the mixture model context:
the building of the data partition mainly depends on:
H
k
(x) = −2 log(π
k
f(x,θ
k
)),
model FullGMM:
H
k
(x) = (x−µ
k
)
t
Σ
−1
k
(x−µ
k
) +log(det Σ
k
) −2 log(π
k
) +γ.
model ComGMM which assumes that ∀k,Σ
k
= Σ:
H
k
(x) = µ
t
k
Σ
−1
µ
k
−2µ
t
k
Σ
−1
x −2 log(π
k
) +γ.
Important remarks:
it is necessary to invert Σ
k
or Σ,
and this will cause big diﬃculties in certain cases!
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 13/64
The curse of dimensionality
In the mixture model context:
the number of parameters grows up with p
2
,
0
10
20
30
40
50
60
70
80
90
100
0
5000
10000
15000
20000
25000
Dimension
Nb de paramètres
Full−GMM
Com−GMM
Fig.Number of parameters to estimate for the models FullGMM
and ComGMM regarding to the dimension and with k = 4.
if n is small compared to p
2
,the estimates of Σ
k
are
illconditionned or singular,
it is therefore diﬃcult or impossible to invert Σ
k
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 14/64
The blessings of dimensionality
As Bellman thought:
all is not bad in highdimensional spaces (hopefully!)
there are interesting things which happen in highdimensional
spaces.
The emptyspace phenomenum [Scott83]:
classical thoughts true in 1,2 or 3dimensional spaces are in
fact wrong in higher dimensions,
particularly,highdimensional spaces are almost empty!
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 15/64
The blessings of dimensionality
First example:the volume of a sphere
V (p) =
π
p/2
Γ(p/2 +1)
,
0
5
10
15
20
25
30
35
40
45
50
0
1
2
3
4
5
6
X: 5
Y: 5.264
X: 10
Y: 2.55
X: 15
Y: 0.3814
X: 25
Y: 0.0009577
Dimension
Volume
X: 40
Y: 3.605e−09
Fig.Volume of a sphere of radius 1 regarding to the dimension p.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 16/64
The blessings of dimensionality
Second example:
since highdimensional spaces are almost empty,
it should be easier to separate groups in highdimensional
space with an adapted classiﬁer.
20
50
100
150
200
250
0.94
0.95
0.96
0.97
0.98
0.99
1
Dimension
Taux de classification correcte
Classifieur optimal
de Bayes
Fig.Correct classiﬁcation rate of the optimal classiﬁer
versus the data dimension on simulated data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 17/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 18/64
Classical ways to avoid the curse of dimensionality
Dimension reduction:
the problem comes from that p is too large,
therefore,reduce the data dimension to d p,
such that the curse of dimensionality vanishes!
Parsimonious models:
the problem comes from that the number of parameters to
estimate is too large,
therefore,make additional assumptions to the model,
such that the number of parameters to estimate becomes
more “decent”!
Regularization:
the problem comes from that parameter estimates are instable,
therefore,regularize these estimates,
such that the parameter are correctly estimated!
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 19/64
Dimension reduction
Linear dimension reduction methods:
feature combination:PCA,
feature selection:...
Non linear dimension reduction methods:
Kohonen algorithms,Self Organising Maps,
LLE,Isomap,...
Kernel PCA,principal curves,...
Supervised dimension reduction methods:
the old fashion method:Fisher Discriminant Analysis (FDA),
many recent works on this topic...but useless in our context!
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 20/64
Parsimonious models
Parsimonious models:
can be obtained by making additional assumptions on the
original model
in order to adapt the model to the available data.
Parsimonious Gaussian models:
comGMM:
the assumption:Σ
k
= Σ,
nb of par.for K = 4 and p = 100:5453
diagGMM:
the assumption:Σ
k
= diag(σ
k1
,...,σ
kp
),
nb of par.for K = 4 and p = 100:803
spheGMM:
the assumption:Σ
k
= σ
k
I
p
,
nb of par.for K = 4 and p = 100:407
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 21/64
Parsimonious models
A family of parsimonious Gaussian models [Banﬁeld93,Celeux95]:
Fig.The family of 14 parsimonious Gaussian models [Celeux95].
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 22/64
Regularization
Regularization of the covariance matrix estimates:
ridgelike regularization:
˜
Σ
k
=
ˆ
Σ
k
+σ
k
I
p
,
PDA [Hast95]:
˜
Σ
k
=
ˆ
Σ
k
+σ
k
Ω,
RDA [Frie89] proposed a regularized classiﬁer which varies
between a quadratic and a linear classiﬁer:
˜
Σ
k
(λ,γ) = (1 −γ)S
k
(λ) +γ
tr(S
k
(λ))
p
I
p
where S
k
is deﬁned by:
S
k
(λ) =
(n
k
−1)(1 −λ)
ˆ
Σ
k
+(n −K)λ
ˆ
Σ
(1 −λ)(n
k
−1) +λ(n −K)
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 23/64
Regularization
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
γ = 0,λ = 0 γ = 0,λ = 0.5 γ = 0,λ = 1
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
γ = 0.5,λ = 0 γ = 0.5,λ = 0.5 γ = 0.5,λ = 1
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
3
2
1
0
1
2
3
4
γ = 1,λ = 0 γ = 1,λ = 0.5 γ = 1,λ = 1
1 Analyse dis riminan te régularisée (RD A):le paramètre λ permet de
faire v arier le lassieur en tre QD A et LD A tandis que le paramètre γ on trle
l'estimation des v aleurs propres des matri es de o v arian e.
1
Fig.4.Inﬂuence des paramètres γ et λ sur le classiﬁeur RDA.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 24/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 25/64
Subspace clustering methods
Recent approaches propose:
to model the data of each group in speciﬁc subspaces,
to keep all dimensions in order to facilitate the discrimination
of the groups.
Several works on this topic in the last years:
mixture of factor analyzers:Ghahramani et al.(1996) and
McLachlan et al.(2003),
mixture of probabilistic PCA:Tipping & Bishop (1999),
mixture of HD Gaussian models:Bouveyron & Girard (2007),
mixture of parsimonious FA:McNicholas and Murphy (2008),
mixture of common FA:Beak et al.(2009).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 26/64
Subspace clustering methods
Mixture of Factor Analyzers
Beak
et al.
McNicholas & Murphy
Tipping & Bishop
Bouveyron & Girard
Common FA:
 1 model,
 unconstrained
noise variance,
 common
orientations,
 common
dimensions
Parsimonious
GMM:
 8 models,
 constrained or
unconstrained
noise variance,
 free or
common
orientations,
 common
dimensions
Mixture of
PPCA:
 1 model,
 isotropic noise
variance,
 free
orientations,
 common
dimensions
HDDC:
 24 models,
 isotropic noise
variance
 free or
common
orientations
 free or
common
dimensions
Figure:A tentative family tree of subspace clustering methods.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 27/64
The model [a
kj
b
k
Q
k
d
k
]
Bouveyron & Girard (2007) proposed to consider the Gaussian mix
ture model:
f(x) =
K
k=1
π
k
f(x,θ
k
),
where θ
k
= {µ
k
,Σ
k
} for each k = 1,...,K.
Based on the spectral decomposition of Σ
k
,we can write:
Σ
k
= Q
k
Δ
k
Q
t
k
,
where:
Q
k
is an orthogonal matrix containing the eigenvectors of Σ
k
,
Δ
k
is diagonal matrix containing the eigenvalues of Σ
k
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 28/64
The model [a
kj
b
k
Q
k
d
k
]
We assume that Δ
k
has the following form:
Δ
k
=
a
k1
0
.
.
.
0 a
kd
k
0
0
b
k
0
.
.
.
.
.
.
0 b
k
d
k
(p −d
k
)
where:
a
kj
≥ b
k
,for j = 1,...,d
k
and k = 1,...,K,
and d
k
< p,for k = 1,...,K.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 29/64
The model [a
kj
b
k
Q
k
d
k
]
Fig.The subspace E
k
and its supplementary E
⊥
k
.
We also deﬁne:
the aﬃne space E
k
generated by eigenvectors associated to
the eigenvalues a
kj
and such that µ
k
∈ E
k
,
the aﬃne space E
⊥
k
such that E
k
⊕E
⊥
k
= R
p
and µ
k
∈ E
⊥
k
,
the projectors P
k
and P
⊥
k
respectively on E
k
and E
⊥
k
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 30/64
The model [a
kj
b
k
Q
k
d
k
] and its submodels
We thus obtain a reparameterization of the Gaussian model:
which depends on a
kj
,b
k
,Q
k
and d
k
,
the model complexity is controlled by the subspace
dimensions.
We obtain increasingly regularized models:
by ﬁxing some parameters to be common within or between
the classes,
from the most complex model to the simplest model.
Our family of GMM contains 28 models and can be splitted into
three branches:
14 models with free orientations,
12 models with common orientations,
2 models with common covariance matrices.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 31/64
The model [a
kj
b
k
Q
k
d
k
] and its submodels
Model
Number of
parameters
Asymptotic
order
Nb of prms k = 4,
d = 10,p = 100
ML
estimation
[a
ij
b
i
Q
i
d
i
] ρ + ¯τ +2k +D kpd 4231 CF
[a
ij
bQ
i
d
i
] ρ + ¯τ +k +D+1 kpd 4228 CF
[a
i
b
i
Q
i
d
i
] ρ + ¯τ +3k kpd 4195 CF
[ab
i
Q
i
d
i
] ρ + ¯τ +2k +1 kpd 4192 CF
[a
i
bQ
i
d
i
] ρ + ¯τ +2k +1 kpd 4192 CF
[abQ
i
d
i
] ρ + ¯τ +k +2 kpd 4189 CF
[a
ij
b
i
Q
i
d] ρ +k(τ +d +1) +1 kpd 4228 CF
[a
j
b
i
Q
i
d] ρ +k(τ +1) +d +1 kpd 4198 CF
[a
ij
bQ
i
d] ρ +k(τ +d) +2 kpd 4225 CF
[a
j
bQ
i
d] ρ +kτ +d +2 kpd 4195 CF
[a
i
b
i
Q
i
d] ρ +k(τ +2) +1 kpd 4192 CF
[ab
i
Q
i
d] ρ +k(τ +1) +2 kpd 4189 CF
[a
i
bQ
i
d] ρ +k(τ +1) +2 kpd 4189 CF
[abQ
i
d] ρ +kτ +3 kpd 4186 CF
[a
ij
b
i
Qd
i
] ρ +τ +D+2k pd 1396 FG
[a
ij
bQd
i
] ρ +τ +D+k +1 pd 1393 FG
[a
i
b
i
Qd
i
] ρ +τ +3k pd 1360 FG
[a
i
bQd
i
] ρ +τ +2k +1 pd 1357 FG
[ab
i
Qd
i
] ρ +τ +2k +1 pd 1357 FG
[abQd
i
] ρ +τ +k +2 pd 1354 FG
[a
ij
b
i
Qd] ρ +τ +kd +k +1 pd 1393 FG
[a
j
b
i
Qd] ρ +τ +k +d +1 pd 1363 FG
[a
ij
bQd] ρ +τ +kd +2 pd 1390 FG
[a
i
b
i
Qd] ρ +τ +2k +1 pd 1357 IP
[ab
i
Qd] ρ +τ +k +2 pd 1354 IP
[a
i
bQd] ρ +τ +k +2 pd 1354 IP
[a
j
bQd] ρ +τ +d +2 pd 1360 CF
[abQd] ρ +τ +3 pd 1351 CF
FullGMM ρ +kp(p +1)/2 kp
2
/2 20603 CF
ComGMM ρ +p(p +1)/2 p
2
/2 5453 CF
DiagGMM ρ +kp 2kp 803 CF
SpheGMM ρ +k kp 407 CF
Table 1:Properties of the HDDC models:ρ = kp+k−1 is the number of parameters
required for the estimation of means and proportions,¯τ =
k
i=1
d
i
[p −(d
i
+1)/2]
and τ = d[p −(d +1)/2] are the number of parameters required for the estimation
of
˜
Q
i
and
˜
Q,and D =
k
i=1
d
i
.For asymptotic orders,we assume that k ≪
d ≪p.CF means that the ML estimates are closed form.IP means that the ML
estimation needs an iterative procedure.FGmeans that the ML estimation requires
the iterative FG algorithm.
1
Table:Properties of the submodels of [a
kj
b
k
Q
k
d
k
]
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 32/64
The model [a
kj
b
k
Q
k
d
k
] and its submodels
Model
Nb of prms,K = 4
d = 10,p = 100
Classiﬁer type
[a
kj
b
k
Q
k
d
k
]
4231
Quadratic
[a
kj
b
k
Qd
k
]
1396
Quadratic
[a
j
bQd]
1360
Linear
FullGMM
20603
Quadratic
ComGMM
5453
Linear
Table.Properties of the submodels of [a
kj
b
k
Q
k
d
k
]
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 33/64
The model [a
kj
b
k
Q
k
d
k
] and its submodels
model [a
k
b
k
Q
k
d] model [ab
k
Q
k
d] model [a
k
bQ
k
d]
model [a
k
b
k
Qd] model [abQd] model [abI
2
d]
Fig.Inﬂuence of parameters a
k
,b
k
et Q
k
on the densities
of 2 classes in dimension 2 with d
1
= d
2
= 1.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 34/64
Construction of the classiﬁer
In the supervised context:
the classiﬁer has been named HDDA,
the estimation of parameters is direct since we have complete
data,
parameters are estimated by maximum likelihood.
In the unsupervised context:
the classiﬁer has been named HDDC,
the estimation of parameters is not direct since we do not
have complete data,
parameters are estimated through a EM algorithm which
iteratively maximizes the likelihood.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 35/64
HDDC:the E step
In the case of the model [a
k
b
k
Q
k
d
k
]:
H
k
(x) =
1
a
k
µ
k
−P
k
(x)
2
+
1
b
k
x −P
k
(x)
2
+d
k
log(a
k
)+(p−d
k
) log(b
k
)−2 log(π
k
).
Fig.The subspaces E
k
and E
⊥
k
of the kth mixture composant.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 36/64
HDDC:the M step
The ML estimators for the model [a
kj
b
k
Q
k
d
k
] are closed forms:
Subspace E
k
:the d
k
ﬁrst columns of Q
k
are estimated by the
eigenvectors associated to the d
k
largest eigenvalues λ
kj
of
the empirical covariance matrix W
k
of the kth class.
Estimator of a
kj
:the parameters a
kj
are estimated by the d
k
largest eigenvalues λ
kj
of W
k
.
Estimator of b
k
:the parameter of b
k
is estimated by:
ˆ
b
k
=
1
(p −d
k
)
trace(W
k
) −
d
k
j=1
λ
kj
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 37/64
HDDC:hyperparameter estimation
0
1
2
3
4
5
6
7
8
9
10
11
12
0
0.02
0.04
0.06
0.08
0.1
1
2
3
4
5
6
7
8
9
0
0.01
0.02
0.03
0.04
0.05
Fig.The screetest of Cattell based on the eigenvalue scree.
Estimation of the intrinsic dimensions d
k
:
we use the screetest of Cattell [Catt66],
it allows to estimate the K parameters d
k
in a common way.
Estimation of the nomber of groups K:
in the supervised context,K is known,
in the unsupervised context,K is chosen using BIC.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 38/64
Numerical considerations
Numerical stability:the decision rule of HDDC does not
depend on the eigenvectors associated with the smallest
eigenvalues of W
k
.
Reduction of computing time:there is no need to compute
the last eigenvectors of W
k
→ reduction of computing time
with a designed procedure (×60 for p = 1000).
Particular case n < p:from a numerical point of view,it is
better to compute the eigenvectors of
¯
X
k
¯
X
t
k
instead of
W
k
=
¯
X
t
k
¯
X
k
(×500 for n = 13 and p = 1000).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 39/64
Eﬀect of the dimensionality
20
30
40
50
60
70
80
90
100
0.4
0.5
0.6
0.7
0.8
0.9
1
Dimension
Taux de classification correcte sur le jeu de validation
HDDA
QDA
LDA
PCA+LDA
EDDA
Bayes
Fig.Correct classiﬁcation rate versus
data dimension (simulated data according to [a
ij
b
i
Q
i
d
i
]).
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 40/64
Estimation of intrinsic dimensions
Nb of classes k
Chosen threshold s
Dimensions d
i
BIC value
2
0.18
2,16
414
3
0.21
2,5,10
407
4
0.25
2,2,5,10
414
5
0.28
2,5,5,10,12
416
6
0.28
2,5,6,10,10,12
424
Table.Selection of discrete parameters using BIC
on simulated data where d
i
are equal to 2,5 and 10.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 41/64
Comparison with variable selection
Model
On original features
With a dimension
reduction step (ACP)
SpheGMM
0.340
0.340
DiagGMM
0.355
0.535
ComGMM
0.625
0.635
FullGMM
0.640
0.845
VSGMM [Raft05]
0.925
/
HDDC [a
i
b
i
Q
i
d
i
]
0.950
/
Table.Correct classiﬁcation rate on a real dataset:Crabs ∈ R
5
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 42/64
HDDC:an EMbased algorithm
100 90 80 70 60 50 40 30 20 10
6
5
4
3
2
1
0
1
2
3
Fig.Projection of the «Crabs» data on the ﬁrst principal axes.
«Crabs» data:
200 observations in a 5dimensional space (5 morphological
features),
4 classes:BM,BF,OM and OF.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 43/64
HDDC:an EMbased algorithm
100
80
60
40
20
6
4
2
0
2
HDDC step 1
100
80
60
40
20
6
4
2
0
2
HDDC step 2
100
80
60
40
20
6
4
2
0
2
HDDC step 3
100
80
60
40
20
6
4
2
0
2
HDDC step 4
100
80
60
40
20
6
4
2
0
2
HDDC step 5
100
80
60
40
20
6
4
2
0
2
HDDC step 6
100
80
60
40
20
6
4
2
0
2
HDDC step 7
100
80
60
40
20
6
4
2
0
2
HDDC step 8
100
80
60
40
20
6
4
2
0
2
HDDC step 9
100
80
60
40
20
6
4
2
0
2
HDDC step 10
100
80
60
40
20
6
4
2
0
2
HDDC step 11
100
80
60
40
20
6
4
2
0
2
HDDC step 12
Fig.Step n° 1 of HDDC on the «Crabs» data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 44/64
HDDC:an EMbased algorithm
100
80
60
40
20
6
4
2
0
2
HDDC step 1
100
80
60
40
20
6
4
2
0
2
HDDC step 2
100
80
60
40
20
6
4
2
0
2
HDDC step 3
100
80
60
40
20
6
4
2
0
2
HDDC step 4
100
80
60
40
20
6
4
2
0
2
HDDC step 5
100
80
60
40
20
6
4
2
0
2
HDDC step 6
100
80
60
40
20
6
4
2
0
2
HDDC step 7
100
80
60
40
20
6
4
2
0
2
HDDC step 8
100
80
60
40
20
6
4
2
0
2
HDDC step 9
100
80
60
40
20
6
4
2
0
2
HDDC step 10
100
80
60
40
20
6
4
2
0
2
HDDC step 11
100
80
60
40
20
6
4
2
0
2
HDDC step 12
Fig.Step n° 4 of HDDC on the «Crabs» data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 45/64
HDDC:an EMbased algorithm
100
80
60
40
20
6
4
2
0
2
HDDC step 1
100
80
60
40
20
6
4
2
0
2
HDDC step 2
100
80
60
40
20
6
4
2
0
2
HDDC step 3
100
80
60
40
20
6
4
2
0
2
HDDC step 4
100
80
60
40
20
6
4
2
0
2
HDDC step 5
100
80
60
40
20
6
4
2
0
2
HDDC step 6
100
80
60
40
20
6
4
2
0
2
HDDC step 7
100
80
60
40
20
6
4
2
0
2
HDDC step 8
100
80
60
40
20
6
4
2
0
2
HDDC step 9
100
80
60
40
20
6
4
2
0
2
HDDC step 10
100
80
60
40
20
6
4
2
0
2
HDDC step 11
100
80
60
40
20
6
4
2
0
2
HDDC step 12
Fig.Step n° 7 of HDDC on the «Crabs» data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 46/64
HDDC:an EMbased algorithm
100
80
60
40
20
6
4
2
0
2
HDDC step 1
100
80
60
40
20
6
4
2
0
2
HDDC step 2
100
80
60
40
20
6
4
2
0
2
HDDC step 3
100
80
60
40
20
6
4
2
0
2
HDDC step 4
100
80
60
40
20
6
4
2
0
2
HDDC step 5
100
80
60
40
20
6
4
2
0
2
HDDC step 6
100
80
60
40
20
6
4
2
0
2
HDDC step 7
100
80
60
40
20
6
4
2
0
2
HDDC step 8
100
80
60
40
20
6
4
2
0
2
HDDC step 9
100
80
60
40
20
6
4
2
0
2
HDDC step 10
100
80
60
40
20
6
4
2
0
2
HDDC step 11
100
80
60
40
20
6
4
2
0
2
HDDC step 12
Fig.Step n° 10 of HDDC on the «Crabs» data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 47/64
HDDC:an EMbased algorithm
100
80
60
40
20
6
4
2
0
2
HDDC step 1
100
80
60
40
20
6
4
2
0
2
HDDC step 2
100
80
60
40
20
6
4
2
0
2
HDDC step 3
100
80
60
40
20
6
4
2
0
2
HDDC step 4
100
80
60
40
20
6
4
2
0
2
HDDC step 5
100
80
60
40
20
6
4
2
0
2
HDDC step 6
100
80
60
40
20
6
4
2
0
2
HDDC step 7
100
80
60
40
20
6
4
2
0
2
HDDC step 8
100
80
60
40
20
6
4
2
0
2
HDDC step 9
100
80
60
40
20
6
4
2
0
2
HDDC step 10
100
80
60
40
20
6
4
2
0
2
HDDC step 11
100
80
60
40
20
6
4
2
0
2
HDDC step 12
Fig.Step n° 12 of HDDC on the «Crabs» data.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 48/64
Categorization of the Martian surface
0
50
100
150
200
250
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Spectral band
data1
data2
data3
data4
data5
Fig.Categorization of the Martian surface based on HD spectral images.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 49/64
Object localization in natural images
Fig.Object localization of an object “bike” in a natural image.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 50/64
Texture recognition
Fig.Segmentation of an image containing several textures:diagGMM,HDGMM,
diagGMM with hidden Markov ﬁeld and HDGMM with hidden Markov ﬁeld.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 51/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 52/64
Intrinsic dimension selection
In subspace clustering:
the diﬀerent models are all parametrized by the intrinsic
dimension of the subspaces,
Bouveyron et al.have proposed to use the screetest of
Cattell to determine the dimensions d
k
,
this approach works quite well in practice and can be combine
to either crossvalidation or BIC to select the threshold.
A priori,ML should not be used to determine the d
k
:
since the d
k
determine the model complexity and therefore the
likelihood increases with d
k
,
except for the model [a
k
b
k
Q
k
d
k
]!
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 53/64
The isotropic PPCA model
To simplify,let us deﬁne the isotropic PPCA model:
the observed variable Y ∈ R
p
and the latent variable X ∈ R
d
are assumed to be linked:
Y =
¯
QX +µ +ε,
where X and ε have Gaussian distributions such that
Δ
k
= Q
t
ΣQ has the following form:
Δ
k
=
a 0
.
.
.
0 a
0
0
b 0
.
.
.
.
.
.
0 b
d
(p −d)
where a ≥ b and d < p.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 54/64
ML estimate of d is asymptically consistent
Proposition:
The maximum likelihood estimate of the actal intrinsic
dimension d
∗
is asymptotically unique and consistent.
Sketch of the proof:
At the optimum,the maximization of (
ˆ
θ) is equivalent to the minimization of:
f
n
(d) = dlog(ˆa) +(p −d) log(
ˆ
b) +p.
1 If d ≤ d
∗
:ˆa →a and
ˆ
b →
1
p−d
[(d
∗
−d)a +(p −d
∗
)b] almost surely when
n →∞and f
n
→f:
f(d) = dlog(a) +(p −d) log
(d
∗
−d)
(p −d)
a +
(p −d
∗
)
(p −d)
,
which has a unique minimum in d = d
∗
.
2 If d ≥ d
∗
:ˆa →
1
d
(d
∗
a +(d −d
∗
)b) and
ˆ
b →b almost surely when n →∞and
f
n
→f:
f(d) = dlog
d
∗
d
a +
d −d
∗
d
b
+(p −d) log(b),
which has as well a unique minimum in d = d
∗
.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 55/64
Experimental setup
To verify the practical interest of the result:
we deﬁne the parameters α and β:
α =
n
p
,
β =
d
∗
a
(p −d
∗
)b
,
α controls the estimation conditions through the ratio
between the number of observations and the observation
space dimension,
β controls the signal to noise ratio through the ratio between
the variances in the latent subspace and in its orthogonal
subspace.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 56/64
An introductory simulation
β = 4
Eigenvalue scree
Dimension
Eigenvalue
02468
0 10 20 30 40 50
−24500−23500−22500
Max. Likelihood
Dimension
Lik. const. PPCA
0 10 20 30 40 50
−26000−25000−24000−23000
AIC
Dimension
AIC
0 10 20 30 40 50
−28000−27000−26000−25000
BIC
Dimension
BIC
β = 2
Eigenvalue scree
Dimension
Eigenvalue
01234
0 10 20 30 40 50
−21200−20800−20400
Max. Likelihood
Dimension
Lik. const. PPCA
0 10 20 30 40 50
−22500−22000−21500−21000
AIC
Dimension
AIC
0 10 20 30 40 50
−24500−23500−22500−21500
BIC
Dimension
BIC
β = 1
Eigenvalue scree
Dimension
Eigenvalue
0.00.51.01.52.02.5
0 10 20 30 40 50
−18800−18600−18400
Max. Likelihood
Dimension
Lik. const. PPCA
0 10 20 30 40 50
−20000−19600−19200
AIC
Dimension
AIC
0 10 20 30 40 50
−22000−21000−20000−19000
BIC
Dimension
BIC
Figure:Intrinsic dimension estimation with d
∗
= 20 and α = 5.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 57/64
Inﬂuence of the signal to noise ratio
0 5 10 15 20 25 30
05101520253035
Signal/noise ratio
Average selected dimension
0 5 10 15 20 25 30
05101520253035
Signal/noise ratio
Average selected dimension
0 5 10 15 20 25 30
05101520253035
Signal/noise ratio
Average selected dimension
0 5 10 15 20 25 30
05101520253035
Signal/noise ratio
Average selected dimension
ML
BIC
AIC
Cattell
MleDim reg.
Laplace
CV lik.
Figure:Average selected dimension according to β for α = 4,3,2 and 1.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 58/64
Inﬂuence of the n/p ratio
0 10 20 30 40
05101520253035
N/p ratio
Average selected dimension
0 10 20 30 40
05101520253035
N/p ratio
Average selected dimension
0 10 20 30 40
05101520253035
N/p ratio
Average selected dimension
0 10 20 30 40
05101520253035
N/p ratio
Average selected dimension
ML
BIC
AIC
Cattell
MleDim reg.
Laplace
CV Lik.
Figure:Average selected dimension according to α for β = 4,3,2 and 1.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 59/64
A graphical summary
1 2 5 10 20 50
125102050
N/p ratio
Signal/noise ratio
Too difficult!
ML
AIC
AIC & ML
All criteria
Figure:Recommended criteria for intrinsic dimension selection according
to α and β for the isotropic PPCA model.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 60/64
Outline
1 Introduction
2 Classical ways to deal with HD data
3 Recent modelbased methods for HD data clustering
4 Intrinsic dimension selection by ML in subspace clustering
5 Conclusion & further works
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 61/64
Conclusion & further works
Dimension reduction:
is usefull for visualization purposes,
but clustering a reduced dataset is suboptimal.
Parsimonious models & regularization:
allow to adapt the model complexity to the data,
parsimonious models are usually valid for data with p<25,
Subspace clustering:
adapted for real high dimensional data (p>25,100,1000,...),
even when n is small compared to p,
the best of dimension reduction and parsimonious models.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 62/64
Conclusion & further works
Intrinsic dimension selection:
intrinsic dimension of the subspaces is the key parameter in
subspace clustering,
the oldfashion method of Cattell works quite well in practice,
BIC,AIC and even ML can also be used in speciﬁc contexts.
Further works:
use ML in HDDA and HDDC to make these methods fully
automatic,
integration of this approach in softwares.
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 63/64
Softwares
HDDA/HDDC:
Matlab toolboxes are available at:
http://samm.univparis1.fr/charlesbouveyron
8 models are available in the Mixmod software:
http://wwwmath.univfcomte.fr/mixmod/
A R package,nammed HDclassif,is available for a few
weeks on the CRAN servers (thanks to L.Bergé & R.Aidan).
FisherEM:
a R package is planned for next year...
Charles BOUVEYRON  Modelbased clustering of highdimensional data:an overview and some recent advances 64/64
Enter the password to open this PDF file:
File name:

File size:

Title:

Author:

Subject:

Keywords:

Creation Date:

Modification Date:

Creator:

PDF Producer:

PDF Version:

Page Count:

Preparing document for printing…
0%
Comments 0
Log in to post a comment