Informatics and Mathematical Modelling / Intelligent Signal Processing
1
Morten Mørup
Extensions of Non

negative
Matrix Factorization to Higher
Order data
Morten Mørup
Informatics and Mathematical Modeling
Intelligent Signal Processing
Technical University of Denmark
Informatics and Mathematical Modelling / Intelligent Signal Processing
2
Morten Mørup
Parts of the work done in
collaboration with
Sæby,
May
22

2006
Lars Kai Hansen, Professor
Department of Signal Processing
Informatics and Mathematical Modeling,
Technical University of Denmark
Mikkel N. Schmidt, Stud. PhD
Department of Signal Processing
Informatics and Mathematical Modeling,
Technical University of Denmark
Sidse M. Arnfred, Dr. Med. PhD
Cognitive Research Unit
Hvidovre Hospital
University Hospital of Copenhagen
Informatics and Mathematical Modelling / Intelligent Signal Processing
3
Morten Mørup
Outline
Non

negativity Matrix Factorization
(NMF)
Sparse coding
(SNMF)
Convolutive PARAFAC models (cPARAFAC)
Higher Order Non

negative Matrix Factorization
(an extension of NMF to the Tucker model)
Informatics and Mathematical Modelling / Intelligent Signal Processing
4
Morten Mørup
NMF is based on Gradient Descent
NMF:
V
WH
s.t.
W
i,d
,
H
d,j
0
Let C be a given cost function, then update the
parameters according to:
Informatics and Mathematical Modelling / Intelligent Signal Processing
5
Morten Mørup
The idea behind multiplicative updates
Positive term
Negative term
Informatics and Mathematical Modelling / Intelligent Signal Processing
6
Morten Mørup
Non

negative matrix factorization (NMF)
(Lee & Seung

2001)
NMF gives Part based representation
(Lee & Seung
–
Nature 1999)
Informatics and Mathematical Modelling / Intelligent Signal Processing
7
Morten Mørup
The NMF decomposition is not unique
Simplical Cone
NMF only unique when data adequately spans the positive orthant
(Donoho & Stodden

2004)
Informatics and Mathematical Modelling / Intelligent Signal Processing
8
Morten Mørup
Sparse Coding NMF (SNMF)
(Eggert & Körner, 2004)
Informatics and Mathematical Modelling / Intelligent Signal Processing
9
Morten Mørup
Illustration (the swimmer problem)
True Expressions
Swimmer Articulations
NMF Expressions
SNMF Expressions
Informatics and Mathematical Modelling / Intelligent Signal Processing
10
Morten Mørup
Why sparseness?
Ensures uniqueness
Eases interpretability
(sparse representation
factor effects pertain to fewer dimensions
)
Can work as model selection
(Sparseness can turn off excess factors by letting them become zero)
Resolves over complete representations
(when model has many more free variables than data points)
Informatics and Mathematical Modelling / Intelligent Signal Processing
11
Morten Mørup
PART I: Convolutive PARAFAC (cPARAFAC)
Informatics and Mathematical Modelling / Intelligent Signal Processing
12
Morten Mørup
By cPARAFAC means PARAFAC convolutive in at
least one modality
Convolution can be in any combination of modalities

Single convolutive, double convolutive etc.
Convolution:
The process of generating
X
by convolving (sending) the sources
S
through the filter
A
Deconvolution:
The process of estimating
the filter
A
from
X
and
S
Informatics and Mathematical Modelling / Intelligent Signal Processing
13
Morten Mørup
Relation to other models
PARAFAC2 (Harshman, Kiers, Bro)
Shifted PARAFAC (Hong and Harshman, 2003)
cPARAFAC can account for echo effects
cPARAFAC becomes shifted PARAFAC
when convolutive filter is sparse
3
3
Informatics and Mathematical Modelling / Intelligent Signal Processing
14
Morten Mørup
Application example of cPARAFAC
Transcription and separation of music
Informatics and Mathematical Modelling / Intelligent Signal Processing
15
Morten Mørup
The ‘ideal’ Log

frequency Magnitude Spectrogram
of an instrument
Different notes played by an
instrument corresponds on a
logarithmic frequency scale to a
translation of the same harmonic
structure of a fixed temporal pattern
Tchaikovsky: Violin Concert in D Major
Mozart Sonate no,. 16 in C Major
Informatics and Mathematical Modelling / Intelligent Signal Processing
16
Morten Mørup
NMF 2D deconvolution (NMF2D
1
): The Basic Idea
Model a log

spectrogram of polyphonic music by an
extended type of non

negative matrix factorization:
–
The frequency signature of a specific note played by an
instrument has a fixed temporal pattern (echo)
model convolutive in time
–
Different notes of same instrument has same time

log

frequency signature but varying in fundamental frequency
(shift)
model convolutive in the log

frequency axis.
(
1
Mørup & Scmidt, 2006)
Informatics and Mathematical Modelling / Intelligent Signal Processing
17
Morten Mørup
NMF2D Model
NMF2D Model
–
extension of NMFD
1
:
(
1
Smaragdis, 2004, Eggert et al. 2004, Fitzgerald et al. 2005)
Informatics and Mathematical Modelling / Intelligent Signal Processing
18
Morten Mørup
Understanding the NMF2D Model
Informatics and Mathematical Modelling / Intelligent Signal Processing
19
Morten Mørup
The NMF2D has inherent ambiguity between the
structure in
W
and
H
To resolve this ambiguity sparsity is imposed
on
H
to force ambiguous structure onto
W
Informatics and Mathematical Modelling / Intelligent Signal Processing
20
Morten Mørup
NMF2D
SNMF2D
Real music example of how imposing sparseness
resolves the ambiguity between
W
and
H
Informatics and Mathematical Modelling / Intelligent Signal Processing
21
Morten Mørup
PARAFAC
(Harshman & Carrol and Chang 1970)
Factor analysis
(Charles Spearman ~1900)
Extension to multi channel analysis by the
PARAFAC model
Informatics and Mathematical Modelling / Intelligent Signal Processing
22
Morten Mørup
cPARAFAC: Sparse Non

negative Tensor Factor 2D
deconvolution (SNTF2D)
(Extension of Fitzgerald et al. 2005, 2006 to form a sparse double deconvolution)
Informatics and Mathematical Modelling / Intelligent Signal Processing
23
Morten Mørup
SNTF2D algorithms
Informatics and Mathematical Modelling / Intelligent Signal Processing
24
Morten Mørup
Tchaikovsky: Violin Concert in D Major
Mozart Sonate no. 16 in C Major
Informatics and Mathematical Modelling / Intelligent Signal Processing
25
Morten Mørup
Stereo recording of ”Fog is Lifting” by Carl Nielsen
Informatics and Mathematical Modelling / Intelligent Signal Processing
26
Morten Mørup
Applications
Applications
–
Source separation.
–
Music information retrieval.
–
Automatic music transcription (MIDI compression).
–
Source localization (beam forming)
Informatics and Mathematical Modelling / Intelligent Signal Processing
27
Morten Mørup
PART II: Higher Order NMF (HONMF)
Informatics and Mathematical Modelling / Intelligent Signal Processing
28
Morten Mørup
Higher Order Non

negative Matrix Factorization
(HONMF)
Motivation:
Many of the data sets previously explored by the Tucker model are
non

negative and could with good reason be decomposed under
constraints of non

negativity on all modalities including the core.
Spectroscopy data
(Smilde et al. 1999,2004, Andersson & Bro 1998, Nørgard & Ridder 1994)
Web mining
(Sun et al., 2004)
Image Analysis
(Vasilescu and Terzopoulos, 2002, Wang and Ahuja, 2003, Jian and Gong, 2005)
Semantic Differential Data
(Murakami and Kroonenberg, 2003)
And many more……
Informatics and Mathematical Modelling / Intelligent Signal Processing
29
Morten Mørup
However, non

negative Tucker decompositions are not
in general unique!
But

Imposing sparseness overcomes this problem!
Informatics and Mathematical Modelling / Intelligent Signal Processing
30
Morten Mørup
The Tucker Model
Informatics and Mathematical Modelling / Intelligent Signal Processing
31
Morten Mørup
Algorithms for HONMF
Informatics and Mathematical Modelling / Intelligent Signal Processing
32
Morten Mørup
Results
HONMF with sparseness, above imposed on the core can
be used for model selection

here indicating the PARAFAC
model is the appropriate model to the data.
Furthermore, the HONMF gives a more part based hence easy
interpretable solution than the HOSVD.
Informatics and Mathematical Modelling / Intelligent Signal Processing
33
Morten Mørup
Evaluation of uniqueness
Informatics and Mathematical Modelling / Intelligent Signal Processing
34
Morten Mørup
Data of a Flow Injection Analysis (Nørrgaard, 1994)
HONMF with sparse core and mixing captures unsupervised
the true mixing and model order!
Informatics and Mathematical Modelling / Intelligent Signal Processing
35
Morten Mørup
Conclusion
HONMF not in general unique, however when
imposing sparseness uniqueness can be achieved.
Algorithms devised for LS and KL able to impose
sparseness on any combination of modalities
The HONMF decompositions more part based hence
easier to interpret than other Tucker decompositions
such as the HOSVD.
Imposing sparseness can work as model selection
turning of excess components
Informatics and Mathematical Modelling / Intelligent Signal Processing
36
Morten Mørup
Coming soon in a MATLAB implementation near You
Informatics and Mathematical Modelling / Intelligent Signal Processing
37
Morten Mørup
References
Carroll, J. D. and Chang, J. J. Analysis of individual differences in multidimensional scaling via an N

way generalization of "E
ckart

Young" decomposition, Psychometrika 35 1970 283

319
Eggert, J. and Korner, E. Sparse coding and NMF. In Neural Networks volume 4, pages 2529

2533, 2004
Eggert, J et al Transformation

invariant representation and nmf. In Neural Networks, volume 4 , pages 535

2539, 2004
Fiitzgerald, D. et al. Non

negative tensor factorization for sound source separation. In proceedings of Irish Signals and System
s Conference, 2005
FitzGerald, D. and Coyle, E. C Sound source separation using shifted non.

negative tensor factorization. In ICASSP2006, 2006
Fitzgerald, D et al. Shifted non

negative matrix factorization for sound source separation. In Proceedings of the IEEE conferenc
e on Statistics in Signal Processing. 2005
Harshman, R. A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi

modal factor analysis},UC
LA Working Papers in Phonetics 16 1970 1
—
84
Harshman, Richard A.Harshman and Hong, Sungjin Lundy, Margaret E. Shifted factor analysis
—
Part I: Models and properties J. Chemo
metrics (17) pages 379
–
388, 2003
Kiers, Henk A. L. and Berge, Jos M. F. ten and Bro, Rasmus PARAFAC2

Part I. A direct fitting algorithm for the PARAFAC2 model,
Journal of Chemometrics (13) nr.3

4 pages 275

294, 1999
Lathauwer, Lieven De and Moor, Bart De and Vandewalle, Joos MULTILINEAR SINGULAR VALUE DECOMPOSITION.SIAM J. MATRIX ANAL. AP
PL.
2000 (21)1253
–
1278
Lee, D.D. and Seung, H.S. Algorithms for non

negative matrix factorization. In NIPS, pages 556

462, 2000
Lee, D.D and Seung, H.S. Learning the parts of objects by non

negative matrix factorization, NATURE 1999
Murakami, Takashi and Kroonenberg, Pieter M. Three

Mode Models and Individual Differences in Semantic Differential Data, Mult
ivariate Behavioral Research(38) no. 2 pages 247

283, 2003
Mørup, M. and Hansen, L.K.and Arnfred, S.M.Decomposing the time

frequency representation of EEG using nonnegative matrix and mul
ti

way factorization Technical report, Institute for Mathematical
Modeling, Technical University of Denmark, 2006a
Mørup, M. and Schmidt, M.N. Sparse non

negative matrix factor 2

D deconvolution. Technical report, Institute for Mathematical M
odeling, Tehcnical University of Denmark, 2006b
Mørup, M and Schmidt, M.N. Non

negative Tensor Factor 2D Deconvolution for multi

channel time

frequency analysis. Technical repo
rt, Institute for Mathematical Modeling, Technical University of
Denmark, 2006c
Schmidt, M.N. and Mørup, M. Non

negative matrix factor 2D deconvolution for blind single channel source separation. In ICA2006,
pages 700

707, 2006d
Mørup, M. and Hansen, L.K.and Arnfred, S.M. Algorithms for Sparse Higher Order Non

negative Matrix Factorization (HONMF), Techni
cal report, Institute for Mathematical Modeling, Technical
University of Denmark, 2006e
Nørgaard, L and Ridder, C.Rank annihilation factor analysis applied to flow injection analysis with photodiode

array detection C
hemometrics and Intelligent Laboratory Systems 1994 (23) 107

114
Schmidt, M.N. and Mørup, M. Sparse Non

negative Matrix Factor 2

D Deconvolution for Automatic Transcription of Polyphonic Music,
Technical report, Institute for Mathematical Modelling, Tehcnical
University of Denmark, 2005
Smaragdis, P. Non

negative Matrix Factor deconvolution; Extraction of multiple sound sources from monophonic inputs. Internation
al Symposium on independent Component Analysis and Blind Source
Separation (ICA)W
Smilde, Age K. Smilde and Tauller, Roma and Saurina, Javier and Bro, Rasmus, Calibration methods for complex second

order data A
nalytica Chimica Acta 1999 237

251
Sun, Jian

Tao and Zeng, Hua

Jun and Liu, Huanand Lu Yuchang and Chen Zheng CubeSVD: a novel approach to personalized Web search
WWW '05: Proceedings of the 14th international conference on
World Wide Web pages 382
—
390, 2005
Tamara G. Kolda Multilinear operators for higher

order decompositions technical report Sandia national laboratory 2006 SAND2006

2081.
Tucker, L. R. Some mathematical notes on three

mode factor analysis Psychometrika 31 1966 279
—
311
Welling, M. and Weber, M. Positive tensor factorization. Pattern Recogn. Lett. 2001
Vasilescu , M. A. O. and Terzopoulos , Demetri Multilinear Analysis of Image Ensembles: TensorFaces, ECCV '02: Proceedings of
th
e 7th European Conference on Computer Vision

Part I, 2002
Comments 0
Log in to post a comment