Extensions of Non-negative

finickyontarioAI and Robotics

Oct 29, 2013 (3 years and 1 month ago)

132 views

Informatics and Mathematical Modelling / Intelligent Signal Processing

1

Morten Mørup

Extensions of Non
-
negative
Matrix Factorization to Higher
Order data

Morten Mørup

Informatics and Mathematical Modeling

Intelligent Signal Processing

Technical University of Denmark

Informatics and Mathematical Modelling / Intelligent Signal Processing

2

Morten Mørup

Parts of the work done in
collaboration with

Sæby,
May

22
-
2006

Lars Kai Hansen, Professor

Department of Signal Processing

Informatics and Mathematical Modeling,

Technical University of Denmark

Mikkel N. Schmidt, Stud. PhD

Department of Signal Processing

Informatics and Mathematical Modeling,

Technical University of Denmark

Sidse M. Arnfred, Dr. Med. PhD

Cognitive Research Unit

Hvidovre Hospital

University Hospital of Copenhagen

Informatics and Mathematical Modelling / Intelligent Signal Processing

3

Morten Mørup

Outline


Non
-
negativity Matrix Factorization

(NMF)


Sparse coding

(SNMF)


Convolutive PARAFAC models (cPARAFAC)


Higher Order Non
-
negative Matrix Factorization

(an extension of NMF to the Tucker model)

Informatics and Mathematical Modelling / Intelligent Signal Processing

4

Morten Mørup

NMF is based on Gradient Descent

NMF:



V

WH

s.t.
W
i,d
,
H
d,j

0


Let C be a given cost function, then update the
parameters according to:

Informatics and Mathematical Modelling / Intelligent Signal Processing

5

Morten Mørup

The idea behind multiplicative updates

Positive term

Negative term

Informatics and Mathematical Modelling / Intelligent Signal Processing

6

Morten Mørup

Non
-
negative matrix factorization (NMF)

(Lee & Seung
-

2001)

NMF gives Part based representation

(Lee & Seung


Nature 1999)

Informatics and Mathematical Modelling / Intelligent Signal Processing

7

Morten Mørup

The NMF decomposition is not unique

Simplical Cone

NMF only unique when data adequately spans the positive orthant


(Donoho & Stodden
-

2004)

Informatics and Mathematical Modelling / Intelligent Signal Processing

8

Morten Mørup

Sparse Coding NMF (SNMF)

(Eggert & Körner, 2004)

Informatics and Mathematical Modelling / Intelligent Signal Processing

9

Morten Mørup

Illustration (the swimmer problem)

True Expressions

Swimmer Articulations

NMF Expressions

SNMF Expressions

Informatics and Mathematical Modelling / Intelligent Signal Processing

10

Morten Mørup

Why sparseness?


Ensures uniqueness


Eases interpretability

(sparse representation


factor effects pertain to fewer dimensions
)


Can work as model selection

(Sparseness can turn off excess factors by letting them become zero)


Resolves over complete representations

(when model has many more free variables than data points)

Informatics and Mathematical Modelling / Intelligent Signal Processing

11

Morten Mørup

PART I: Convolutive PARAFAC (cPARAFAC)

Informatics and Mathematical Modelling / Intelligent Signal Processing

12

Morten Mørup

By cPARAFAC means PARAFAC convolutive in at
least one modality

Convolution can be in any combination of modalities

-
Single convolutive, double convolutive etc.

Convolution:

The process of generating
X

by convolving (sending) the sources
S


through the filter
A

Deconvolution:

The process of estimating

the filter

A
from
X

and
S


Informatics and Mathematical Modelling / Intelligent Signal Processing

13

Morten Mørup

Relation to other models


PARAFAC2 (Harshman, Kiers, Bro)


Shifted PARAFAC (Hong and Harshman, 2003)


cPARAFAC can account for echo effects

cPARAFAC becomes shifted PARAFAC

when convolutive filter is sparse

3

3

Informatics and Mathematical Modelling / Intelligent Signal Processing

14

Morten Mørup

Application example of cPARAFAC

Transcription and separation of music

Informatics and Mathematical Modelling / Intelligent Signal Processing

15

Morten Mørup

The ‘ideal’ Log
-
frequency Magnitude Spectrogram
of an instrument


Different notes played by an

instrument corresponds on a

logarithmic frequency scale to a

translation of the same harmonic

structure of a fixed temporal pattern

Tchaikovsky: Violin Concert in D Major

Mozart Sonate no,. 16 in C Major

Informatics and Mathematical Modelling / Intelligent Signal Processing

16

Morten Mørup

NMF 2D deconvolution (NMF2D
1
): The Basic Idea


Model a log
-
spectrogram of polyphonic music by an
extended type of non
-
negative matrix factorization:


The frequency signature of a specific note played by an
instrument has a fixed temporal pattern (echo)



model convolutive in time


Different notes of same instrument has same time
-
log
-
frequency signature but varying in fundamental frequency
(shift)



model convolutive in the log
-
frequency axis.

(
1
Mørup & Scmidt, 2006)

Informatics and Mathematical Modelling / Intelligent Signal Processing

17

Morten Mørup

NMF2D Model


NMF2D Model


extension of NMFD
1
:





















(
1
Smaragdis, 2004, Eggert et al. 2004, Fitzgerald et al. 2005)

Informatics and Mathematical Modelling / Intelligent Signal Processing

18

Morten Mørup

Understanding the NMF2D Model

Informatics and Mathematical Modelling / Intelligent Signal Processing

19

Morten Mørup

The NMF2D has inherent ambiguity between the
structure in
W

and
H


To resolve this ambiguity sparsity is imposed

on
H

to force ambiguous structure onto
W

Informatics and Mathematical Modelling / Intelligent Signal Processing

20

Morten Mørup

NMF2D

SNMF2D

Real music example of how imposing sparseness
resolves the ambiguity between
W

and
H

Informatics and Mathematical Modelling / Intelligent Signal Processing

21

Morten Mørup

PARAFAC

(Harshman & Carrol and Chang 1970)

Factor analysis

(Charles Spearman ~1900)

Extension to multi channel analysis by the
PARAFAC model

Informatics and Mathematical Modelling / Intelligent Signal Processing

22

Morten Mørup

cPARAFAC: Sparse Non
-
negative Tensor Factor 2D
deconvolution (SNTF2D)

(Extension of Fitzgerald et al. 2005, 2006 to form a sparse double deconvolution)

Informatics and Mathematical Modelling / Intelligent Signal Processing

23

Morten Mørup

SNTF2D algorithms

Informatics and Mathematical Modelling / Intelligent Signal Processing

24

Morten Mørup

Tchaikovsky: Violin Concert in D Major

Mozart Sonate no. 16 in C Major

Informatics and Mathematical Modelling / Intelligent Signal Processing

25

Morten Mørup

Stereo recording of ”Fog is Lifting” by Carl Nielsen

Informatics and Mathematical Modelling / Intelligent Signal Processing

26

Morten Mørup

Applications


Applications


Source separation.


Music information retrieval.


Automatic music transcription (MIDI compression).


Source localization (beam forming)


Informatics and Mathematical Modelling / Intelligent Signal Processing

27

Morten Mørup

PART II: Higher Order NMF (HONMF)

Informatics and Mathematical Modelling / Intelligent Signal Processing

28

Morten Mørup

Higher Order Non
-
negative Matrix Factorization
(HONMF)

Motivation:

Many of the data sets previously explored by the Tucker model are
non
-
negative and could with good reason be decomposed under
constraints of non
-
negativity on all modalities including the core.



Spectroscopy data




(Smilde et al. 1999,2004, Andersson & Bro 1998, Nørgard & Ridder 1994)


Web mining



(Sun et al., 2004)


Image Analysis

(Vasilescu and Terzopoulos, 2002, Wang and Ahuja, 2003, Jian and Gong, 2005)


Semantic Differential Data

(Murakami and Kroonenberg, 2003)


And many more……

Informatics and Mathematical Modelling / Intelligent Signal Processing

29

Morten Mørup

However, non
-
negative Tucker decompositions are not

in general unique!

But
-

Imposing sparseness overcomes this problem!

Informatics and Mathematical Modelling / Intelligent Signal Processing

30

Morten Mørup

The Tucker Model

Informatics and Mathematical Modelling / Intelligent Signal Processing

31

Morten Mørup

Algorithms for HONMF

Informatics and Mathematical Modelling / Intelligent Signal Processing

32

Morten Mørup

Results

HONMF with sparseness, above imposed on the core can

be used for model selection
-
here indicating the PARAFAC

model is the appropriate model to the data.

Furthermore, the HONMF gives a more part based hence easy
interpretable solution than the HOSVD.

Informatics and Mathematical Modelling / Intelligent Signal Processing

33

Morten Mørup

Evaluation of uniqueness

Informatics and Mathematical Modelling / Intelligent Signal Processing

34

Morten Mørup

Data of a Flow Injection Analysis (Nørrgaard, 1994)

HONMF with sparse core and mixing captures unsupervised

the true mixing and model order!

Informatics and Mathematical Modelling / Intelligent Signal Processing

35

Morten Mørup

Conclusion


HONMF not in general unique, however when
imposing sparseness uniqueness can be achieved.


Algorithms devised for LS and KL able to impose
sparseness on any combination of modalities


The HONMF decompositions more part based hence
easier to interpret than other Tucker decompositions
such as the HOSVD.


Imposing sparseness can work as model selection
turning of excess components

Informatics and Mathematical Modelling / Intelligent Signal Processing

36

Morten Mørup

Coming soon in a MATLAB implementation near You

Informatics and Mathematical Modelling / Intelligent Signal Processing

37

Morten Mørup

References

Carroll, J. D. and Chang, J. J. Analysis of individual differences in multidimensional scaling via an N
-
way generalization of "E
ckart
-
Young" decomposition, Psychometrika 35 1970 283
--
319

Eggert, J. and Korner, E. Sparse coding and NMF. In Neural Networks volume 4, pages 2529
-
2533, 2004

Eggert, J et al Transformation
-
invariant representation and nmf. In Neural Networks, volume 4 , pages 535
-
2539, 2004

Fiitzgerald, D. et al. Non
-
negative tensor factorization for sound source separation. In proceedings of Irish Signals and System
s Conference, 2005

FitzGerald, D. and Coyle, E. C Sound source separation using shifted non.
-
negative tensor factorization. In ICASSP2006, 2006

Fitzgerald, D et al. Shifted non
-
negative matrix factorization for sound source separation. In Proceedings of the IEEE conferenc
e on Statistics in Signal Processing. 2005

Harshman, R. A. Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi
-
modal factor analysis},UC
LA Working Papers in Phonetics 16 1970 1

84

Harshman, Richard A.Harshman and Hong, Sungjin Lundy, Margaret E. Shifted factor analysis

Part I: Models and properties J. Chemo
metrics (17) pages 379

388, 2003

Kiers, Henk A. L. and Berge, Jos M. F. ten and Bro, Rasmus PARAFAC2

-

Part I. A direct fitting algorithm for the PARAFAC2 model,

Journal of Chemometrics (13) nr.3
-
4 pages 275
-
294, 1999

Lathauwer, Lieven De and Moor, Bart De and Vandewalle, Joos MULTILINEAR SINGULAR VALUE DECOMPOSITION.SIAM J. MATRIX ANAL. AP
PL.
2000 (21)1253

1278

Lee, D.D. and Seung, H.S. Algorithms for non
-
negative matrix factorization. In NIPS, pages 556
-
462, 2000

Lee, D.D and Seung, H.S. Learning the parts of objects by non
-
negative matrix factorization, NATURE 1999

Murakami, Takashi and Kroonenberg, Pieter M. Three
-
Mode Models and Individual Differences in Semantic Differential Data, Mult
ivariate Behavioral Research(38) no. 2 pages 247
-
283, 2003

Mørup, M. and Hansen, L.K.and Arnfred, S.M.Decomposing the time
-
frequency representation of EEG using nonnegative matrix and mul
ti
-
way factorization Technical report, Institute for Mathematical
Modeling, Technical University of Denmark, 2006a

Mørup, M. and Schmidt, M.N. Sparse non
-
negative matrix factor 2
-
D deconvolution. Technical report, Institute for Mathematical M
odeling, Tehcnical University of Denmark, 2006b

Mørup, M and Schmidt, M.N. Non
-
negative Tensor Factor 2D Deconvolution for multi
-
channel time
-
frequency analysis. Technical repo
rt, Institute for Mathematical Modeling, Technical University of
Denmark, 2006c

Schmidt, M.N. and Mørup, M. Non
-
negative matrix factor 2D deconvolution for blind single channel source separation. In ICA2006,
pages 700
-
707, 2006d

Mørup, M. and Hansen, L.K.and Arnfred, S.M. Algorithms for Sparse Higher Order Non
-
negative Matrix Factorization (HONMF), Techni
cal report, Institute for Mathematical Modeling, Technical
University of Denmark, 2006e

Nørgaard, L and Ridder, C.Rank annihilation factor analysis applied to flow injection analysis with photodiode
-
array detection C
hemometrics and Intelligent Laboratory Systems 1994 (23) 107
-
114

Schmidt, M.N. and Mørup, M. Sparse Non
-
negative Matrix Factor 2
-
D Deconvolution for Automatic Transcription of Polyphonic Music,

Technical report, Institute for Mathematical Modelling, Tehcnical
University of Denmark, 2005

Smaragdis, P. Non
-
negative Matrix Factor deconvolution; Extraction of multiple sound sources from monophonic inputs. Internation
al Symposium on independent Component Analysis and Blind Source
Separation (ICA)W

Smilde, Age K. Smilde and Tauller, Roma and Saurina, Javier and Bro, Rasmus, Calibration methods for complex second
-
order data A
nalytica Chimica Acta 1999 237
-
251

Sun, Jian
-
Tao and Zeng, Hua
-
Jun and Liu, Huanand Lu Yuchang and Chen Zheng CubeSVD: a novel approach to personalized Web search

WWW '05: Proceedings of the 14th international conference on
World Wide Web pages 382

390, 2005

Tamara G. Kolda Multilinear operators for higher
-
order decompositions technical report Sandia national laboratory 2006 SAND2006
-
2081.

Tucker, L. R. Some mathematical notes on three
-
mode factor analysis Psychometrika 31 1966 279

311

Welling, M. and Weber, M. Positive tensor factorization. Pattern Recogn. Lett. 2001

Vasilescu , M. A. O. and Terzopoulos , Demetri Multilinear Analysis of Image Ensembles: TensorFaces, ECCV '02: Proceedings of

th
e 7th European Conference on Computer Vision
-
Part I, 2002