Dense Error Correction via l1-minimization - University of California ...

hordeprobableΒιοτεχνολογία

4 Οκτ 2013 (πριν από 3 χρόνια και 10 μήνες)

72 εμφανίσεις


Matrix Extensions to Sparse
Recovery


Yi Ma
1,2

Allen Yang
3

John Wright
1





CVPR Tutorial, June 20, 2009

1
Microsoft
Research Asia

3
University of
California Berkeley

2
University of Illinois

at Urbana
-
Champaign

TexPoint

fonts used in EMF.

Read the TexPoint manual before you delete this box.:
A
A
A
A
A
A
A
A
A
A
A

FINAL TOPIC


Generalizations: sparsity to degeneracy

The tools and phenomena underlying
sparse recovery


generalize very nicely to
low
-
rank matrix recovery


???

FINAL TOPIC


Generalizations: sparsity to degeneracy

The tools and phenomena underlying
sparse recovery


generalize very nicely to
low
-
rank matrix recovery



Matrix completion:


Given an incomplete subset of the entries of a



low
-
rank matrix, fill in the missing values.






Robust PCA:


Given a low
-
rank matrix which has been grossly



corrupted, recover the original matrix.



???

Face images


Degeneracy: illumination models


Errors: occlusion, corruption

Relevancy data


Degeneracy: user preferences co
-
predict


Errors: Missing rankings, manipulation

Video


Degeneracy: temporal, dynamic structures


Errors: anomalous events, mismatches…

Examples of degenerate data:

THIS TALK


From sparse recovery to low
-
rank recovery

KEY ANALOGY


Connections between rank and sparsity

Sparse recovery

Rank minimization

Unknown


Vector

x


Matrix
A

Observations


y = Ax


y

= L[A]

(linear map)

Combinatorial


objective

Convex
relaxation




Algorithmic
tools


Linear programming


Semidefinite

programming

KEY ANALOGY


Connections between rank and sparsity

This talk: exploiting this connection for
matrix completion
and
RPCA

Sparse recovery

Rank minimization

Unknown


Vector

x


Matrix
A

Observations


y = Ax


y

= L[A]

(linear map)

Combinatorial


objective

Convex
relaxation




Algorithmic
tools


Linear programming


Semidefinite

programming

CLASSICAL PCA


Fitting degenerate data

If degenerate observations are stacked as columns of a matrix


then

CLASSICAL PCA


Fitting degenerate data


If degenerate observations are stacked as columns of a matrix


then

Principal Component Analysis
via singular value decomposition:



Stable, efficient computation




Optimal estimate of under
iid

Gaussian noise




Fundamental statistical tool, huge impact in vision, search,


bioinformatics

CLASSICAL PCA


Fitting degenerate data

If degenerate observations are stacked as columns of a matrix


then

But…
PCA breaks down under even a single corrupted observation.

Principal Component Analysis via singular value decomposition:



Stable, efficient computation




Optimal estimate of under
iid

Gaussian noise




Fundamental statistical tool, huge impact in vision, search,


bioinformatics

ROBUST PCA


Problem formulation





D
-

observation

A


low
-
rank

E


sparse error



Properties of the errors:




Each multivariate data sample (column) may be corrupted in some entries




Corruption can be arbitrarily large in magnitude (not Gaussian!)



Problem
: Given recover .

Low
-
rank structure

Sparse errors

ROBUST PCA


Problem formulation



Problem
: Given recover .

Low
-
rank structure

Sparse errors

Numerous heuristic methods in the literature:




Random sampling [
Fischler

and
Bolles

‘81]



Multivariate trimming [
Gnanadesikan

and Kettering ‘72]



Alternating minimization [
Ke

and
Kanade

‘03]



Influence functions [de la Torre and Black ‘03]





No polynomial
-
time algorithm with strong performance guarantees!





D
-

observation

A


low
-
rank

E


sparse error



ROBUST PCA


Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

ROBUST PCA


Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

ROBUST PCA


Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

Semidefinite

program, solvable in polynomial time

Convex envelope over

MATRIX COMPLETION


Motivation for the nuclear norm

Related problem:
we observe only a small
known

subset




of entries of a rank
-

matrix . Can we exactly recover ?



MATRIX COMPLETION


Motivation for the nuclear norm

Related problem:
recover a rank matrix from a
known

subset of entries




Convex optimization heuristic
[
Candes

and
Recht
]
:

Spectral trimming also succeeds with for

For
incoherent

,
exact recovery
with

[
Keshavan
,
Montanari

and Oh
]

[
Candes

and Tao]

ROBUST PCA


Exact recovery?

CONJECTURE
:

If with sufficiently low
-
rank and


exactly recovers .

Sparsity of error

sufficiently sparse, then solving

Empirical evidence
: probability of correct recovery
vs

rank and sparsity

Perfect recovery

Rank

Decompose as or ?

ROBUST PCA


Which matrices and which errors?

Fundamental ambiguity


very sparse matrices are also low
-
rank:

rank
-
1

rank
-
0

0
-
sparse

1
-
sparse

Obviously we can only hope to uniquely recover that are


incoherent

with the standard basis.



Can we recover almost all low
-
rank matrices from almost all sparse errors?

ROBUST PCA


Which matrices and which errors?

Random orthogonal model
(of rank r)
[
Candes

&
Recht

‘08]
:


independent samples from invariant measure


on
Steifel

manifold of
orthobases

of rank r.

arbitrary.

ROBUST PCA


Which matrices and which errors?

Random orthogonal model
(of rank r)
[
Candes

&
Recht

‘08]
:


independent samples from invariant measure


on
Steifel

manifold of
orthobases

of rank r.

arbitrary.

Bernoulli error signs
-
and
-
support
(with parameter ):

Magnitude of is arbitrary.

MAIN RESULT


Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”

BONUS RESULT


Matrix completion in proportional growth

“Convex optimization exactly recovers matrices of rank , even with

entries missing!”

MATRIX COMPLETION


Contrast with literature



[
Candes

and Tao 2009]:




Correct completion
whp

for

Does not apply to the large
-
rank case



This work:




Correct completion
whp

for even with

Proof exploits rich regularity and independence in random orthogonal model.

Caveats:




-

[C
-
T ‘09] tighter for small r.


-

[C
-
T ‘09] generalizes better to other matrix ensembles.

MAIN RESULT


Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”

ROBUST PCA


Solving the convex program

Semidefinite

program in
millions of unknowns.

Scalable solution
: apply a first
-
order method with convergence to

Sequence of quadratic approximations
[
Nesterov
, Beck &
Teboulle
]
:

Solved via
soft
thresholding

(
E
), and
singular value
thresholding

(
A
).

ROBUST PCA


Solving the convex program



Iteration complexity for suboptimal solution.





Dramatic practical gains from continuation

SIMULATION


Recovery in various growth scenarios

Correct recovery with and fixed, increasing.

Empirically, almost constant number of iterations:



Provably robust PCA at only a constant factor more computation
than conventional PCA.

SIMULATION


Phase Transition in Rank and
Sparsity

Fraction of successes with , varying
(10 trials)

Fraction of successes with , varying
(65 trials)

[0,.5] x [0,.5]

[0,1] x [0,1]

[0,1] x [0,1]

[0,.4] x [0,.4]

EXAMPLE


Background modeling from video

Video

Low
-
rank
appx
.

Sparse error

Static camera

surveillance video


200 frames,

72 x 88 pixels,


Significant foreground

motion

EXAMPLE


Background modeling from video

Video

Low
-
rank
appx
.

Sparse error

Static camera

surveillance video


550 frames,

64 x 80 pixels,


significant illumination

variation

Background

variation

Anomalous activity

EXAMPLE


Faces under varying illumination





RPCA

29 images of one
person under varying
lighting:


EXAMPLE


Faces under varying illumination





RPCA

29 images of one
person under varying
lighting:



Self
-


shadowing

Specularity

EXAMPLE


Face tracking and alignment

Initial alignment
, inappropriate for recognition:

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

EXAMPLE


Face tracking and alignment

Final result
: per
-
pixel alignment

EXAMPLE


Face tracking and alignment

Final result
: per
-
pixel alignment

SIMULATION


Phase Transition in Rank and
Sparsity

Fraction of successes with , varying
(10 trials)

Fraction of successes with , varying
(65 trials)

[0,.5] x [0,.5]

[0,1] x [0,1]

[0,1] x [0,1]

[0,.4] x [0,.4]

CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

Hypothesized breakdown behavior as m





CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

What we know so far:

This work

Classical PCA

CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE I:
convex programming succeeds in proportional growth

CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE II:
for small ranks ,


any fraction of errors can eventually be corrected.

Similar to
Dense Error Correction via L1 Minimization
, Wright and Ma ‘08

CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE III:
for any rank fraction, ,


there exists a nonzero fraction of errors that can eventually be


corrected with high probability.

CONJECTURES


Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE IV:
there is an
asymptotically sharp phase transition



between correct recovery with overwhelming probability, and


failure with overwhelming probability.

CONJECTURES


Connections to Matrix Completion

Our results also suggest the possibility of a proportional growth
phase transition for
matrix completion
.

1

1

0

0



How do the two breakdown points compare?



How much is gained by knowing the location of the corruption?

Robust PCA

Matrix Completion

Similar to
Recht
,
Xu

and
Hassibi

‘08

Matrix Completion

Robust PCA

FUTURE WORK


Stronger results on RPCA?



RPCA with noise and errors
:







Tradeoff between estimation error and robustness to corruption?





Deterministic conditions on the matrix





Simultaneous error correction and matrix completion:


bounded noise

(e.g., Gaussian)

Conjecture: stable recovery with

we observe




Faster algorithms:



Smarter continuation strategies



Parallel implementations, GPU, multi
-
machine





Further applications:



Computer vision: photometric stereo, tracking, video repair



Relevancy data: search, ranking and collaborative filtering



Bioinformatics



System Identification

FUTURE WORK


Algorithms and Applications



Reference:


Robust Principal Component Analysis:


Exact Recovery of Corrupted Low
-
Rank Matrices by Convex Optimization


submitted to the Journal of the ACM




Collaborators:


Prof
. Yi Ma
(UIUC, MSRA)

Dr.
Zhouchen

Lin
(MSRA)

Dr.
Shankar
Rao

(UIUC)

Arvind

Ganesh

(UIUC)

Yigang

Peng

(MSRA)



Funding:


Microsoft Research Fellowship (sponsored by Live Labs)

Grants
NSF CRS
-
EHS
-
0509151, NSF CCF
-
TF
-
0514955,
ONR
YIP N00014
-
04
-
1
-
0633, NSF IIS 07
-
03756


REFERENCES + ACKNOWLEDGEMENT

Questions, please?

THANK
YOU!

John Wright
Robust PCA: Exact Recovery of Corrupted Low
-
Rank Matrices