Matrix Extensions to Sparse
Recovery
Yi Ma
1,2
Allen Yang
3
John Wright
1
CVPR Tutorial, June 20, 2009
1
Microsoft
Research Asia
3
University of
California Berkeley
2
University of Illinois
at Urbana

Champaign
TexPoint
fonts used in EMF.
Read the TexPoint manual before you delete this box.:
A
A
A
A
A
A
A
A
A
A
A
FINAL TOPIC
–
Generalizations: sparsity to degeneracy
The tools and phenomena underlying
sparse recovery
generalize very nicely to
low

rank matrix recovery
???
FINAL TOPIC
–
Generalizations: sparsity to degeneracy
The tools and phenomena underlying
sparse recovery
generalize very nicely to
low

rank matrix recovery
Matrix completion:
Given an incomplete subset of the entries of a
low

rank matrix, fill in the missing values.
Robust PCA:
Given a low

rank matrix which has been grossly
corrupted, recover the original matrix.
???
Face images
Degeneracy: illumination models
Errors: occlusion, corruption
Relevancy data
Degeneracy: user preferences co

predict
Errors: Missing rankings, manipulation
Video
Degeneracy: temporal, dynamic structures
Errors: anomalous events, mismatches…
Examples of degenerate data:
THIS TALK
–
From sparse recovery to low

rank recovery
KEY ANALOGY
–
Connections between rank and sparsity
Sparse recovery
Rank minimization
Unknown
Vector
x
Matrix
A
Observations
y = Ax
y
= L[A]
(linear map)
Combinatorial
objective
Convex
relaxation
Algorithmic
tools
Linear programming
Semidefinite
programming
KEY ANALOGY
–
Connections between rank and sparsity
This talk: exploiting this connection for
matrix completion
and
RPCA
Sparse recovery
Rank minimization
Unknown
Vector
x
Matrix
A
Observations
y = Ax
y
= L[A]
(linear map)
Combinatorial
objective
Convex
relaxation
Algorithmic
tools
Linear programming
Semidefinite
programming
CLASSICAL PCA
–
Fitting degenerate data
If degenerate observations are stacked as columns of a matrix
then
CLASSICAL PCA
–
Fitting degenerate data
If degenerate observations are stacked as columns of a matrix
then
Principal Component Analysis
via singular value decomposition:
•
Stable, efficient computation
•
Optimal estimate of under
iid
Gaussian noise
•
Fundamental statistical tool, huge impact in vision, search,
bioinformatics
CLASSICAL PCA
–
Fitting degenerate data
If degenerate observations are stacked as columns of a matrix
then
But…
PCA breaks down under even a single corrupted observation.
Principal Component Analysis via singular value decomposition:
•
Stable, efficient computation
•
Optimal estimate of under
iid
Gaussian noise
•
Fundamental statistical tool, huge impact in vision, search,
bioinformatics
ROBUST PCA
–
Problem formulation
…
…
D

observation
A
–
low

rank
E
–
sparse error
…
Properties of the errors:
•
Each multivariate data sample (column) may be corrupted in some entries
•
Corruption can be arbitrarily large in magnitude (not Gaussian!)
Problem
: Given recover .
Low

rank structure
Sparse errors
ROBUST PCA
–
Problem formulation
Problem
: Given recover .
Low

rank structure
Sparse errors
Numerous heuristic methods in the literature:
•
Random sampling [
Fischler
and
Bolles
‘81]
•
Multivariate trimming [
Gnanadesikan
and Kettering ‘72]
•
Alternating minimization [
Ke
and
Kanade
‘03]
•
Influence functions [de la Torre and Black ‘03]
•
No polynomial

time algorithm with strong performance guarantees!
…
…
D

observation
A
–
low

rank
E
–
sparse error
…
ROBUST PCA
–
Semidefinite
programming formulation
Seek the lowest

rank that agrees with the data up to some sparse error:
ROBUST PCA
–
Semidefinite
programming formulation
Seek the lowest

rank that agrees with the data up to some sparse error:
Not directly tractable, relax:
ROBUST PCA
–
Semidefinite
programming formulation
Seek the lowest

rank that agrees with the data up to some sparse error:
Not directly tractable, relax:
Semidefinite
program, solvable in polynomial time
Convex envelope over
MATRIX COMPLETION
–
Motivation for the nuclear norm
Related problem:
we observe only a small
known
subset
of entries of a rank

matrix . Can we exactly recover ?
MATRIX COMPLETION
–
Motivation for the nuclear norm
Related problem:
recover a rank matrix from a
known
subset of entries
Convex optimization heuristic
[
Candes
and
Recht
]
:
Spectral trimming also succeeds with for
For
incoherent
,
exact recovery
with
[
Keshavan
,
Montanari
and Oh
]
[
Candes
and Tao]
ROBUST PCA
–
Exact recovery?
CONJECTURE
:
If with sufficiently low

rank and
exactly recovers .
Sparsity of error
sufficiently sparse, then solving
Empirical evidence
: probability of correct recovery
vs
rank and sparsity
Perfect recovery
Rank
Decompose as or ?
ROBUST PCA
–
Which matrices and which errors?
Fundamental ambiguity
–
very sparse matrices are also low

rank:
rank

1
rank

0
0

sparse
1

sparse
Obviously we can only hope to uniquely recover that are
incoherent
with the standard basis.
Can we recover almost all low

rank matrices from almost all sparse errors?
ROBUST PCA
–
Which matrices and which errors?
Random orthogonal model
(of rank r)
[
Candes
&
Recht
‘08]
:
independent samples from invariant measure
on
Steifel
manifold of
orthobases
of rank r.
arbitrary.
ROBUST PCA
–
Which matrices and which errors?
Random orthogonal model
(of rank r)
[
Candes
&
Recht
‘08]
:
independent samples from invariant measure
on
Steifel
manifold of
orthobases
of rank r.
arbitrary.
Bernoulli error signs

and

support
(with parameter ):
Magnitude of is arbitrary.
MAIN RESULT
–
Exact Solution of Robust PCA
“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”
BONUS RESULT
–
Matrix completion in proportional growth
“Convex optimization exactly recovers matrices of rank , even with
entries missing!”
MATRIX COMPLETION
–
Contrast with literature
•
[
Candes
and Tao 2009]:
Correct completion
whp
for
Does not apply to the large

rank case
•
This work:
Correct completion
whp
for even with
Proof exploits rich regularity and independence in random orthogonal model.
Caveats:

[C

T ‘09] tighter for small r.

[C

T ‘09] generalizes better to other matrix ensembles.
MAIN RESULT
–
Exact Solution of Robust PCA
“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”
ROBUST PCA
–
Solving the convex program
Semidefinite
program in
millions of unknowns.
Scalable solution
: apply a first

order method with convergence to
Sequence of quadratic approximations
[
Nesterov
, Beck &
Teboulle
]
:
Solved via
soft
thresholding
(
E
), and
singular value
thresholding
(
A
).
ROBUST PCA
–
Solving the convex program
•
Iteration complexity for suboptimal solution.
•
Dramatic practical gains from continuation
SIMULATION
–
Recovery in various growth scenarios
Correct recovery with and fixed, increasing.
Empirically, almost constant number of iterations:
Provably robust PCA at only a constant factor more computation
than conventional PCA.
SIMULATION
–
Phase Transition in Rank and
Sparsity
Fraction of successes with , varying
(10 trials)
Fraction of successes with , varying
(65 trials)
[0,.5] x [0,.5]
[0,1] x [0,1]
[0,1] x [0,1]
[0,.4] x [0,.4]
EXAMPLE
–
Background modeling from video
Video
Low

rank
appx
.
Sparse error
Static camera
surveillance video
200 frames,
72 x 88 pixels,
Significant foreground
motion
EXAMPLE
–
Background modeling from video
Video
Low

rank
appx
.
Sparse error
Static camera
surveillance video
550 frames,
64 x 80 pixels,
significant illumination
variation
Background
variation
Anomalous activity
EXAMPLE
–
Faces under varying illumination
…
…
RPCA
29 images of one
person under varying
lighting:
EXAMPLE
–
Faces under varying illumination
…
…
RPCA
29 images of one
person under varying
lighting:
Self

shadowing
Specularity
EXAMPLE
–
Face tracking and alignment
Initial alignment
, inappropriate for recognition:
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
EXAMPLE
–
Face tracking and alignment
Final result
: per

pixel alignment
EXAMPLE
–
Face tracking and alignment
Final result
: per

pixel alignment
SIMULATION
–
Phase Transition in Rank and
Sparsity
Fraction of successes with , varying
(10 trials)
Fraction of successes with , varying
(65 trials)
[0,.5] x [0,.5]
[0,1] x [0,1]
[0,1] x [0,1]
[0,.4] x [0,.4]
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
Hypothesized breakdown behavior as m
∞
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
What we know so far:
This work
Classical PCA
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
CONJECTURE I:
convex programming succeeds in proportional growth
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
CONJECTURE II:
for small ranks ,
any fraction of errors can eventually be corrected.
Similar to
Dense Error Correction via L1 Minimization
, Wright and Ma ‘08
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
CONJECTURE III:
for any rank fraction, ,
there exists a nonzero fraction of errors that can eventually be
corrected with high probability.
CONJECTURES
–
Phase Transition in Rank and
Sparsity
1
1
0
0
CONJECTURE IV:
there is an
asymptotically sharp phase transition
between correct recovery with overwhelming probability, and
failure with overwhelming probability.
CONJECTURES
–
Connections to Matrix Completion
Our results also suggest the possibility of a proportional growth
phase transition for
matrix completion
.
1
1
0
0
•
How do the two breakdown points compare?
•
How much is gained by knowing the location of the corruption?
Robust PCA
Matrix Completion
Similar to
Recht
,
Xu
and
Hassibi
‘08
Matrix Completion
Robust PCA
FUTURE WORK
–
Stronger results on RPCA?
•
RPCA with noise and errors
:
Tradeoff between estimation error and robustness to corruption?
•
Deterministic conditions on the matrix
•
Simultaneous error correction and matrix completion:
bounded noise
(e.g., Gaussian)
Conjecture: stable recovery with
we observe
•
Faster algorithms:
Smarter continuation strategies
Parallel implementations, GPU, multi

machine
•
Further applications:
Computer vision: photometric stereo, tracking, video repair
Relevancy data: search, ranking and collaborative filtering
Bioinformatics
System Identification
FUTURE WORK
–
Algorithms and Applications
•
Reference:
Robust Principal Component Analysis:
Exact Recovery of Corrupted Low

Rank Matrices by Convex Optimization
submitted to the Journal of the ACM
•
Collaborators:
Prof
. Yi Ma
(UIUC, MSRA)
Dr.
Zhouchen
Lin
(MSRA)
Dr.
Shankar
Rao
(UIUC)
Arvind
Ganesh
(UIUC)
Yigang
Peng
(MSRA)
•
Funding:
Microsoft Research Fellowship (sponsored by Live Labs)
Grants
NSF CRS

EHS

0509151, NSF CCF

TF

0514955,
ONR
YIP N00014

04

1

0633, NSF IIS 07

03756
REFERENCES + ACKNOWLEDGEMENT
Questions, please?
THANK
YOU!
John Wright
Robust PCA: Exact Recovery of Corrupted Low

Rank Matrices
Comments 0
Log in to post a comment