# Dense Error Correction via l1-minimization - University of California ...

Biotechnology

Oct 4, 2013 (4 years and 7 months ago)

92 views

Matrix Extensions to Sparse
Recovery

Yi Ma
1,2

Allen Yang
3

John Wright
1

CVPR Tutorial, June 20, 2009

1
Microsoft
Research Asia

3
University of
California Berkeley

2
University of Illinois

at Urbana
-
Champaign

TexPoint

fonts used in EMF.

Read the TexPoint manual before you delete this box.:
A
A
A
A
A
A
A
A
A
A
A

FINAL TOPIC

Generalizations: sparsity to degeneracy

The tools and phenomena underlying
sparse recovery

generalize very nicely to
low
-
rank matrix recovery

???

FINAL TOPIC

Generalizations: sparsity to degeneracy

The tools and phenomena underlying
sparse recovery

generalize very nicely to
low
-
rank matrix recovery

Matrix completion:

Given an incomplete subset of the entries of a

low
-
rank matrix, fill in the missing values.

Robust PCA:

Given a low
-
rank matrix which has been grossly

corrupted, recover the original matrix.

???

Face images

Degeneracy: illumination models

Errors: occlusion, corruption

Relevancy data

Degeneracy: user preferences co
-
predict

Errors: Missing rankings, manipulation

Video

Degeneracy: temporal, dynamic structures

Errors: anomalous events, mismatches…

Examples of degenerate data:

THIS TALK

From sparse recovery to low
-
rank recovery

KEY ANALOGY

Connections between rank and sparsity

Sparse recovery

Rank minimization

Unknown

Vector

x

Matrix
A

Observations

y = Ax

y

= L[A]

(linear map)

Combinatorial

objective

Convex
relaxation

Algorithmic
tools

Linear programming

Semidefinite

programming

KEY ANALOGY

Connections between rank and sparsity

This talk: exploiting this connection for
matrix completion
and
RPCA

Sparse recovery

Rank minimization

Unknown

Vector

x

Matrix
A

Observations

y = Ax

y

= L[A]

(linear map)

Combinatorial

objective

Convex
relaxation

Algorithmic
tools

Linear programming

Semidefinite

programming

CLASSICAL PCA

Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

CLASSICAL PCA

Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

Principal Component Analysis
via singular value decomposition:

Stable, efficient computation

Optimal estimate of under
iid

Gaussian noise

Fundamental statistical tool, huge impact in vision, search,

bioinformatics

CLASSICAL PCA

Fitting degenerate data

If degenerate observations are stacked as columns of a matrix

then

But…
PCA breaks down under even a single corrupted observation.

Principal Component Analysis via singular value decomposition:

Stable, efficient computation

Optimal estimate of under
iid

Gaussian noise

Fundamental statistical tool, huge impact in vision, search,

bioinformatics

ROBUST PCA

Problem formulation

D
-

observation

A

low
-
rank

E

sparse error

Properties of the errors:

Each multivariate data sample (column) may be corrupted in some entries

Corruption can be arbitrarily large in magnitude (not Gaussian!)

Problem
: Given recover .

Low
-
rank structure

Sparse errors

ROBUST PCA

Problem formulation

Problem
: Given recover .

Low
-
rank structure

Sparse errors

Numerous heuristic methods in the literature:

Random sampling [
Fischler

and
Bolles

‘81]

Multivariate trimming [

and Kettering ‘72]

Alternating minimization [
Ke

and

‘03]

Influence functions [de la Torre and Black ‘03]

No polynomial
-
time algorithm with strong performance guarantees!

D
-

observation

A

low
-
rank

E

sparse error

ROBUST PCA

Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

ROBUST PCA

Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

ROBUST PCA

Semidefinite

programming formulation

Seek the lowest
-
rank that agrees with the data up to some sparse error:

Not directly tractable, relax:

Semidefinite

program, solvable in polynomial time

Convex envelope over

MATRIX COMPLETION

Motivation for the nuclear norm

Related problem:
we observe only a small
known

subset

of entries of a rank
-

matrix . Can we exactly recover ?

MATRIX COMPLETION

Motivation for the nuclear norm

Related problem:
recover a rank matrix from a
known

subset of entries

Convex optimization heuristic
[
Candes

and
Recht
]
:

Spectral trimming also succeeds with for

For
incoherent

,
exact recovery
with

[
Keshavan
,
Montanari

and Oh
]

[
Candes

and Tao]

ROBUST PCA

Exact recovery?

CONJECTURE
:

If with sufficiently low
-
rank and

exactly recovers .

Sparsity of error

sufficiently sparse, then solving

Empirical evidence
: probability of correct recovery
vs

rank and sparsity

Perfect recovery

Rank

Decompose as or ?

ROBUST PCA

Which matrices and which errors?

Fundamental ambiguity

very sparse matrices are also low
-
rank:

rank
-
1

rank
-
0

0
-
sparse

1
-
sparse

Obviously we can only hope to uniquely recover that are

incoherent

with the standard basis.

Can we recover almost all low
-
rank matrices from almost all sparse errors?

ROBUST PCA

Which matrices and which errors?

Random orthogonal model
(of rank r)
[
Candes

&
Recht

‘08]
:

independent samples from invariant measure

on
Steifel

manifold of
orthobases

of rank r.

arbitrary.

ROBUST PCA

Which matrices and which errors?

Random orthogonal model
(of rank r)
[
Candes

&
Recht

‘08]
:

independent samples from invariant measure

on
Steifel

manifold of
orthobases

of rank r.

arbitrary.

Bernoulli error signs
-
and
-
support
(with parameter ):

Magnitude of is arbitrary.

MAIN RESULT

Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”

BONUS RESULT

Matrix completion in proportional growth

“Convex optimization exactly recovers matrices of rank , even with

entries missing!”

MATRIX COMPLETION

Contrast with literature

[
Candes

and Tao 2009]:

Correct completion
whp

for

Does not apply to the large
-
rank case

This work:

Correct completion
whp

for even with

Proof exploits rich regularity and independence in random orthogonal model.

Caveats:

-

[C
-
T ‘09] tighter for small r.

-

[C
-
T ‘09] generalizes better to other matrix ensembles.

MAIN RESULT

Exact Solution of Robust PCA

“Convex optimization recovers almost any matrix of rank from
errors affecting of the observations!”

ROBUST PCA

Solving the convex program

Semidefinite

program in
millions of unknowns.

Scalable solution
: apply a first
-
order method with convergence to

[
Nesterov
, Beck &
Teboulle
]
:

Solved via
soft
thresholding

(
E
), and
singular value
thresholding

(
A
).

ROBUST PCA

Solving the convex program

Iteration complexity for suboptimal solution.

Dramatic practical gains from continuation

SIMULATION

Recovery in various growth scenarios

Correct recovery with and fixed, increasing.

Empirically, almost constant number of iterations:

Provably robust PCA at only a constant factor more computation
than conventional PCA.

SIMULATION

Phase Transition in Rank and
Sparsity

Fraction of successes with , varying
(10 trials)

Fraction of successes with , varying
(65 trials)

[0,.5] x [0,.5]

[0,1] x [0,1]

[0,1] x [0,1]

[0,.4] x [0,.4]

EXAMPLE

Background modeling from video

Video

Low
-
rank
appx
.

Sparse error

Static camera

surveillance video

200 frames,

72 x 88 pixels,

Significant foreground

motion

EXAMPLE

Background modeling from video

Video

Low
-
rank
appx
.

Sparse error

Static camera

surveillance video

550 frames,

64 x 80 pixels,

significant illumination

variation

Background

variation

Anomalous activity

EXAMPLE

Faces under varying illumination

RPCA

29 images of one
person under varying
lighting:

EXAMPLE

Faces under varying illumination

RPCA

29 images of one
person under varying
lighting:

Self
-

Specularity

EXAMPLE

Face tracking and alignment

Initial alignment
, inappropriate for recognition:

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

EXAMPLE

Face tracking and alignment

Final result
: per
-
pixel alignment

EXAMPLE

Face tracking and alignment

Final result
: per
-
pixel alignment

SIMULATION

Phase Transition in Rank and
Sparsity

Fraction of successes with , varying
(10 trials)

Fraction of successes with , varying
(65 trials)

[0,.5] x [0,.5]

[0,1] x [0,1]

[0,1] x [0,1]

[0,.4] x [0,.4]

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

Hypothesized breakdown behavior as m

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

What we know so far:

This work

Classical PCA

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE I:
convex programming succeeds in proportional growth

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE II:
for small ranks ,

any fraction of errors can eventually be corrected.

Similar to
Dense Error Correction via L1 Minimization
, Wright and Ma ‘08

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE III:
for any rank fraction, ,

there exists a nonzero fraction of errors that can eventually be

corrected with high probability.

CONJECTURES

Phase Transition in Rank and
Sparsity

1

1

0

0

CONJECTURE IV:
there is an
asymptotically sharp phase transition

between correct recovery with overwhelming probability, and

failure with overwhelming probability.

CONJECTURES

Connections to Matrix Completion

Our results also suggest the possibility of a proportional growth
phase transition for
matrix completion
.

1

1

0

0

How do the two breakdown points compare?

How much is gained by knowing the location of the corruption?

Robust PCA

Matrix Completion

Similar to
Recht
,
Xu

and
Hassibi

‘08

Matrix Completion

Robust PCA

FUTURE WORK

Stronger results on RPCA?

RPCA with noise and errors
:

Tradeoff between estimation error and robustness to corruption?

Deterministic conditions on the matrix

Simultaneous error correction and matrix completion:

bounded noise

(e.g., Gaussian)

Conjecture: stable recovery with

we observe

Faster algorithms:

Smarter continuation strategies

Parallel implementations, GPU, multi
-
machine

Further applications:

Computer vision: photometric stereo, tracking, video repair

Relevancy data: search, ranking and collaborative filtering

Bioinformatics

System Identification

FUTURE WORK

Algorithms and Applications

Reference:

Robust Principal Component Analysis:

Exact Recovery of Corrupted Low
-
Rank Matrices by Convex Optimization

submitted to the Journal of the ACM

Collaborators:

Prof
. Yi Ma
(UIUC, MSRA)

Dr.
Zhouchen

Lin
(MSRA)

Dr.
Shankar
Rao

(UIUC)

Arvind

Ganesh

(UIUC)

Yigang

Peng

(MSRA)

Funding:

Microsoft Research Fellowship (sponsored by Live Labs)

Grants
NSF CRS
-
EHS
-
0509151, NSF CCF
-
TF
-
0514955,
ONR
YIP N00014
-
04
-
1
-
0633, NSF IIS 07
-
03756

REFERENCES + ACKNOWLEDGEMENT