1
Sparse Coding and Its Extensions for
Visual Recognition
Kai Yu
M
edia Analytics Department
NEC Labs America, C
upertino, CA
V
isual Recognition is HOT in Computer Vision
10/15/2013
2
C
altech 101
PASCAL VOC
80 Million Tiny
Images
I
mageNet
T
he pipeline of machine visual perception
10/15/2013
3
Low

level
sensing
Pre

processing
Feature
extract.
Feature
selection
Inference:
prediction,
recognition
•
M
ost c
ritical for accuracy
•
A
ccount for most of the
computation
•
Most time

consuming
in
development
cycle
•
O
ften hand

craft in practice
M
ost Efforts in
Machine Learning
Computer vision features
SIFT
Spin image
HoG
RIFT
S
lide Credit: Andrew Ng
GLOH
L
earning everything from data
10/15/2013
5
Low

level
sensing
Pre

processing
Feature
extract.
Feature
selection
Inference:
prediction,
recognition
M
achine
L
earning
Machine Learning
BoW + SPM Kernel
10/15/2013
6
•
Combining multiple features, this method had been the state

of

the

art
on Caltech

101, PASCAL, 15 Scene Categories, …
Figure credit: Fei

Fei Li, Svetlana Lazebnik
Bag

of

visual

words
representation (BoW)
based on vector quantization (VQ)
Spatial pyramid matching (SPM) kernel
W
inning Method in PASCAL VOC before 2009
10/15/2013
M
ultiple Feature
Sampling Methods
Multiple
Visual
Descriptors
VQ C
oding
,
H
istogram,
SPM
N
onlinear SVM
7
Convolution Neural Networks
8
•
T
he architectures of some successful methods are
not so much different from CNNs
Conv. Filtering Pooling Conv. Filtering Pooling
BoW+SPM: the same architecture
9
e.g, SIFT, HOG
VQ Coding
Average Pooling
(obtain histogram)
Nonlinear
SVM
Local Gradients
Pooling
Observations:
•
Nonlinear SVM is not scalable
•
VQ coding may be too coarse
•
Average pooling is not optimal
•
Why not learn the whole thing?
D
evelop better methods
10
Better Coding
Better Pooling
Scalable
Linear
Classifier
B
etter Coding
B
etter
Pooling
S
parse Coding
10/15/2013
11
Sparse coding (Olshausen & Field,1996). Originally
developed to explain early visual processing in the brain
(edge detection).
T
raining: given a set of random patches x, learning a
dictionary of bases [Φ
1,
Φ
2,
…]
Coding: for data vector x, solve LASSO to find the
sparse coefficient vector a
Sparse Coding E
xample
Natural Images
Learned bases (
f
1 , …,
f
64
): “Edges”
0.8 * + 0.3 * + 0.5 *
x
0.8 *
f
36
+ 0.3 *
f
42
+ 0.5 *
f
63
[a
1
, …, a
64
] =
[
0, 0, …, 0,
0.8
,
0, …, 0,
0.3
,
0, …, 0,
0.5
, 0
]
(feature representation)
Test example
Compact & easily interpretable
Slide credit: Andrew Ng
Testing:
What is this?
Motorcycles
Not motorcycles
Unlabeled images
…
[Raina, Lee, Battle, Packer & Ng, ICML 07]
S
elf

taught Learning
Testing:
What is this
?
Slide credit: Andrew Ng
Classification
R
esult on Caltech 101
10/15/2013
14
64%
SIFT VQ + N
onlinear
SVM
50%
Pixel
S
parse Coding
+ Linear SVM
9K
images, 101 classes
15
S
parse
Coding
M
ax
Pooling
Scalable
Linear
Classifier
Local Gradients
Pooling
e.g, SIFT, HOG
S
parse Coding on SIFT
[Y
ang, Yu, Gong & Huang
, CVPR09]
10/15/2013
16
64%
SIFT VQ + N
onlinear
SVM
73%
SIFT
S
parse Coding +
Linear SVM
C
altech

101
S
parse Coding on SIFT
[Y
ang, Yu, Gong & Huang
, CVPR09]
W
hat we have learned?
17
S
parse
Coding
M
ax
Pooling
Scalable
Linear
Classifier
Local Gradients
Pooling
1.
S
parse coding is a useful stuff (why?)
2.
H
ierarchical architecture is needed
e.g, SIFT, HOG
MNIST E
xperiments
10/15/2013
18
Error: 4.54%
•
When SC achieves the best classification accuracy, the
learned bases are like digits
–
each basis has a clear local
class association.
Error: 3.75%
Error: 2.64%
Distribution of coefficient (SIFT, Caltech101)
10/15/2013
19
Neighbor bases tend to
get nonzero coefficients
10/15/2013
20
I
nterpretation
2
Geometr
y
of data manifold
•
Each basis an “
anchor point
”
•
Sparsity
is induced by locality
:
each datum is a linear
combination of neighbor
anchors.
I
nterpretation
1
Discover
subspaces
•
Each basis is a “
direction
”
•
Sparsity
: each datum is a
linear combination of
only
several bases.
•
Related to topic model
A F
unction Approximation View to Coding
10/15/2013
21
•
S
etting
:
f(x) is a nonlinear
feature extraction function
on image patches x
•
Coding
: nonlinear mapping
x
a
typically, a is high

dim &
sparse
•
Nonlinear Learning
:
f(x) = <w, a>
A
coding scheme is good if it helps learning f(x)
10/15/2013
22
A F
unction Approximation View to Coding
–
The General Formulation
F
unction
Approx.
Error
≤
A
n unsupervised
learning objective
Local Coordinate Coding (LCC)
10/15/2013
23
•
D
ictionary Learning: k

means (or hierarchical
k

means)
•
C
oding for x, to obtain its sparse representation
a
Step 1
–
ensure locality
: find the K nearest bases
Step 2
–
ensure low coding error
:
Yu, Zhang & Gong, NIPS 09
W
ang, Yang, Yu, Lv, Huang CVPR 10
Super

Vector Coding (SVC)
10/15/2013
24
•
D
ictionary Learning: k

means (or hierarchical
k

means)
•
C
oding for x, to obtain its sparse representation
a
Step 1
–
find the nearest bas
i
s of x, obtain its VQ
coding
e.g. [0, 0, 1, 0, …]
Step 2
–
form super vector coding:
e.g. [0, 0, 1, 0, …, 0, 0, (x

m
3
），
0
，
…]
Zhou, Yu, Zhang, and Huang, ECCV 10
Zero

order
Local tangent
F
unction Approximation based on
LCC
10/15/2013
25
data points
bases
locally linear
Yu, Zhang, Gong, NIPS 10
Function Approximation based on SVC
data points
cluster centers
Piecewise local linear (
first

order)
Local tangent
Zhou, Yu, Zhang, and Huang, ECCV 10
PASCAL VOC C
hallenge
2009
10/15/2013
27
Ours
Best of
Other Teams
Difference
Classes
N
o.1 for 18 of 20
categories
W
e used only HOG
feature on gray images
I
mageNet Challenge 2010
10/15/2013
28
~40%
VQ + I
ntersection Kernel
64%~73%
Various Coding
Methods + Linear SVM
1.4
million images, 1000 classes,
top5 hit rate
50%
Classification accuracy
H
ierarchical sparse coding
29
Conv. Filtering Pooling Conv. Filtering Pooling
L
earning from
unlabeled data
Yu, Lin, & Lafferty, CVPR 11
A two

layer sparse coding formulation
10/15/2013
30
MNIST Results

classification
HSC vs. CNN:
HSC provide even better performance than CNN
more amazingly, HSC learns features in
unsupervised
manner!
31
MNIST results

effect of hierarchical learning
C
omparing the Fisher score of HSC and SC
Discriminative power:
is significantly improved by HSC although HSC is
unsupervised coding
32
MNIST results

learned codebook
33
One dimension in the second layer: invariance to
translation, rotation, and deformation
Caltech101 results

classification
Learned descriptor:
performs slightly better than SIFT + SC
34
Conclusion and Future Work
“
function approximation
” view to derive novel sparse coding
methods.
Locality
–
one way to achieve sparsity and it’s really useful. But we
need deeper understanding of the feature learning methods
Interesting directions
–
Hierarchical coding
–
Deep Learning (many papers now!)
–
Faster methods for sparse coding (e.g. from LeCun’s group)
–
Learning features from a richer structure of data, e.g., video
(learning invariance to out plane rotation)
References
10/15/2013
37
•
L
earning Image Representations from Pixel Level via Hierarchical Sparse Coding,
K
ai Yu,
Yuanqing
Lin, John Lafferty.
CVPR 2011
•
Large

scale Image Classification: Fast Feature Extraction and SVM Training,
Y
uanqing
Lin,
Fengjun
Lv
,
Liangliang
Cao,
Shenghuo
Zhu, Ming Yang,
Timothee
Cour
, Thomas Huang, Kai Yu
in
CVPR 2011
•
ECCV 2010 Tutorial, Kai Yu, Andrew Ng (with links to some source codes)
•
Deep Coding Networks,
Yuanqing
Lin, Tong Zhang,
Shenghuo
Zhu, Kai Yu. In
NIPS 2010
.
•
Image Classification using Super

Vector Coding of Local Image Descriptors,
Xi Zhou, Kai Yu, Tong Zhang, and Thomas Huang. In
ECCV 2010
.
•
Efficient Highly Over

Complete Sparse Coding using a Mixture Model,
Jianchao
Yang, Kai Yu, and Thomas Huang. In
ECCV 2010
.
•
Improved Local Coordinate Coding using Local Tangents,
Kai Yu and Tong Zhang. In
ICML 2010
.
•
Supervised translation

invariant sparse coding,
Jianchao
Yang, Kai Yu, and Thomas Huang, In
CVPR 2010
•
Learning locality

constrained linear coding for image classification,
Jingjun
Wang,
Jianchao
Yang, Kai Yu,
Fengjun
Lv
, Thomas Huang. In
CVPR 2010
.
•
Nonlinear learning using local coordinate coding,
Kai Yu, Tong Zhang, and
Yihong
Gong. In
NIPS 2009
.
•
Linear spatial pyramid matching using sparse coding for image classification,
Jianchao
Yang, Kai Yu,
Yihong
Gong, and Thomas Huang. In
CVPR 2009
.
Comments 0
Log in to post a comment