Restricted Boltzmann Machines and Deep Networks for Unsupervised Learning

builderanthologyΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 3 χρόνια και 8 μήνες)

272 εμφανίσεις

Restricted Boltzmann Machines and

Deep Networks for Unsupervised Learning

Instituto Italiano di Tecnologia, Genova


June 7th, 2011

Loris Bazzani

University of Verona

Brief Intro


Unsupervised Learning



Learning features from
(
visual
) data



Focus here on Restricted
Boltzmann Machines

2

Outline Presentation


Restricted Boltzmann Machines (RBMs)


Binary RBMs


Gaussian
-
binary RBMs


RBMs for Classification


Deep Belief Networks (DBNs)


Learning Algorithms

Applications

Theor
y



RBMs for Modeling Natural Scenes
[Ranzato, CVPR 2010]



Learning Attentional Policies
[Bazzani, ICML 2011]

Outline Presentation


Restricted Boltzmann Machines (RBMs)


Binary RBMs


Gaussian
-
binary RBMs


RBMs for Classification


Deep Belief Networks (DBNs)


Learning Algorithms

Applications

Theor
y



RBMs for Modeling Natural Scenes
[Ranzato, CVPR 2010]



Learning Attentional Policies
[Bazzani, ICML 2011]

Restricted Boltzmann Machines

5


Bipartite Probabilistic Graphical Model




W
: parameters governing the interactions
between
visible
and
hidden units


Property:
”given the hidden units, all of the visible units
become independent and given the visible units, all of the
hidden units become independent”



Binary RBMs




We can sample from





Use the expected value of the hidden units as
features:



6

Gaussian
-
binary RBMs



Popular extension for modeling natural images


Make the visible units conditionally Gaussian
given the hidden units




Conditional distributions

7

RBMs for Classification

1)
Feed the hidden representation into a
standard classifier (e.g., multinomial logistic
regression, SVM, random forest,…)

2)
Embed the class into the visible units



and, the class vector will be sample from


8

Deep Belief Networks


Goal
: reach a high level of abstraction,
so that classification becomes simple
(
e.g.
, linear)


Multiple stacked RBMs


Learning consists in greedy training each
level sequentially from the bottom


Add fine
-
tuning with back
-
propagation


Or a non
-
linear classifier can be used


PB
: how many layers?

9

Outline Presentation


Restricted Boltzmann Machines (RBMs)


Binary RBMs


Gaussian
-
binary RBMs


RBMs for Classification


Deep Belief Networks (DBNs)


Learning Algorithms

Applications

Theor
y



RBMs for Modeling Natural Scenes
[Ranzato, CVPR 2010]



Learning Attentional Policies
[Bazzani, ICML 2011]


Maximum Likelihood (ML) techniques


No close
-
form solution for the maximization


Problem
: Partition function usually not
efficiently computable


Solutions
:


Approximate ML


Sacrifices convergence properties to make it
computationally feasible


Alternatives
:
variational methods, max
-
margin learning,
etc.



How to Learn the Parameters

11

ML Problem

12



Gradient:


Match the gradient of the free energy under the data
distribution with the gradient under the model distribution

Marginalizing

Contrastive Divergence (1)


It is just a gradient descent


At each step, it contrasts the data distribution
with the model distribution




E.g.,

binary RBM

13

Contrastive Divergence (2)


Algorithm for binary RBMs:

14

Outline Presentation


Restricted Boltzmann Machines (RBMs)


Binary RBMs


Gaussian
-
binary RBMs


RBMs for Classification


Deep Belief Networks (DBNs)


Learning Algorithms

Applications

Theor
y



RBMs for Modeling Natural Scenes
[Ranzato, CVPR 2010]



Learning Attentional Policies
[Bazzani, ICML 2011]

Modeling Natural Images


Motivations:


Learning a generative model of natural images


Extracting features that capture regularities


Opposed to using engineered features


RBM with two set of hidden units:


One represents the pixel intensity


Another one, the pair
-
wise dependencies


Called Mean
-
Covariance RBM (mc
-
RBM)


It is still a Gaussian
-
binary RBM


17

mc
-
RBM Model (1)


Capture pair
-
wire interactions with:




Sketch:


18

Covariance

hiddens

mc
-
RBM Model (2)


Representation of mean pixel intensities:




Conditional distributions:




19

mc
-
RBM Model (3)


Final Energy term:




Free Energy formulation is also computable


Learning with


Stochastic gradient descent


And, Contrastive Divergence


Sampling using Hybrid Monte Carlo

20

Regularization

Training Protocol for Recognition


Images are pre
-
processed by PCA whitening


Train the mc
-
RBM


Extract features with mc
-
RBM


Train a classifier for object recognition:


Multinomial Logistic Classifier

21

Object Recognition on CIFAR 10

22

Outline Presentation


Restricted Boltzmann Machines (RBMs)


Binary RBMs


Gaussian
-
binary RBMs


RBMs for Classification


Deep Belief Networks (DBNs)


Learning Algorithms

Applications

Theor
y



RBMs for Modeling Natural Scenes
[Ranzato, CVPR 2010]



Learning Attentional Policies
[Bazzani, ICML 2011]

Where do you look at?

Original video source:
http://gpu4vision.icg.tugraz.at/index.php?content=subsites/prost/prost.php


Goal


Human tracking and recognition is amazingly
efficient and effective


Large stream of data is filtered by attention


We propose a model for tracking and recognition
that takes inspiration from human visual system


Tracking and recognition of “something” that is
moving in the scene


Accumulate gaze data


Plan where to look at in the next future

25

Parallelism with Human Brain

26

Source image:
http://www.waece.org/cd_morelia2006/ponencias/stoodley.htm


Sketch of the Model

27

(mc
-
)RBM

Multi
-
fixation RBM

Classifier

Policy Learning

Learning


Offline Training


Extract gaze data from a training dataset


Train the (mc
-
)RBM


Train the multi
-
fixation RBM (3 random gazes)


Train the multinomial logistic classifier

28


Online Learning


Hedge algorithm for policy learning


Online


from moving “things” with multiple
saccades

Modularity

autoencoders, sparse coding
, etc.

SVM, random forest,
etc.

other bandit techniques or Bayesian optimization


10 synthetic video sequences with moving and
background digits (from MNIST dataset)

Experiments (1)

Tracking error in pixels

Classification accuracy

Code available at:
http://www.lorisbazzani.info/code
-
datasets/rbm
-
tracking/


Experiments (2)

Dataset available at:
http://seqam.rutgers.edu/softdata/facedata/facedata.html


Summary


Several RBMs models


How to train RBMs


Their extensions for classification


RBMs as block for deep architectures


They are useful for learning features from
images, without engineering them


Taking inspiration from human learning, DBNs
have been used

32

References (1)

Learning

attentional

policies

for

tracking

and

recognition

in

video

with

deep

networks
,

Loris

Bazzani,

Nando

de

Freitas,

Hugo

Larochelle,

Vittorio

Murino,

and

Jo
-
Anne

Ting,

International

Conference

on

Machine

Learning,

2011

Tutorial

on

Stochastic

Approximation

Algorithms

for

Training

Restricted

Boltzmann

Machines

and

Deep

Belief

Nets
,

Swersky

and

Bo

Chen,

Benjamin

Marlin,

and

Nando

de

Freitas,

Information

Theory

and

Applications

(ITA)

Workshop,

2010

Inductive

Principles

for

Restricted

Boltzmann

Machine

Learning
,

Benjamin

Marlin,

Kevin

Swersky,

Bo

Chen,

and

Nando

de

Freitas,

AISTATS,

2010

Modeling

Pixel

Means

and

Covariances

Using

Factorized

Third
-
Order

Boltzmann
,

Marc'Aurelio

Ranzato

and

Geoffrey

E
.

Hinton,

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition,

2010

Factored

3
-
Way

Restricted

Boltzmann

Machines

For

Modeling

Natural

Images
,

Marc'Aurelio

Ranzato,

Alex

Krizhevsky

and

Geoffrey

E
.

Hinton,

International

Conference

on

Artificial

Intelligence

and

Statistics,

2010

On

Deep

Generative

Models

with

Applications

to

Recognition
,

Marc'Aurelio

Ranzato,

Joshua

Susskind,

Volodymyr

Mnih,

and

Geoffrey

Hinton,

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition,

2011


Learning

to

combine

foveal

glimpses

with

a

third
-
order

Boltzmann

machine
,

Hugo

Larochelle

and

Geoffrey

E
.

Hinton,

Neural

Information

Processing

Systems,

2010


33

References (2)

Stacks

of

Convolutional

Restricted

Boltzmann

Machines

for

Shift
-
Invariant

Feature

Learning
,

Mohammad

Norouzi,

Mani

Ranjbar,

and

Greg

Mori,

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition,
2009

Deconvolutional

Networks
,

Matthew

D
.

Zeiler,

Dilip

Krishnan,

Graham

W
.

Taylor,

and

Rob

Fergus,

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition,

2010

Convolutional

deep

belief

networks

for

scalable

unsupervised

learning

of

hierarchical

representation
s,

Lee,

Honglak,

Grosse,

Roger,

Ranganath,

Rajesh

and

Ng,

Andrew,

International

Conference

on

Machine

Learning,

2009

A

deep

learning

approach

to

machine

transliteration
,

Deselaers,

Thomas,

Hasan,

Savsa,

Bender,

Oliver

and

Ney,

Hermann,

Proceedings

of

the

Fourth

Workshop

on

Statistical

Machine

Translation,

2009

Learning

Multilevel

Distributed

Representations

for

High
-
dimensional

Sequences
,

Sutskever,

I
.

and

Hinton,

G
.

E
.
,

Proceeding

of

the

Eleventh

International

Conference

on

Artificial

Intelligence

and

Statistics,

2007

On

Contrastive

Divergence

Learning
,

Miguel

A
.

Carreira
-
Perpinan

and

Geoffrey

E
.

Hinton,

International

Conference

on

Artificial

Intelligence

and

Statistics,

2005

Convolutional

learning

of

spatio
-
temporal

features
,

Taylor,

Graham

W
.
,

Fergus,

Rob,

LeCun,

Yann

and

Bregler,

Christoph,

Proceedings

of

the

11
th

European

conference

on

Computer

vision,

2010

A

Practical

Guide

to

Training

Restricted

Boltzmann

Machines
,

Geoffrey

E
.

Hinton,

University

of

Toronto,

2010
,

TR
2010
-
003



34