Neural Networks:
Backpropagation
algorithm
Data Mining and Semantic Web
University
of Belgrade
School
of Electrical Engineering
Chair of Computer Engineering and
Information Theory
Miroslav
Ti
šma
tisma.etf
@gmail.com
You see this:
But the camera sees this:
What is this?
23.12.2011.
Miroslav Tišma
2/21
Computer Vision: Car detection
Testing
:
What is this?
Not a car
Cars
23.12.2011.
Miroslav Tišma
3/21
pixel 1
pixel 2
Raw image
Cars
“Non”
-
Cars
50 x 50 pixel images
→ 2500 pixels
(7500 if RGB)
pixel 1 intensity
pixel 2 intensity
pixel 2500 intensity
Quadratic features ( ): ≈3 million
features
Learning
Algorithm
pixel 1
pixel 2
23.12.2011.
Miroslav Tišma
4/21
Neural Networks
•
Origins: Algorithms that try to mimic the brain
•
Was very widely used in 80s and early 90s;
popularity diminished in late 90s.
•
Recent resurgence: State
-
of
-
the
-
art technique
for many applications
23.12.2011.
Miroslav Tišma
5/21
Neurons in the brain
Dendr
(I)
tes
Ax
(O)
n
23.12.2011.
Miroslav Tišma
6/21
Neuron model: Logistic unit
Sigmoid (logistic) activation function.
ℎ
Θ
𝑥
=
1
1
+
𝑒
−
Θ
𝑇
𝑥
𝑔
𝑧
=
1
1
+
𝑒
−
𝑧
𝑥
0
=
1
“bias unit”
“output”
“input wires”
“weights”
-
parameters
23.12.2011.
Miroslav Tišma
7/21
Neural Network
Layer 3
Layer 1
Layer 2
“bias unit”
“output layer”
“hidden layer”
“input layer”
23.12.2011.
Miroslav Tišma
8/21
Neural Network
“activation” of unit in layer
m
atrix of weights controlling
function mapping from layer to
layer
If network has units in layer , units in layer , then
will be of dimension .
23.12.2011.
Miroslav Tišma
9/21
Simple example: AND
0
0
0
1
1
0
1
1
-
30
+20
+20
ℎ
Θ
𝑥
=
𝑔
(
−
30
+
20
𝑥
1
+
20
𝑥
2
)
𝑔
(
−
30
)
≈
0
𝑔
(
−
10
)
≈
0
𝑔
(
−
10
)
≈
0
𝑔
(
10
)
≈
1
ℎ
Θ
𝑥
≈
𝑥
1
𝐴 𝐷
𝑥
2
23.12.2011.
Miroslav Tišma
10/21
Example: OR function
0
0
0
1
1
0
1
1
-
10
+20
+20
ℎ
Θ
𝑥
=
𝑔
(
−
10
+
20
𝑥
1
+
20
𝑥
2
)
𝑔
(
−
10
)
≈
0
𝑔
(
10
)
≈
1
𝑔
(
10
)
≈
1
𝑔
(
30
)
≈
1
ℎ
Θ
𝑥
≈
𝑥
1
𝑅
𝑥
2
23.12.2011.
Miroslav Tišma
11/21
Multiple output units: One
-
vs
-
all.
Pedestrian
Car
Motorcycle
Truck
Want ,
w
hen pedestrian
23.12.2011.
Miroslav Tišma
12/21
when car
when motorcycle
, etc.
,
Neural Network (Classification)
Binary classification
1 output unit
Layer 1
Layer 2
Layer 3
Layer 4
Multi
-
class classification
(K classes)
K output units
t
otal no. of layers in network
no. of units (not counting bias unit) in
layer
pedestrian car motorcycle truck
E.g. , , ,
23.12.2011.
Miroslav Tišma
13/21
Cost function
Logistic regression:
23.12.2011.
Miroslav Tišma
14/21
Neural network:
Gradient computation
Need code to compute:
-
-
23.12.2011.
Miroslav Tišma
15/21
Our goal is to minimize
the
cost function
Given one training example ( , ):
Forward propagation:
Layer 1
Layer 2
Layer 3
Layer 4
𝑎
1
𝑎
2
𝑎
3
𝑎
4
23.12.2011.
Miroslav Tišma
16/21
Backpropagation
algorithm
Backpropagation
algorithm
Intuition: “error” of node in layer .
Layer 1
Layer 2
Layer 3
Layer 4
For each output unit (layer L = 4)
𝛿
4
𝛿
3
𝛿
2
ℎ
𝜃
𝑥
the derivate of activation function
can be written as
𝑎
(
𝑙
)
.
∗
(
1
−
𝑎
𝑙
)
𝜕
𝜕
Θ
𝐽
𝜃
=
𝑎
(
𝑙
)
𝛿
(
𝑙
+
1
)
element
-
wise multiplication operator
23.12.2011.
Miroslav Tišma
17/21
Backpropagation algorithm
Training set
Set (for all ).
For
Set
Perform forward propagation to compute for
Using , compute
Compute
u
sed to compute
𝜕
𝜕
Θ
𝐽
𝜃
23.12.2011.
Miroslav Tišma
18/21
Advantages
:
-
Relatively simple implementation
-
Standard method and generally wokrs well
-
Many practical applications:
* handwriting recognition, autonomous driving car
Disadvantages
:
-
Slow and inefficient
-
Can get stuck in local minima resulting in sub
-
optimal solutions
23.12.2011.
Miroslav Tišma
19/21
Literature
:
-
http://
en.wikipedia.org/wiki/Backpropagation
-
http://
www.ml
-
class.org
-
http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
23.12.2011.
Miroslav Tišma
20/21
23.12.2011.
Miroslav Tišma
21/21
Thank you for your attention!
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Σχόλια 0
Συνδεθείτε για να κοινοποιήσετε σχόλιο