Neural activ ation functions
In a neural net w ork eac h neuron has an activation function whic h sp ecies the output of a
neuron to a giv en input Neurons are switc hes that output a when they are sucien tly
activ ated and a when not One of the activ ation functions commonly used for neurons is the
sigmoid function
f
I R I R f x
e
x
This function lo oks lik e an S hence its name
4
2
2
4
0.2
0.4
0.6
0.8
1
When one uses the neural net w ork to learn the up dating of the con tributions dep ends on
the steepness slop e of the activ ation function Ho w and wh y is explained in the lectures on
neural net w orks with the delta rule and b ackpr op agation and w e cannot treat that here But
mathematically the steepness of the activ ation function is giv en b y the derivative and w e can
compute with that T o sa v e computation the neural net w ork engineer attempts to express this
deriv ativ e in terms of the function v alue itself so to express f
x in terms of f x
V erify that the gure indeed represen ts the sigmoid function
c hec k what happ ens when
x x and at x
F rom the gure it seems that the function has a certain symmetry around the p oin t
Ho w w ould y ou express that symmetry mathematically Can y ou pro v e that it holds
Answ er f x f f f x so f x f x f
Compute the deriv ativ e f
x
The form ula for the deriv ativ e f
x can b e expressed as a function of f x Do so
Answ er f
x f x
f x
The previous answ er is a dieren tial equation its solution giv e the complete family of
functions with this prop ert y Giv e the family so solv e the dieren tial equation Ho w do
the mem b ers dier Answ er y
ce
x
with c and also y and y
Apart from
those t w o the mem b ers di er in steepness
No w a neural net w ork engineer w an ts to use the nal neuron to c hange the direction of a motor
T o do so she prefers its output to b e b et w een and She decides to use a new activ ation
function g deriv ed simply from the old as
g x f x
Again she w an ts to sa v e computation b y expressing the deriv ativ e in terms of the function v alue
so to express g
x in terms of g x But she needs to b e careful
Chec k that this is indeed a go o d c hoice dra w this new function
F rom the denition of g y ou should exp ect g
to b e just t wice f
V erify this b y computing
the deriv ativ e
Express g
x in terms of g x Answ er g
x
g x
Note that the deriv ativ e f
x in terms of x b ecomes t wice as big but that the form ula for
the deriv ativ e in terms of g x c hanges more than just b y a factor of
it is
g x
rather than g x g x
Moral
b e careful with deriv ativ es reexpressed in terms of the function v alues
Enter the password to open this PDF file:
File name:

File size:

Title:

Author:

Subject:

Keywords:

Creation Date:

Modification Date:

Creator:

PDF Producer:

PDF Version:

Page Count:

Preparing document for printing…
0%
Comments 0
Log in to post a comment