Neural activation functions

companyscourgeAI and Robotics

Oct 19, 2013 (3 years and 7 months ago)

106 views

Neural activ ation functions
In a neural net w ork eac h neuron has an activation function whic h sp ecies the output of a
neuron to a giv en input Neurons are switc hes that output a  when they are sucien tly
activ ated and a   when not One of the activ ation functions commonly used for neurons is the
sigmoid function

f
I R ￿ I R  f x

  e
￿ x

This function lo oks lik e an S hence its name

-4
-2
2
4
0.2
0.4
0.6
0.8
1
When one uses the neural net w ork to learn the up dating of the con tributions dep ends on
the steepness slop e of the activ ation function Ho w and wh y is explained in the lectures on
neural net w orks with the delta rule and b ackpr op agation and w e cannot treat that here But
mathematically  the steepness of the activ ation function is giv en b y the derivative  and w e can
compute with that T o sa v e computation the neural net w ork engineer attempts to express this
deriv ativ e in terms of the function v alue itself so to express f
￿
x in terms of f x 
￿￿￿ V erify that the gure indeed represen ts the sigmoid function
c hec k what happ ens when
x ￿ ￿￿  x ￿ ￿  and at x 
￿￿￿ F rom the gure it seems that the function has a certain symmetry around the p oin t 
￿
￿

Ho w w ould y ou express that symmetry mathematically  Can y ou pro v e that it holds
Answ er  f x  ￿ f   f  ￿ f ￿ x  so f x   f ￿ x    f  
￿￿￿ Compute the deriv ativ e f
￿
x 
￿￿￿ The form ula for the deriv ativ e f
￿
x can b e expressed as a function of f x  Do so
Answ er  f
￿
x   f x 
￿ f x  
￿￿￿ The previous answ er is a dieren tial equation its solution giv e the complete family of
functions with this prop ert y  Giv e the family so solv e the dieren tial equation  Ho w do
the mem b ers dier Answ er  y 
￿
￿￿ ce
￿ x
 with c   and also y   and y 
Apart from
those t w o the mem b ers di er in steepness
No w a neural net w ork engineer w an ts to use the nal neuron to c hange the direction of a motor
T o do so she prefers its output to b e b et w een ￿  and  She decides to use a new activ ation
function g deriv ed simply from the old as

g x  f x ￿  
Again she w an ts to sa v e computation b y expressing the deriv ativ e in terms of the function v alue
so to express g
￿
x in terms of g x  But she needs to b e careful
￿￿￿ Chec k that this is indeed a go o d c hoice dra w this new function 

￿￿￿ F rom the denition of g  y ou should exp ect g
￿
to b e just t wice f
￿
 V erify this b y computing
the deriv ativ e
￿￿￿ Express g
￿
x in terms of g x  Answ er  g
￿
x  
￿
￿

￿ g x 
￿
 
￿￿￿ Note that the deriv ativ e f
￿
x in terms of x b ecomes t wice as big but that the form ula for
the deriv ativ e in terms of g x c hanges more than just b y a factor of 
it is
￿
￿
 ￿ g x
￿

rather than  g x  ￿ g x 
Moral
b e careful with deriv ativ es reexpressed in terms of the function v alues