Neural activ ation functions

In a neural net w ork eac h neuron has an activation function whic h sp ecies the output of a

neuron to a giv en input Neurons are switc hes that output a when they are sucien tly

activ ated and a when not One of the activ ation functions commonly used for neurons is the

sigmoid function

f

I R I R f x

e

x

This function lo oks lik e an S hence its name

-4

-2

2

4

0.2

0.4

0.6

0.8

1

When one uses the neural net w ork to learn the up dating of the con tributions dep ends on

the steepness slop e of the activ ation function Ho w and wh y is explained in the lectures on

neural net w orks with the delta rule and b ackpr op agation and w e cannot treat that here But

mathematically the steepness of the activ ation function is giv en b y the derivative and w e can

compute with that T o sa v e computation the neural net w ork engineer attempts to express this

deriv ativ e in terms of the function v alue itself so to express f

x in terms of f x

V erify that the gure indeed represen ts the sigmoid function

c hec k what happ ens when

x x and at x

F rom the gure it seems that the function has a certain symmetry around the p oin t

Ho w w ould y ou express that symmetry mathematically Can y ou pro v e that it holds

Answ er f x f f f x so f x f x f

Compute the deriv ativ e f

x

The form ula for the deriv ativ e f

x can b e expressed as a function of f x Do so

Answ er f

x f x

f x

The previous answ er is a dieren tial equation its solution giv e the complete family of

functions with this prop ert y Giv e the family so solv e the dieren tial equation Ho w do

the mem b ers dier Answ er y

ce

x

with c and also y and y

Apart from

those t w o the mem b ers di er in steepness

No w a neural net w ork engineer w an ts to use the nal neuron to c hange the direction of a motor

T o do so she prefers its output to b e b et w een and She decides to use a new activ ation

function g deriv ed simply from the old as

g x f x

Again she w an ts to sa v e computation b y expressing the deriv ativ e in terms of the function v alue

so to express g

x in terms of g x But she needs to b e careful

Chec k that this is indeed a go o d c hoice dra w this new function

F rom the denition of g y ou should exp ect g

to b e just t wice f

V erify this b y computing

the deriv ativ e

Express g

x in terms of g x Answ er g

x

g x

Note that the deriv ativ e f

x in terms of x b ecomes t wice as big but that the form ula for

the deriv ativ e in terms of g x c hanges more than just b y a factor of

it is

g x

rather than g x g x

Moral

b e careful with deriv ativ es reexpressed in terms of the function v alues

## Comments 0

Log in to post a comment