Cellular Neural Networks

overwhelmedblueearthΤεχνίτη Νοημοσύνη και Ρομποτική

1 Δεκ 2013 (πριν από 3 χρόνια και 11 μήνες)

90 εμφανίσεις

Polynomial Discrete Time
Cellular Neural Networks

Eduardo Gomez
-
Ramirez †

Giovanni Egidio Pazienza‡


† LIDETEA, POSGRADO E INVESTIGACION

Universidad La Salle


México, D.F.



Department d’Electronica, EALS

Universitat “Ramon Llull”


Barcelona, Spain

Outline


Cellular Neural Networks (CNN)


Introduction and Objective


Genetic Algorithms (GA)


Polynomial Discrete Time CNNs (PDTCNNs)


XOR Problem


Game of Life


Learning vs Design


Conclusions and future work

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

CNN: Introduction


CNN for complex task (linearly nonseparable
data)



Multilayer CNNs


Include more degrees of freedom for the
output state of each layer


Search in a finite set of templates


Single layer: Polynomial CNNs

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions


Improve the representation power of a single
layer CNN including a simple nonlinear term
to solve problems with linearly nonseparable
data (XOR)

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Objective

CNN: mathematical model

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions


The simplified mathematical model is:

where x
c

is the state of the cell, u
c

the input
and y
c

the output

i
u
b
t
y
a
t
x
dt
dx
d
c
d
d
c
d
c
c





)
(
)
(
)
1
)
(
1
)
(
(
2
1




t
x
t
x
y
c
c
c
CNN: Activation Function

-1.5
-1
-0.5
0
0.5
1
1.5
-3
-2
-1
0
1
2
3
x
y
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

CNN: Block Diagram

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

CNN: Discrete Model

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions


Computing x(∞), the model can be
represented as

i
u
b
n
y
a
n
x
d
c
d
d
c
d
c



)
(
)
(









0
)
(
,
1
0
)
(
,
1
)
(
n
x
n
x
n
y
c
c
c
using the following activation function



Steps:



Crossover
C(Fg)



Mutation
M(*)



Adding random parent
Ag
(

)


GA: main steps proposed

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions








































d
c
b
c
d
a
b
a
F
C
d
c
M
b
a
M
M
M
F
g
g
2
1
2
1
,
,
GA: Crossover

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions












125
.
0
125
.
0
0
0
0
0
0
A











125
.
125
.
0
0
0
0
0
0
0
B

I=0














125
.
0
0
0
0
0
0
0
0
A











125
.
125
.
125
.
0
0
0
0
0
0
B

I=0
























2
1
2
1
2
2
1
1
2
2
2
2
2
1
2
1
2
2
1
1
1
1
1
1
1
c
c
c
c
b
b
b
b
a
a
a
a
c
c
c
c
b
b
b
b
a
a
a
a
P


Individual 1



Individual 2

GA: Crossover

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

a
1

b
1

c
1

a
2

b
2

c
2

GA: Mutation

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions














m
ij
m
ij
m
ij
P
r
F
P
r
F
P
F
M
)
(
)
(
)
,
(


where
r

U(0,1)

is a random variable with
uniform distribution defined on a probability
space (

,

,P),







GA: Mutation (resolution)

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions






















'
2
'
1
'
2
'
1
'
2
'
2
'
1
'
1
'
2
'
2
'
2
'
2
'
2
'
1
'
2
'
1
'
2
'
2
'
1
'
1
'
1
'
1
'
1
'
1
'
1
c
c
c
c
b
b
b
b
a
a
a
a
c
c
c
c
b
b
b
b
a
a
a
a
P











125
.
0
125
.
0
0
0
0
0
0
A











125
.
125
.
0
0
0
0
0
0
0
B

I=0














125
.
0
0
0
0
0
0
0
0
A











125
.
125
.
125
.
0
0
0
0
0
0
B

I=0





Individual 1



Individual 2

a
1

b
1

c
1

a
2

b
2

c
2

Population=sons + sons mutated







g
g
g
C F
A
M C F
 
 

 
 








g
g
n
p
g
g
A
O
n
A
S
p
min
,



GA: Selecting Parents

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions















)
(
1
w
F
F
F
g
g
g
GA: Adding Random Parent

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions


THEOREM 1: (Weierstrass’s Approximation Theorem)



Let g be a continuous real valued function defined on
a closed interval [a,b]. Then, given any


positive,
there exists a polynomial y (which may depend on

)
with real coefficients such that:










For every x


[a,b].

Polynomial Discrete Time
Cellular Neural Network




)
(
)
(
x
y
x
g
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions


THEOREM 2 *:

Any Boolean Function of n
-
variables
can be realized using a Polynomial Threshold gates
of order s

n.


The quadratic threshold gate can be defined:














And s is the number of inputs and T is the
threshold constant.

Polynomial Discrete Time
Cellular Neural Network













otherwise
T
x
x
w
x
w
if
y
n
i
n
i
j
i
ij
i
i
0
1
1
1
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

*
N. J. Nilsson. The Mathematical Foundations of Learning
Machines. McGraw Hill, New York, 1990.

PDTCNN: the model (I)

c
d
d
c
N
d
d
c
d
c
N
d
d
c
d
c
i
y
u
g
u
B
k
y
A
k
x
r
r








)
,
(
)
(
)
(
)
(
)
(











0
)
1
(
1
0
)
1
(
1
))
1
(
(
)
(
k
x
f
k
x
if
k
x
f
k
y
c
c
c
c
















)
0
(
)
0
(
)
0
(
)
,
(
)
(
)
(
)
(
c
d
c
N
d
c
d
c
d
c
N
d
c
d
d
d
c
N
d
c
d
d
d
y
u
P
u
y
P
y
u
P
y
u
g
r
r
r
















)
(
)
(
)
(
)
,
(
)
(
)
(
)
(
k
y
u
P
u
k
y
P
k
y
u
P
y
u
g
c
d
c
N
d
c
d
c
d
c
N
d
c
d
d
d
c
N
d
c
d
d
d
r
r
r
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

PDTCNN: the model (II)

)
(
)
(
)
,
(
)
(
)
(
k
u
Q
k
y
P
y
u
g
d
c
N
d
c
d
d
c
N
d
c
d
d
d
r
r





c
d
d
c
N
d
d
c
d
c
N
d
d
c
d
c
i
y
u
g
u
B
k
y
A
k
x
r
r








)
,
(
)
(
)
(
)
(
)
(
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

PDTCNN: Solving XOR problem

Some papers:


Z. Yang, Y. Nishio, A. Ushida,
Templates and algorithms for two
-
layer
cellular neural networks.
IJCNN’02, 2002.




F. Chen, G. He, G. Chen & X. Xu,
Implementation of Arbitrary Boolean
Functions via CNN.
CNNA’06, 2006.

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

PDTCNN:Solving XOR problem


M. Balsi, Generalized CNN: Potentials of a
CNN with Non
-
Uniform Weights.
CNNA
-
92,
2002

.



E. Bilgili, I. C. Göknar and O. N. Ucan,
Cellular neural network with trapezoidal
activation function.
Int. J. Circ. Theor. Appl
.,
2005

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Learning parameters


Initialpop=20000


Number of fathers=7


Maximum number of random parents to
be add = 3


Kpro=0.8


Increment=1


Mutation Probability=0.15

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

PDTCNN:
First Scheme U:u
ij
=x
ij

x
ij+1












1
4
0
1
1
5
0
1
4
0










i
P
B
A









0
3
0
0
0
1
0
0
3
0






i
P
B
A










0
1
0
0
0
1
0
0
1
0






i
P
B
A

)
(
)
(
)
(
1
k
y
u
u
k
y
k
x
c
c
c
c
c




















)
(
)
(
)
(
)
,
(
)
(
)
(
)
(
k
y
u
P
u
k
y
P
k
y
u
P
y
u
g
c
d
c
N
d
c
d
c
d
c
N
d
c
d
d
d
c
N
d
c
d
d
d
r
r
r
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

PDTCNN:
Second Scheme U:u
ij
=x
ij

y
ij











2
2
8
1
0
0
0
0
0
0





i
P
B
A
b)









1
1
5
1
0
0
0
0
0
1







i
P
B
A
c)









0 1 0 0 0 0
0 1 0 0
A B
P i
  
  
)
(
)
(
)
(
k
y
u
k
y
k
x
c
c
c
c


















)
0
(
)
0
(
)
0
(
)
,
(
)
(
)
(
)
(
c
d
c
N
d
c
d
c
d
c
N
d
c
d
d
d
c
N
d
c
d
d
d
y
u
P
u
y
P
y
u
P
y
u
g
r
r
r
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

The Game of Life (I)


The

Game

of

Life

(GoL)

is

a

totalistic

cellular

automaton

consisting

in

a

two
-
dimensional

grid

cells,

that

may

be

either

alive

(black)

or

dead

(white)
.


Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

The Game of Life (II)


The

state

of

each

cell

varies

according

to

the

following

rules
:



Birth
: a cell that is dead at time t becomes alive at
time t + 1 only if exactly 3 of its neighbors were alive
at time t;



Survival
: a cell that was living at time t will remain
alive at t + 1 if and only if it had exactly 2 or 3 alive
neighbors at time t.

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

The Game of Life (III)



Every

sufficient

well
-
stated

mathematical

problem

can

be

reduced

to

a

question

about

Life
;



It

is

possible

to

make

a

life

computer

(logic

gates,

storage

etc
.
)
;



Life

is

universal
:

it

can

be

programmed

to

perform

any

desired

calculation
;



Given

a

large

enough

Life

space

and

enough

time,

self
-
reproducing

animals

will

emerge
...



The

whole

universe

is

a

CA!

(E
.
Fredkin,

MIT)
.

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

The Game of Life


NOT gate

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

A

CNN & GoL


Multilayer CNN (Chua, Roska)


1990


Activation function (Chua, Roska)


1990


CNN
-
UM (Roska,Chua)
-
1990


CNN Universal Cells (Dogaru, Chua)


1999


Simplicity vs. Computational power

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Polynomial CNN (I)

)
,
(
)
(
)
(
d
d
e
d
e
e
d
e
d
y
u
g
i
u
b
n
y
a
n
x













0
)
(
,
1
0
)
(
,
1
)
(
n
x
n
x
n
y
d
d
d
What’s g(u
d
,y
d
)?

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Polynomial CNN (II)


In the simplest case g(u
d
, y
d
) is a second
degree polynomial, whose general form is






2
0
2
)
)
(
)
(
)
(
)
((
)
,
(
i
i
e
d
e
i
i
e
d
e
i
d
d
y
q
u
p
y
u
g
2
0
0
)
(
)
(
)
1
(
)
(
e
d
e
e
d
e
y
q
p


)
(
)
(
)
(
)
(
1
1
e
d
e
e
d
e
y
q
u
p


)
1
(
)
(
)
(
)
(
2
2
2
e
d
e
e
d
e
q
u
p


Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Polynomial CNN (III)

Thanks to some considerations we find that

)
(
)
(
)
(
)
(
)
(
)
(
)
(
1
2
0
d
e
d
e
e
d
e
e
d
e
e
d
e
d
y
u
p
y
q
i
u
b
n
y
a
n
x

















0
0
0
0
0
0
0
0
c
a
A











p
p
p
p
c
p
p
p
p
b
b
b
b
b
b
b
b
b
B











0
0
0
0
0
0
0
0
0
0
c
q
Q













p
p
p
p
c
p
p
p
p
p
p
p
p
p
p
p
p
p
P
1
1
1
1
1
1
1
1
1
1
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Polynomial CNN (IV)






p
c
p
p
c
c
c
c
d
u
y
p
b
u
y
p
b
n
x
)
(
)
(
)
(
i
y
q
y
a
c
c
c
c



2

u
c

and appear in the state equation




direct link with totalistic Cellular Automata


p
u
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

GoL: Rules (I)

GoL: Rules (II)



Rule 1: a cell will be alive at least 3 of the 9 cells
in its 3
×

3 neigh. are alive

Black pixel= +1

White pixel=
-
1

pixel centr. = 1 (black)

Σ neigh. =
-
2 (5 w, 2 b)

next state =
-
1 (white)

GoL: Rules (III)



Rule 2: a cell will be alive if at most 3 of its 8
neighbors are alive

Black pixel= +1

White pixel=
-
1

pixel centr. = 1 (black)

Σ neigh. =
-
2 (5 w, 2 b)

next state = 1 (black)

Design algorithm (I)


First iteration: we try to perform the first rule (a cell
will be alive at least 3 of the 9 cells in its 3
×

3 neigh.
are alive)



If Y(0)=0

b
c
=1 b
p
=1 i=3






p
c
p
p
c
c
c
c
d
u
y
p
b
u
y
p
b
n
x
)
(
)
(
)
(
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Design algorithm (II)


Second iteration: we try to accomplish the second
rule (
a cell will be alive if at most 3 of its 8 neighbors
are alive
)







p
c
p
c
c
c
d
u
y
p
u
y
p
n
x
)
1
(
)
1
(
)
(
3
2



c
c
c
c
y
q
y
a
Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Design algorithm (III)


Hyp: p
c
=0

Templates found using learning


Coming soon...

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Conclusions (I)


In

general
:



In

some

cases

it

is

possible

to

reduce

a

multilayer

DTCNN

to

a

single

layer

PDTCNN


Thanks

to

the

GoL

we

can

explore

the

capacity

of

PDTCNNs

for

Universal

Machine

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Conclusions (II)


About

learning
:


The

resolution

used

reduces

the

search

space


The

step

“Add

random

parent”

improves

the

behavior

to

avoid

local

minimas


About

design


We

give

a

simple

algorithm

to

design

templates

for

the

Polynomial

CNN

Future Work


Implementations of mathematical morphology
functions with PDTCNNs

Intro

CNN & GA

Polyn. CNN

XOR

GoL

Conclusions

Polynomial Discrete Time
Cellular Neural Networks

Eduardo Gomez
-
Ramirez

Giovanni Egidio Pazienza


egr@ci.ulsa.mx

gpazienza@salle.url.edu