GEP-NN - Department of Computer Science - Ryerson University

jinksimaginaryAI and Robotics

Nov 7, 2013 (3 years and 9 months ago)

98 views

GEP‐NN: Evolving Neural 
Networks for Classification
Stephen Johns
sjohns@scs.ryerson.ca
Ryerson University –Department of Computer Science
October 21, 2008
CP8102 –Graduate Seminar
Agenda
Agenda

IntroductiontoGEP

NN
Introduction
?
to
?
GEP
NN
•GEP‐NN for Iris Classification
l
•Resu
l
ts
•Demonstration
•Future Work
2
10/21/2008
GEP‐NN for Classification                     
Stephen Johns
INTRODUCTIONTOGEP

NN
INTRODUCTION
?
TO
?
GEP
NN
GEP‐NN for Classification                     
Stephen Johns
3
GEP

NN
GEP
NN

AnextensionofGEPtofacilitatethedesignof
An
 
extension
?
of
?
GEP
?
to
?
facilitate
?
the
?
design
?
of
?
neural networks (Ferreira, 2006)

Twoadditionaldomainstorepresentweights

Two
 
additional
?
domains
?
to
?
represent
?
weights
?
and thresholds of a neural network
GEP‐NN for Classification                     
Stephen Johns
4
10/21/2008
GEP

NN
GEP
NN

Allowsforcompleteinductionofneural
Allows
 
for
?
complete
?
induction
?
of
?
neural
?
networks

DemonstratedbyevolvingNNforXORand6

Demonstrated
 
by
?
evolving
?
NN
?
for
?
XOR
?
and
?
6
?r
Multuplexer (Ferreira, 2006)
GEP‐NN for Classification                     
Stephen Johns
5
10/21/2008
ModificationsfromStandardGEP
Modifications
 
from
?
Standard
?
GEP

FunctionsforNN:DTQ
Functions
 
for
?
NN:
??
D

T

Q
•Arrays for Weights and Threshold values
iilfidi
•Dw
, Dt
i
n ta
il
 
f
or array 
i
n
di
ces
GEP‐NN for Classification                     
Stephen Johns
6
10/21/2008
ASimpleNeuralNetwork
A
 
Simple
?
Neural
?
Network
(Ferreira2006)
DDDabab
(Ferreira

2006)
GEP‐NN for Classification                     
Stephen Johns
7
10/21/2008
AMoreComplexGEP

NNIndividual
A
?
More
?
Complex
?
GEP
NN
?
Individual
QcQTdddcacbbcbaad45163901653605000377
D
D
Tail
Head
D
t
D
w
Tail
Head
GEP‐NN for Classification                     
Stephen Johns
8
10/21/2008
AMoreComplexGEP

NNIndividual
A
?
More
?
Complex
?
GEP
NN
?
Individual
1.983
0.389
D=
Q
c
Q
Tdddcacbbcbaad45163901653605000377
QQ
W={1.983,0.542,1.971,0.155,0.405,0.389,1.478,1.861,1.164,0.977}
T={0.612,1.734,0.515,0.336,0.831,0.172,1.059,1.23,1.346,0.394}
GEP‐NN for Classification                     
Stephen Johns
9
10/21/2008
GEP

NNEvolvedXOR
GEP
NN
?
Evolved
?
XOR
(Ferreira, 2006)
GEP‐NN for Classification                     
Stephen Johns
10
10/21/2008
GEP

NNFORIRISCLASSIFICATION
GEP
NN
?
FOR
?
IRIS
?
CLASSIFICATION
GEP‐NN for Classification                     
Stephen Johns
11
IrisDataSet
Iris
 
Data
?
Set
AttributeExampleGEP-NN
Terminal
1
Sepallengthincm
a
2
Sepalwidthincm
b
3
Petal
length
in
cm
c
3
Petal
length
in
cm
c
4
Petalwidthincm
d
5
Class
FisherIrisDataSet(Fisher1936)
Fisher
 
Iris
?
Data
?
Set
?
(Fisher

1936)
5.3,3.7,1.5,0.2,Iris-setosa
5.0,3.3,1.4,0.2,Iris-setosa
70324714Iris
versicolor
7
.
0
,
3
.
2
,
4
.
7
,
1
.
4
,
Iris
-
versicolor
6.4,3.2,4.5,1.5,Iris-versicolor
6.4,2.7,5.3,1.9,Iris-virginica
6.8,3.0,5.5,2.1,Iris-virginica
GEP‐NN for Classification                     
Stephen Johns
12
10/21/2008
EvaluationofIndividuals
Evaluation
 
of
?
Individuals
TDTabacbdcdcb039825289416198
W={0.429,0.052,0.951,1.472,1.283,1.875,1.505,1.815,1.592,0.252}
T
=
{1.621,0.980,1.348,0.314,0.842,1.219,0.468,1.340,1.208,1.489}
[T]
[T]
[D][T][a]
[b][a][c][b][d]
D
[0.429][1.472][1.815]
[0.951][1.875][0.951][1.815][1.592]
Dw
[0.980]
[1.208][1.340]
Dt
GEP‐NN for Classification                     
Stephen Johns
13
10/21/2008
FitnessDetermination
Fitness
 
Determination
4
3
3.5
2
2.5
1
1.5
0.5
1
GEP‐NN for Classification                     
Stephen Johns
14
10/21/2008
0
Implementation
Implementation

UnigenicGEP

NNSystem
Unigenic
?
GEP
NN
?
System
•Developed in C#/WPF
flClli
•XML 
f
or Resu
l
ts 
C
o
ll
ect
i
on
GEP‐NN for Classification                     
Stephen Johns
15
10/21/2008
RESULTS
RESULTS
GEP‐NN for Classification                     
Stephen Johns
16
BestIndividual(h
=
4)
Best
 
Individual
?
(h4)
D=QcQTdddcacbbcbaad45163901653605000377
W={1.983,0.542,1.971,0.155,0.405,0.389,1.478,1.861,1.164,0.977}
T
=
{
0
.
612
,
1
.
734
,
0
.
515
,
0
.
336
,
0
.
831
,
0
.
172
,
1
.
059
,
1
.
23
,
1
.
346
,
0
.
394
}
T
{
0
.
612
,
1
.
734
,
0
.
515
,
0
.
336
,
0
.
831
,
0
.
172
,
1
.
059
,
1
.
23
,
1
.
346
,
0
.
394
}
Maximum Fitness: 96.67% (145 of 150 records classified correctly) 
Reached in the 74th
generation. 
GEP‐NN for Classification                     
Stephen Johns
17
10/21/2008
ConventionalNNRepresentation
Conventional
 
NN
?
Representation
o1
h1
h1
i1
i2
i3
i4
GEP‐NN for Classification                     
Stephen Johns
18
10/21/2008
AveragevsBestFitness(h
=
4)
Average
 
vs
?
Best
?
Fitness
?
(h4)
100.00%
80.00%
90.00%
50.00%
60.00%
70.00%
Avg
30.00%
40.00%
Best
10.00%
20.00%
GEP‐NN for Classification                     
Stephen Johns
19
10/21/2008
0.00%
BestIndividual(h
=
3)
Best
 
Individual
?
(h3)
Q
D
d
c
c
a
d
D=QcDdcdaddcadd156363413859797
W={1.446,0.002,1.085,1.395,0.305,0.074,0.878,1.609,0.874,0.651}
T
{
1
507
1
4
0
82
1
854
0
932
1
204
1
63
0
696
1
292
1
996
}
T
=
{
1
.
507
,
1
.
4
,
0
.
82
,
1
.
854
,
0
.
932
,
1
.
204
,
1
.
63
,
0
.
696
,
1
.
292
,
1
.
996
}
Maximum Fitness: 96.67% (145 of 150 records classified correctly) 
Reached in the 69th
generation. 
GEP‐NN for Classification                     
Stephen Johns
20
10/21/2008
BestIndividual(h
=
5)
Best
 
Individual
?
(h5)
D = TcdQTcdccbdbdaccdbcad6446499541846063600403335
W = {0.785,0.219,1.273,0.938,0.243,1.247,0.068,0.879,1.515,1.967}
T{1904047806510335015901511619032817581072}
T
=
{1
.
904
,
0
.
478
,
0
.
651
,
0
.
335
,
0
.
159
,
0
.
151
,
1
.
619
,
0
.
328
,
1
.
758
,
1
.
072}
Maximum Fitness: 96.67% (145 of 150 records classified correctly) 
Reached in the 37th
generation. 
GEP‐NN for Classification                     
Stephen Johns
21
10/21/2008
BestIndividual(h
=
6)
Best
 
Individual
?
(h6)
D = TcdDaTdcccaddbacbaadbbabd497026093652130146929473012218
W = {0.279,1.644,0.325,1.473,1.332,1.757,0.374,1.597,1.957,0.688}
T{0397017401570202143208150259006518660724}
T
=
{0
.
397
,
0
.
174
,
0
.
157
,
0
.
202
,
1
.
432
,
0
.
815
,
0
.
259
,
0
.
065
,
1
.
866
,
0
.
724}
Maximum Fitness: 96.67% (145 of 150 records classified correctly) 
Reached in the 60th
generation. 
GEP‐NN for Classification                     
Stephen Johns
22
10/21/2008
Observations
Observations

Identicalmaximumclassificationrate(9667%)
Identical
 
maximum
?
classification
?
rate
?
(96
.
67%)
•Terminal inclusion
fffdh
•E
ff
ect o
f
 Hea
d
 Lengt
h
•Hidden Layers
GEP‐NN for Classification                     
Stephen Johns
23
10/21/2008
TerminalInclusion
Terminal
 
Inclusion
HeadLength(
h
)
Terminals
Head

Length

(
h
)
Terminals
3
a,c,d
4
a,b,c,d
5
bcd
5
b
,
c
,
d
6
a,c,d
AttributeGEP-NNTerminalSepallengthincm
a
Sepalwidthincm
b
Petal
length
in
cm
c
Petal
length
in
cm
c
Petalwidthincm
d
GEP‐NN for Classification                     
Stephen Johns
24
10/21/2008
HiddenLayers
Hidden
 
Layers
Head 
Length (h)
FunctionsHidden 
Layers
3
2
1
4
3
1
5
3
2
5
3
2
6
3
2
GEP‐NN for Classification                     
Stephen Johns
25
10/21/2008
PerformanceComparison
Performance
 
Comparison
AlgorithmClassification Rate
GPforNNDesign
(
Riero
Rabñ
alDorado&Paos
2007
)
9866%
GP
 
for
?
NN
?
Design
?
(
Ri
v
ero

Rab
u
ñ
al

Dorado

&
?
Pa
z
os

2007
)
98
.
66%
GEP‐NN96.67%
GEP for Rule Evolution (Zhou, Xiao, Tirpak, & Nelson, 2003)96.00%
GPforRuleEvolution(ZhouXiaoTirpak&Nelson2003)
9400%
GP
 
for
?
Rule
?
Evolution
?
(Zhou

Xiao

Tirpak

&
?
Nelson

2003)
94
.
00%
GEP‐NN for Classification                     
Stephen Johns
26
10/21/2008
DEMONSTRATION
DEMONSTRATION
GEP‐NN for Classification                     
Stephen Johns
27
FUTUREWORK
FUTURE
 
WORK
GEP‐NN for Classification                     
Stephen Johns
28
FutureWork
Future
 
Work

Implementmultigenicsystem
Implement
 
multigenic
?
system
•Improved fitness function
lidd
•More comp
li
cate
d
 
d
ata sets
•Application of adaptive mutation
GEP‐NN for Classification                     
Stephen Johns
29
10/21/2008
AdaptiveMutation
Adaptive
 
Mutation

Two

LevelAdaptiveMutation
Two
Level
?
Adaptive
?
Mutation
–Level 1: Adaptive Region Selection
Level2:AdaptiveMutationofspecificpositions

Level
 
2:
?
Adaptive
?
Mutation
?
of
?
specific
?
positions
•Self‐Emergent Properties

Which areas benefit most from mutation?
GEP‐NN for Classification                     
Stephen Johns
30
10/21/2008
AdaptiveMutation
Adaptive
 
Mutation

4regionsofgene
4
 
regions
?
of
?
gene
ih
QcQTdddcacbbcbaad45163901653605000377
•We
i
g
h
ts
•Thresholds
GEP‐NN for Classification                     
Stephen Johns
31
10/21/2008
REFERENCES
REFERENCES
GEP‐NN for Classification                     
Stephen Johns
32
References
References
•Ferreira
,
 C. 
(
2006
)
. Desi
g
nin
g
 Neural Networks Usin
g
 Gene 
,()ggg
Expression Programming. In A. Abraham, B. de Baets, M. 
Köppen, & B. Nickolay, Applied Soft Computing Technologies: 
TheChallengeofComplexity
(pp517
536)Springer
Verlag
The
?
Challenge
?
of
?
Complexity
(pp

517
?r
536)

Springer
?r
Verlag
.
•Ferreira, C. (2001). Gene Expression Programming: A New 
Adaptive Algorithm for Solving Problems. Complex Systems, 
13(2), 87‐129.
•Fisher, R. A. (1936). The use of multiple measurements in 
taxonomicproblems
AnnualEugenics7PartII
179
188
taxonomic
?
problems

Annual
 
Eugenics

7

Part
?
II

179
?r
188
.
33
10/21/2008
GEP‐NN for Classification                     
Stephen Johns
GEP‐NN: Evolving Neural 
Networks for Classification
Stephen Johns
sjohns@scs.ryerson.ca
Ryerson University –Department of Computer Science
October 21, 2008
CP8102 –Graduate Seminar