Seminar (METU)

journeycartΤεχνίτη Νοημοσύνη και Ρομποτική

15 Οκτ 2013 (πριν από 4 χρόνια και 25 μέρες)

64 εμφανίσεις



20
May
2011


Middle East Technical University

Faculty of Mathematic & Computer Science


Majid M. Gomainy
-

Çankaya University

majid@cankaya.edu.tr


Adaptive Learning With Using

Dynamic Structure ANN





Outline :





• Introduction to AI


• Biological Neurons


• Single Neuron Model


• Artificial Neural Networks


• Learning and Classification


• Supervised Learning


• Unsupervised Learning


• Adaptive Learning


• Sample Programs Demonstration

References

Introduction to AI

Definitions

History

AI Tests

Intelligent Systems


AI is a branch of computer science,
which aims to study, research, design
and produce intelligent machines
which can act like human, or simulate
human brain functions.

Some of human brain functions are:



Decision
-
making


䍯湴牯氠⡏灴業楺慴楯温


剥捯R湩穥n⡄整散e楯温



Learning



Problem Solving


䕳瑩浡m楯i


. . .


AI is not against the humanity; it is under servicing to
human, in order to make a safe and easy (Comfort) life.



AI Goal is not to simulate an electronic man at all !


History of AI:


The idea of automatic machine was in Ancient Egypt, Greece and


China around
10
AD (~
2000
Years ago !)

1950

Invention of the Computers made a suitable platform for AI

1960

The First AI software named LISP (List Processing) and Prolog


(Programming in logic) as next one.

1970

Expert Systems generated using Rule Based systems

1980

Artificial Neural Networks (ANN) did a big revolution in AI

1990

Application of AI is used in many fields of human life.





Chess
-
playing machine in
1769
.

It was a purely mechanical device;

The machine was a fake.





AI Tests:

Turing Test:

Is a test to recognize a system is intelligent or not.
(Alan Turing
-
1950
)

Chinese Room Test:

Is a test to see, can AI mach to human brain one
-
day, or never AI can be equal to the human brain! (John Searle
-
1980
)





Intelligent Systems:


A

system

in

general

is

a

set

of

collected

objects

with

a

regular

function

from

input

to

output

for

a

specific

goal
.

A

system

can

be

mechanical,

electronically,

social,

biologic,

governmental







An

Intelligence

system

is

a

normal

system,

which

uses

an

Intelligent

Agent

as

Inference

Engine
.

The

Intelligent

Agent

as

seed

of

the

system

simulates

one

or

more

functions

of

the

human

brain
.



Intelligent Agents:


-

Rule

Based

Systems



-

Artificial

Neural

Networks

(ANN)




The

rule
-
based

systems

are

produced

with

prolog

language,

which

uses

a

bank

of

rules

and

facts

for

reasoning
.

Rules

and

facts

are

logical

predicates

that

are

operands

for

logical

operators

as

AND,

OR,

NOT,





These

systems

are

symbolic,

so

they

are

limited

and

not

flexible
.

A

chess

game

can

be

design

as

a

rule
-
based

system

but

in

real

world

everything

cannot

be

defined

as

rules

or

facts
.



The Artificial Neural Networks, are abstract mathematical
model of the human brain, They are non
-
symbolic systems
which are very powerful and flexible.

ANN has ability to simulate of almost any mathematic or
Boolean functions.


Biological

Neurons:


The function of the biological neurons is based on the electro static
potential of the polarized ions inside the cell body. The electric
potentials in human brain is generated from ions of Sodium (Na+),
Potassium (K+) and Calcium (Ca
-
). The total summation of ions
potential inside the cells determines the state of the neuron.


Single Neuron Model:


A mathematic model of
biological neuron is defined by
following abstract structure.
Generally cell comes into existence
form five components, which are
inputs, weights, movement
function, transfer function and
outputs.



A
positive

weight causes one unit to excite
another, while a
negative

weight causes one
unit to inhibit another.



Single Neuron Structure:

X: Input Value

W: Weights Value

T: Threshold Value

b: Bias Value

Y: Output Value

j
: Transformation function


Transformation function

Transformation function (
j
⤠捡渠扥n慮礠浡瑨敭慴楣慬a晵湣瑩潮t慳㨠


q

Linear

q

Thangent

q

却数S⡈慲搠䱩浩瑥爩

q

卩杭潩o

q

偯楳獯渠⡇慵獳(慮a




Artificial Neural Networks

ANN is a Non
-
Symbolic Agent

ANN is an adaptive system

ANN is a time based dynamic system

ANN can be used as state machine

ANN can be used as
learning machine

ANN can be used as Associative Memory

ANN can be used as distributed processing


ANN Application Fields:



q

Pattern Recognition

q

䵡捨楮M 噩獩潮

q

Machine Learning

q

Optimization

q

䍬慳獩晩捡c楯i

q

Function Simulation

q

Control System


ANN has application in almost all fields of the AI as:


ANN Structure:


An ANN consists of a network of many single
Neurons. Where output of any neuron is connected to
the input of the other neurons. And inputs, also is
come from output of the other neurons.

ANN Characteristics:

In order to distinguish an ANN from another one,
there are parameters as follow:



q

乵浢敲



乥畲潮N

⡁湤

䱡祥牳L

q

乵浢敲



䱩湫L



䕡捨

乯摥

q

呯灯T潧o



周T

乥瑷潲k

q

啰摡瑥

䙵湣瑩潮

⡅牲(r

䍯湴牯氩




ANN Action Phases:

The ANN after design and implementation can acts
(works) in two phases:




Learning Phase




Application Phase



In learning phase ANN is under Training. That means the ANN is
adapting itself to the problem conditions and environment using
dummy data, and decreasing the error level. After passing this
phase now ANN is ready for application phase with real data.


But in general Learning implies changes of neurons weights and
the other parameters as threshold.


Learning and Classification

Supervised Learning


Unsupervised Learning


Dynamic Learning


Learning Block Diagram


Machine Learning is a subfield of Artificial
Intelligence that is concerned with design
and development of algorithms and
techniques that allow computers to "Learn"
In general.

Classification:



Actually the concept of learning is very close to the classification.


That is, if a system is able to distinguish input patterns and classify
them into different output classes, We can say the system is learned
those patterns.


Consider Followings:

C Class of patterns

O Object inside Class

n Number of seen Objects






Cardinality of Class C members



Learning Capacity




If n=


瑨t渠†

=
ㄠ†
乯 䱥慲湩湧 ⡍敭潲楺攩(


䱥慲湩湧 楳⁷桥渠h

<
1




If n<<

††
䙡獴 䱥慲湩湧




Two Classes of Objects Three Classes of Objects


Simple learning uses one or more straight line(s) to separate classes.
But for complex problems, straight lines are not enough. That’s why
those problems viewed as nonlinear systems.


Non linear Classification
3
D Classification

Learning Block Diagram:



In this block diagram, Input feeds to both Real system and ANN, and
then another unit compares the output of them. If they are same there
is no error, otherwise if they does not mach, there is an error, which a
signal must return back to the ANN to correct it. After a long iteration
with dummy data, if the error level is decreasing that means ANN is
learning. (Convergent)


Convergence and Divergence :



After a long iteration with dummy data, if the error level is not
decreasing that means ANN is not learning. (Divergent)


Supervised and Unsupervised Learning:


In Supervised Learning both the input patterns and
correct outputs class number are supplied.

But in Unsupervised Learning only inputs are supplied.


In Supervised learning, the learner (ANN) is under
training of a supervisor as a teacher for students who
finds and correct their errors.

But in Unsupervised learning, the learner (ANN) is
trying to learn itself the patterns. As the students who
are learning without teacher.


Dynamic Learning :



Is a special type of learning that the system makes a Compatibility with
the problem environment and can adapt itself to new condition. The
number of the neurons, number of the layers and even the topology can
change during learning and application phase. It also involves modifying
the actual topology of the network, that is adding or deleting neurons
and connections from the network.

Multi Layer Perceptron (MLP):


MLP, also known as Error Backpropagation or the Generalized
Delta Rule, that is the most widely used supervised training
algorithm for neural networks.


Multi Layer Perceptron (MLP):


Input Layer
:


Input layer is a part, which receives data as input pattern from
the real world (Extracted features from the sensor).

Number of
Input Layer neurons is equal to number of features. (Problem
dimension)


Hidden layer :

Hidden layers are midway layers and their numbers is depends
to problem complexity.


Output layer :

Output layer neurons are equal to the number of the problem
classes. (ψ
1
, ψ
2
, ψ
3
,…)



If the output is
1
and should be
1
or if the
output is
0
and should be
0
, so nothing. (No
change to weight)



If the output is
0
(inactive) and should be
1
(active), increase the weight values on the all
active input links



If the output is
1
(active) and should be
0
,
decrease the weight values on the all active
input links


MLP Learning Rules:



MLP Learning Rules:


Learning Rate


MLP Implementation:



I W
1
H W
2
O


Kohonen Network :


The model was first described as an artificial neural network by
the Finnish professor Teuvo Kohonen; it is an unsupervised learner
ANN with Self
-
Organizing Map (SOM) method.



Kohonen Network :

The learner is given only unlabeled examples as input, and it
must map them to labeled areas in output. The function is very
similar to the Clustering technique in the density estimation in
statistics.


Kohonen Network :

The usual arrangement of
output nodes is a regular
spacing in a hexagonal or
rectangular grid with using
large numbers of nodes (in
some maps consisting of
thousands of nodes). The
output layer is very similar to
cerebral cortex in the human
brain.


Uses a neighborhood
function, it is a competitive
process, also called vector
quantization. Mapping
automatically classifies a new
input vector.



Implementation:

O

W

I


Hopfield Network:

A Hopfield Network is a form of recurrent ANN invented by John
Hopfield in
1982
. Hopfield nets serve as content
-
addressable
memory systems with binary threshold units. They are guaranteed
to converge to a local minimum, but convergence to one of the
stored patterns is not guaranteed.



Hopfield Network:

The units in Hopfield nets are binary
threshold units, i.e. the units only
take on two different values for their
states and the value is determined by
whether or not the units' input
exceeds their threshold. Hopfield
nets can either have units that take
on values of
1
or
-
1
, or units that take
on values of
1
or
0
.



Hopfield Network:

Hopfield nets have a scalar value
associated with each state of the
network referred to as the "energy", E,
of the network; The main
characteristic of the H
-
ANN is that as
iterations progress, the computational
energy function reduced and stabilize.

wij is the weight from unit j to unit i

sj is the state of unit j

θi is the threshold of unit i



Hopfield Network:

Running:

At each step, pick a node at random. The node's
behavior is then deterministic: it moves to a state to minimize
the energy of itself and its neighbors.

In contrast, the Boltzmann machine has a stochastic update rule.

Training:


a Hopfield net involves lowering the energy of states
that the net should "remember". This allows the net to serve as a
content addressable memory system; that is to say, the network
will converge to a "remembered" state if it is given only part of the
state. The net can be used to recover from a distorted input the
trained state that is most similar to that input. This is called
associative memory because it recovers memories on the basis of
similarity. For example, if we train a Hopfield net with five units so
that the state (
1
,
0
,
1
,
0
,
1
) is an energy minimum, and we give the
network the state (
1
,
0
,
0
,
0
,
1
) it will converge to (
1
,
0
,
1
,
0
,
1
).
Thus, the network is properly trained when the energy of states
which the network should remember are local minima.



Hopfield Implementation:

I
/
O

W


Dynamic Structure ANN:


=>


As ANN must adapt itself to the problem conditions; It must be able to
change its Characteristics during the Learning phase (like human
brain). One of this parameters can be number of the neurons. In
kohonen model with using Merge and Split, we can add or delete
neurons in specific location.


Human Brain Characteristics :




q

坥楧桴㨠
ㄮ㐠
K朮

q

N畭扥爠潦o瑯t慬aN敵牯湳㨠†

^

⡍潲攠瑨慮a湵浢敲n潦o慬氠s瑡牳t楮i瑨攠
卯S慲 卹獴敭e䵩汫M睡礠䝡污硹⤠ ⡃(唠桡猠潮汹o

^

汯杩捡氠条瑥t)

q

N畭扥爠潦o捯湮散瑩潮s 扥瑷敥渠湥畲潮n ⡓祮y灳攩㨠

^
ㄵ1

q

N畭扥爠潦o捯湮散瑩潮o瑯t敡捨e湥畲潮n 啰U瑯t
㈰〰

q

200
Million connections between left and right sides.

q

The length of total axons
8
Million Km (
200
times of the Earth round)

q

䍯C瑥t s畲晡捥c

m

q

䥴 桡猠
1
/
㔰5
潦o桵浡渠瑯t慬a睥w杨琠扵b 畳敳
1
/

潦o慬氠O硹x敮e潦⁢汯潤

q

Density is
1000
time more than VLSI technology

q

Human Brain receives
20
Million unit of information in each second, but
only less than
1
% is used. (Data Pruning: Removing useless data)



Human Brain Characteristics :




q

䍹捬攠瑩浥ms灥敤㨠

^
-
㌠††
⡃(唠㨠

^
-
㤠 †
潮攠浩汬m潮o瑩浥m晡s瑥爩

q

Memory capacity:
10
^
14
unit (Computer
10
^
9
)
100000
time more
than a computer. (Brain works based on the Associative Memory)

q

䥮I敡捨es散潮搠

^

灲潣敳s 楳 灥p景f浩湧m楮i桵浡渠扲b楮i扵b 潮汹o

^

楳 捯湳捩潵s湥ss 慮a 牥浩湤敲e慲攠畮捯us捩潵sn敳s⸠⡃潮(牯氠潦o
biological balance, as body temperature, blood pressure, …)

q

䥮I浩捲潳捯灩挠癩敷 楳 摩晦敲敮琠晲潭o浡渠瑯t浡渮n⡌楫攠晩湧敲 灲楮i)





“The human brain is the most complex object in the known universe”



And the important Question is:



Can human brain solve itself puzzle one day?


Comparison of the Brains and
Traditional Computers


Memory capacity:
10
^
14
unit


Element size:
10
-
6

m


Energy use:
25
W


Processing speed:
100
Hz


Parallel, Distributed


Fault Tolerant




1
billion bytes RAM but
trillions of bytes on disk


Element size:
10
-
9
m


Energy watt:
30
-
90
W (CPU)


Processing speed:
10
9
Hz


Serial, Centralized


Generally not Fault Tolerant

Conclusion



Here we evaluate the learning as classification.
As in real world the conditions are subject to
change, Then for a Real Learning we need a
dynamic classification which the best way is
using a Dynamic Structure ANN



I also designed and develop a few computer
program for the Learning simulation. The results
is tested by real data set and the final error can be
estimated,


Now let us see the results practically



[
1
] Performance of Perceptron Predictors for Lossless EEG Signal Compression, N.Sriraani,


1
-
C.Eswaran Center for Multimedia Computing, Faculty of information Technology, Multimedia
University,
63 100
, Cybcrjaya, Malaysia, May
2003
.


0
-
7803
-
5041
-
3
/
99
$
10.00 0 1999
IEEE
1273
-

1286


[
2
] Signal Compression by Piecewise Linear Non
-
Interpolating Approximation, Ranveig Nygaard, John
Hcikon Hus@y, Dag Haugland and Sven Ole Aase Hggskolen i Stavanger,


Department of Electrical and Computer Technology P.
0
. Box
2557
Ullandhaug,
4004
Stavanger,
Norway


Proceedings of the First International Symposium on Cyber Worlds (CW.
02
)


0
-
7695
-
1862
-
1
/
02
$
17.00
©
2002
IEEE


[
3
] PRDC: An ASIC Device for Lossless Data Compression Implementing the Rice Algorithm,


Raffaele Vitulli On
-
Board Payload Data Processing Section


European Space Agency


ESTEC TEC/EDP
2004
IEEE


PRDC (Payload Rice Data Compressor)


0
-
7803
-
765
I
-
X/
03
/$
17.000 2003
IEEE


[
4
] TRLE


An Efficient Data Compression Scheme for Image Composition of Parallel Volume
Rendering Systems, Chin
-
Feng Lin, Yeh
-
Ching Chung, and Don
-
Lin Yang, Department of Information
Engineering Feng Chia University, Taichung, Taiwan
2002
TRLE (Template Run Length
Encoding)
0
-
7803
-
8742
-
2
/
04
/$
20.00
(C)
2004
IEEE


[
5
] Ultrasonic Data Compression via Parameter Estimation, Guilherme Cardoso and Jafar Saniie,
Senior Member, IEEE
2005


IEEE Transactions on Ultrasonic, ferroelectrics, and frequency control, vol.
52
, no.
2
, February
2005
.


[
6
] Lossless Data Compression with Error Correcting Codes,Giuseppe Caire Shlomo Shamai Sergio
Institute Eurkcom Techno Princeton University,


ISlT Yokohama, Japan, June
2003


0
-
7803
-
7728
-
1103
/$
17.00200 03
IEEE Trans. on Information Theory,
48
:
1061
-
1081




References