SELF-ORGANIZATION IN THE COMPUTATION MODELS OF NEURAL NETWORKS AND DECLARATIVE PROGRAMS

yalechurlishAI and Robotics

Nov 7, 2013 (4 years and 3 days ago)

86 views



D.Zerbino, Y.Tsymbal, Y.Kynash

Lviv Polytechnic National University,

Automated Control Systems Department


SELF
-
ORGANIZATION IN THE COMPUTATION MODELS OF NEURAL
NETWORKS AND DECLARATIVE PROGRAMS




Zerbino D., Tsymbal Y., Kynash Y., 2013


The two models
of computations have been compared


the artificial neural networks
and declarative programs based on logic. The generalization of these models is proposed in
the part where the computational process can cause changes
of
the program. The principles of
self
-
organization for these models have been formulated. According to these principles the
above
-
mentioned changes will be determined as a result of the search. Complementarity of the
discussed models is necessary for a profound study of the self
-
organization.

Keywords: PROLOG, artificial neural networks, model of computations, self
-
organization, genetic algorithms.


Introduction

It’s g
enerally accepted that a living organism is a highly organized system that specifically
supports the own existence in a
variabl
e

environment and can
re
produce the organisms with a
positive changes that improves the properties of
the
adaptation to the environment.

This definition
perfectly suited to the term "self
-
organization", which further emphasizes that the ordering of the
sys
tem is growing and the reason for this is the system itself.

If self
-
organization does exist in nature, the "
theoretically

alive" may appear the quantum
systems on the surface of the sun, and crystal structures on the distant planets at very low
temperatur
es, and artificially created technical systems.

To make this article not a science fiction, it
is necessary to solve the following:



Are there any general principles of self
-
organization without which self
-
organization is
impossible;



If the principles of se
lf
-
organization exist, is it possible to apply them to the software systems
in the form of formal operators;



Is it possible
to establish acc
o
rding to

these principles a new programm
ing technology that
would allow

programmatic capturing all of the positive
changes in programs for the purpose
of self
-
improvement, and automatically organize program objectives and optimize the ways to
achieve programming

purpose
s, or modify the algorithm as a
whole?


It is generally recognized that
for

the
effective management
of the system
,

its complexity
should not be high, otherwise

the

control
of all

processes becomes impossible.

So
,

highly

orga
nized
system can
not exist without the laws of
the
self
-
regulation and self
-
organizations which must
simplify
the
management through
spontaneous contours of regulation.

When organisms reach a
certain level of organization, they begin to interact and form
the
other systems with its
own
contours
of
regulation, which is also a result of self
-
organization.

The phenomenon of
multiv
arious
lif
e forms is still no
well understood, that suggested to the actuality

o
f the studing the

self
-
organized
algorithms.

We also believe that there is a need to explore how highly organized systems

are
formed

and how spontaneous control circuits that coordinate
the overall management system

arize
.

The aim of this work is to clarify the concepts of adaptation and self
-
organization, identifying
characteristics and principles of self
-
organization, to identify the models of artificial neural
networks and declarative
programs properties that define symptoms of self
-
organization.


The b
asic principles of self
-
organization

The
complex, but not self
-
organized systems
exist i
n the environment.

Self
-
organization is
necessary but not
a
sufficient property of living systems.

We consider the self
-
organization as a
purposeful change the own structure (possibly to optimize circuits regulation therein).

Thus, in
order to make a meaningful step
it is necessary

to identify possible alternative steps, and then
det
ermine which of the
alternative is

best and by what criteria.

Particular cases are

following
:
impossible to represent all the a
lternatives, criteria too much.
In this case, it is necessary to
look

for
a base
problem and

to solve it through the analysis of logical relationship
s.

Most problems are
interrelated, and
in contrast to

adapting systems

the self
-
organized systems solve problems together.

Obviously,
to

"guess" the correct solution for the system is simply impossible.

It is therefore
logical to assume that the self
-
organ
izing systems have the hidden algorithm of sorting variants with
logical analysis of each, after which the system selects the best one.

The property of self
-
organization of the system does not depend on its nature.

For example, in [1] were considered
simpl
e rules of replacing data on the discrete plane, which led to the deliberate “breeding” of certain
data configurations.

In simple systems all laws
are determined

by the environment.

Under their influence

"screening"
of pre
-
configuration data, as a result

s
ome configurations are preserved and even
incr
eased, while others occurs

transformed into another form and "die".

In

complex systems, there
is some subject that decides which data to "show"
to achieve a certain purpose

in accordance with
the laws of the en
vironment.

Therefore, we propose
to
divide the processes of self
-
organization into
spontaneous (which depends only on the laws of the environment) and purposeful self
-
organization
(if managing entity

exists
).

If the managing entity has too primitive functi
ons, the self
-
organization

can

be spontaneous
ly
, i
.
e
.

mainly depending on the laws of the environment.

The greater
is the
possibility of the subject to

"hide" data
from environment, the "smarter" and
more autonomous self
-
organization

is
.

To study this phen
omenon,
let us
first consider the artificial neural networks that were
designed by developers as an alternative to classical processors and distinguished on the basis of
computations.

If classical architecture processors require the programmer to write a p
rogram based
on a model problem, the computations by neural networks should be based on the following
principles:



Neural Network creates itself the model of the object which is controlled.

This principle will be called the first principle of self
-
organizat
ion of neural network.

For
example, we have a task to create an artificial neural network to recognize any spe
e
ch represented
in the form of digitized signal with certain accuracy.

In this task, we distinguish two approaches:
parametric and algorithmic app
roach.

In the parametric approach
we shall

simply change the scaling
so that the received signal maximum coincided with the model.

Scaling can be nonlinear.

Horizontal axis represents frequency scale
, and vertical
-

amplitude.

Clearly, thi
s approach does n
ot
give qualitive

results when the
announcer delays or shortens

phonemes

or concerned person
speaks
with

another tone of voice.

Attempt

to overcome these problems is given in [2].

Algorithmic approach
is
based on selecting certain logical features of signa
l (on the basis of
repeated analysis
-

getting primary, secondary, tertiary features, etc.).

Neural Network itself must
choose which of these features are essential and which
-

not.

For example, to detect the sound "R" is
an essential process fluctuations
tongue that can recognize either visually or through periodic
characteristic sound of sharp fluctuations on the first front.

Note that the oscillation frequency to
detect sound is less informative than trying to move his tongue speaker in combination with
the
larynx is so that we unde
r
stand sound "R".

During the pronunciation of arbitrary sounds informative
is the sequence of movements and muscle tension relatively the average value.

In any case, by the
nature of sound the listener interprets in what manner

moves the muscles of the lungs, larynx,
tongue and lips, and thus understands that the announcer would say, even if the sound is distorted.

In the example above material is feature extraction algorithms.

The number of features reflects
the diversity and d
epth of analysis.

Therefore, an algorithmic approach to recognition is based on
algorithms that represent the logic of feature extraction based on the physical interpretation of every
slightest change in the signal.

Man interprets the signal by the feature
d speaker’s desire to do this or
that sound, namely, by expressing of this desire through the tension changes of different muscle
groups.

That interpretation of the signal as a carrier of information is possible only if the "recognizer"
has the process mod
el and if the recognizer understands by external signs, how the process takes
place and what events it happen.

For example,
by interpreti
n
g

the smallest features in the human
face image we can instantly recognize the mood and emotions of another person, bu
t the mood of
beetles or birds
is
difficult
to
recognize because recognizing model in this case is incomplete.

Of course, the Neural Network is not a super
-
intelligence that
understands

all the details of
pronunciation, but it can create a model of pronunc
iation
of
appropriate sounds on some primitive
level, which is enough for recognition, not only words but also emotions
of
announcer.

So, the first
step required
for

se
lf
-
organization in recognition

is constructing albeit primitive, but consistent
model
of

recognition.

In the process of self
-
organization, this model will be improved.

The considered above first principle of self
-
organization of neural network is not enough to
make the correct decisions.

We

must also set the purpose

of computation, i
.
e
.
, obje
ctive
of
decision
-
making (or target
of
recognition) as well as evaluation criteria, whic
h play a significant role in

feedbacks

of the system
.

Therefore, we formulate the second principle of self
-
organization for
neural

networks, which includes a purpose
:



I
n the process of computation Neural Network itself cl
arifies the purpose of computation

and
improves the evaluation criteria.

Thi
s purpose

can be h
ierarchical,

multicomponent
and
disordered.

During the self
-
organization a key role
is played
a
system
-
create
d factor, in the purpose to

which the elements
formed the system and interacts

with each other
.

Changing a system
-
created factor leads self
-
organized system to crash.

E
valuation criteria
o
bjective can be regarded as a partial order on the set of attributes

of the
computational process.

This means that
for self
-
organization
the system should monitor itself and
determine the charact
eristics and parameters of its own

work.

Partial order on the
set of features
determines what

features are more desirable.

Charac
teristics generally

are

represent
ed by

a number
of different specific types such as probability, duration, frequency, relative level, quantity, etc.

They
are formed not
randomly, but due to

approaching or distancing

of computation purpose
.

Criteria of
reac
hing the purpose

are
formulated separately.

Usually, you can compare
only
features
of
the same
type, and based on these comparisons, self
-
organizing system can choose the best variant of
function.

Computation algorithm consists of arithmetic and logical op
erations, comparison operations
and transactions by condition.

In determining the results of operations such actions as assignment

should be avoided
to avoid
ed
.

T
he assignment
causes the main problems during modification of
algorithms in systems
.

Therefore

for self
-
organized systems all values must be temporary, i
.
e
.
, if
necessary, could
be
easily replace
d

without assimilating other data associated with them.

This can
be implemented by special hierarchical memory model in which all data have the status of
a
ssumptions.

In the Neural Network programming two principles of self
-
organization are

implemented as
learning,

which
determines the
model of the object and purpose of computing.

In fact, learning
result
s determine

coefficients of connections between neuron
s in adjacent layers or feedback
coefficients (depending on the topology of the neural network).


The sample
of creating self
-
organized system

Suppos
ing

we need
ed

to create a neural network to recognize any word pronounced by
different people.

Solving the
problem directly with the "sound perceptron" [3] does not give the
desired effect, because it would have led

to scaling

the word by a fixed number of receptors,
representing the

digitized signal (since learning is carried out for a fixed number of neurons)
.

W
e
have

to divide speech on
to

separate words,
which

sometimes
is
quite difficult.

We propose to build a system with elements of self
-
organization
taking into account
some
basic knowledge.

So to recognize the human sounds we must gather the following basi
c knowledge:

a) About sounds that a person

forms

during
fixed breathing, fixed
opening of the oral cavity
and different tone

certain muscles of larynx;

b) About sounds that a person
forms

when the

larynx

is relaxed
,
breathing is
fixed,
opening of
the mouth

is
fixed and position
of
tongue
is changed
;

c) About other base sounds, formed by a person when all parameters are fixed, and one
is
changed.

Now the process of self
-
organization will be minimized in order to identify the signal that
uniquely characterize
s

the
tone of

studied muscle groups in order to ensure that the system corre
ctly
identified a group of contracting

muscles, lungs breathing, tongue and mouth position while speaker
is
saying a phoneme.

This concludes the construction of the model of pronun
ciation, based on the
so
-
called "basic knowledge".

To the recognition of words we can attempt only after the successful
constructing a model of pronunciation, mentioned above, presenting each word that says the
speaker through a sequence of combinations of

contractions

of certain muscle groups that appear in
that model.

These properties of neural networks are implemented
i
n the relatively primitive tasks
and models that can be associated with the "conditional reflexes" on a certain data structures.

For
furt
her development of self
-
organizing systems it is necessary to do the few next obvious steps.

1.

To find the best optimal behavior self
-
organized
systems

should contain a mechanism of
c
o
mpreh
e
nsiv
e searching of

all possible
achievements
of the
purpose

w
ith the

possibility of
continuing the search from any

position.

2.

To

reduce the exhaustive search

self
-
organized

systems should have

a mechanism of logical
statements that dividing

all
solutions
in
to more promising and less promising.

3.

During solving
the
problem the

system can
formulate the

logical statements based on
the
basic
knowledge
of the problem, which can later
be used in the feature
s extraction algorithm
.

4.

The p
rinciple
of experience ac
q
uire
ment
depending on the number and variety of solved
problems is realiz
ed by

logical statements accumulation
approved by the

system.

5.

The Logic i
n

the self
-
organized

systems must be converted into a universal
abstraction
level,
using the principle of generalization and substitution
of
concepts.

It consists

in the fact that
eac
h algorithm can be
a
part of another parent algorithm and the same algorithm can act as a

parent


and a “child”
algorithm.

That is, the algorithms must be
universal elements of formal
logical systems that can be replaced

and substituted
.

Now
we could not
to say that modern neural network
s are ready to implement all

these
properties
. The

methods of
their
teaching are imperfect
since they

do not allow you to make
logical
propositions
.

In modern
theory
of neural networks

the

logical analysis is actually repla
ced by the
p
robability that cannot convey some

conventional concepts

which
logically depend on the situation.


The declarative approach to describe self
-
organization

Along with the neural network technology
such

approach
as
the
declarative programs

is bei
ng
developed
, which previously was considered
as unideal
from the point of view of practical
programming. The main reason for such "dislike"
by

software developers is
unpredictability of

declarative program behavior because
instead of sp
e
cifying the
algori
thm

th
e programmer
specifies

the
system
of logic assumptions

on which
that algorithm should be

built
by the
executive system.

The task simple

from the point of view

of alg
orithmic language programming
should
be
formulate
d

in

the declarative language

in com
pletely
unnatural way.

Of course, during

a logical interpretation
of such declarative programs various side effects
arise requiring

additional logical
definitions and
refinements thus annoing the

programmers.

Ultimately, the compiler still translates

the d
eclarative
program into

algorithmic

one
.

With the development of neural network technology declarative programming takes on
a
new
meaning.

In our opinion
, it can
be
develop
ed

in the directi
on of self
-
organizing technologies
.

The main feature of the system
that is capable of self
-
organization is
that it
is not destroyed

during
self
-
organization

moreover it
becomes more stable and reliable.

If in the

program created in
any algorithmic language

something is changed it will

lead to a collapse of the ent
ire soft
ware
product, which does not

happen with a
declarative program.

So,
the declarative approach assumes
the

specific language
s

and programm
ing techniques that improve its

own code.

The

programming language is

a universal way to communicate with computer syste
m,
regardless of how
computations are performs inside the system
.

One of the major disadvantages

of
modern high
-
level languages is the inability to modify its own program
in the
natu
ral and logically
justified way
.

That is, in
modern programming languages
the
means represent
ing the program
as
the
d
ata
with the abilities to
its
correct
modification

do
n

t exist.

There was

only one w
eak attempt in this
direction



the

genetic algorithm
s

[4].

The genetic programming allows

two types of programs
modification,
wh
ich are based on randomness hybridization

and mutation
s
.

The

experiments on real
programs synthesizing
the simple
logical formulas
, in this way showed the significant difficulties
arising

with formulas containing
more than
5 variables.

The obtained result
is either too
approximate or too

complex.

From this we can conclude that the program with elements of self
-
organization should not act

in

completely random way
.

Note that modern processors theoretically have the ability to modify its own program
-

it
requi
res only the results of the program
write to

the memory
segment

of the program
code.

In other
words, the technical capability of modern processors allows the creation of methods and tools to
dev
elop self
-
organized programs

and self
-
organized programming la
nguages.

From general considerations we can assume that a program that is capable o
f self
-
organization
must be
writt
en by a programmer, and the program

can make

change
s

itself
with some submissions
.

For example, we

have considered the programs

for the synt
hesis of functions based on genetic
algorithms that perform c
o
mpreh
e
nsiv
e

search of all possible com
binations of basic functions, from

which the function
is
synthesized.

Logic of search reduction

is
based on the assumption that the
synthesized function
doe
s not differ significantly from the basic function
.

Thus,

in a program that is capable of self
-
organization, it should
be clearly described

the
fundamental relationships, the relationships which can be modified
and
the rules that can
not be
broken
.


The m
et
hods of correct modification of the program code

So, to implement self
-
organization we

must
at
first
include to

t
he programming language tools
that can modify its own program code in the way

that does
not
destroy
the
main
algorithm

of
searching

variants
.

I
t is the most convenient to do this

in declarative programming languages such
as PROLOG.

Since declarative langu
age consists of predicates, expressing

some relationships between
concepts of programming,

the special predicates that

can
change
other
predicat
es

in

the program

should be specified
.

This well agrees with the concept tha
t the initial program is

created

by a
programmer, and the algorithm is modified during its execution by
finding and
introducing
to the
program a

new, more
perfect

predicates.

To im
plement this concept in
the programming language
it
is necessary
:

1.

To refuse

of any limitat
ions of

memory by the number of variables and alternatives, by

the
length
of
predic
ates, variables, lists, etc., because

any
concretization

of
values leads to
collaps
e after modification of program code
.

2.

To inroduce

a specialized predicate COMPILE ("filenames (X1, X2, ..., XN)", ERR), which
can perform compilation given text file.

If the compilation was successful, and the error co
de
ERR = 0, then the

first predicate
f
rom

the newly compiled file
will execute

with the
parameters (X1, X2, ..., XN).

If
the
predicate

was
successful
,

the result is returned to

the main
program

by sending the variables
.

In the case of
infinite
loop

of program

(leading to stack
overflow)

predic
ate return code ERR = 1.

3.

Learn to program synthesis, which is based on replacing the infinite

sequences

by finite one
,
and to allow designation of any data as the second, third, etc. layer of new data.

4.

Introduce the operation

of

predicate

creation in

the
s
ecti
on «Proposals», operation of creating
the variable into

the predicate

and predicate removal operation.

5.

To facilitate the programming
it is necessary
to introduce specialized predicates

of

enumeration
, such as sorting
of
all subsets or
searching all sub
strings that satisfy to some

condition.

6.

For

sol
ving

of
the contradictory problems

the ordering
operation

such as "X is more desirable
than Y
"

should introduce
.


The self
-
organized programs properties

Consider the scheme of classical declarative program wit
hout self
-
organization:

The set of possible options


The rule which describes the purpose


Finding the variants
that satisfy the rule


Choosing the best variant.

Scheme program with elements of self
-
organization:

The rule describes the purpose, possible

actions and the initial state


Analysis of purpose
achievement


Search the factors of

purpose achievement


Accumulation of the promising
variants (hypotheses)


Hypotheses considering


Choosing the best variant.

The
rule that describes the purpose and

rules of searching

the best
variant are present in both
programs
.

But in the first program
the set of variants
from which
is necessary
to choose the best

one
,
is
given a
priori, ie before algorithm execution
.

The input data of the program is only the init
ial
problem condition.

The second program is also a priori given some options, but their content is
completely different
-

these options are considere
d as limitations on computations

that can not be
violated in the process of finding a solution.

Therefore,

a rule that describes the purpose and rules
that define

certain restrictions act together
.

Self
-
organization stipulates that the program does not simply selects a
n item from the
proposed set but

can be purposefully chosen composition of the proposed items
.

With this approach,
you need a po
werful logic block analysis of purpose
.

In the analysis, the program needs to find a set
of concepts t
hat are closer or move away the purpose
.

What
,

in fact
,

is the concept? Conc
epts are some sets, which is built by

some
rule.

Thus,
if we
operate
in terms of logical conditions, in fact, we operate in sets that represent them.

So, choosing
the best variant

-

is to find
ing the maximum element from
sets

intersection, which confine our

task.

However, not every rule denotes con
cept
. Rather, the concept associated
with
algorithm that
gives some quantitative characteristics.

For example,
if we consider the chess playing
, the concepts
can be:

"Value
of the figure
"
-

the numb
er of cells to which it can jump

(exception


the
King);

"
A
ttack"


the situation
when a figure lowers

value

attacking
a
figure greater value;

"Fork"
-

a figure simultaneously attack

more than one opponent's figures
;

"Ligament"
-

the figure can not move

because of

opportunity to attack;

"Value
of
position"
-

the
number of active figures as well as th
e number of possible attacks
from our figures

and
attack absence from

opponent;

"Worth
of
t
he turn
"
-

the value of position after the opponent

choose the
optimal response

t
o

this turn.


Implementing the logic of self
-
o
rganization

It is necessary the mechanism of

replacing each eq
uation Ψi (X, Y, ..., Z), which describes the
problem by

the
sets {(x1, y1, ..., z1), (x2, y2, ..., z2), ...}

of solutions
, where (xi, yi, ..., zi)
-

a sin
gle
solution that satisfies that
equati
on
.

The system of equations means
the
logical conjunction of
equation
s, ie, the

intersection of
sets
solution
s of each equation.

In most cases the finite equation
should be
replace
d by

infinite set
s

of solutions.

The first step for the repr
esentation of in
finite sets

can restrict some n
-
finite sets of solutions
of these equations in combination with solutions that give algorithms

i
, which under cyclic
application will get the rest solutions
{

}

from

infini
te set

for the
i
-
th equation (
in addition to those
already received):


(X,Y,…, Z)


{(x
1
,y
1
,…,z
1
), (x
2
,y
2
,…,z
2
), …, (x
n
,y
n
,…,z
n
)}


{

}

{(x
1
,y
1
,…,z
1
), (x
2
,y
2
,…,z
2
), …, (x
n
,y
n
,…,z
n
)}


{

} =


Of course, the system of equation
s is a partial case of the predicate and in the more general
case the

equation

(
or predicates), which it includes
, may have a different number of variables.


Then
to
taking into account

types of variables
which are
included in the predicate, the solution can be
obtained by the rule
of
intersection relations
,

because
of
every relatio
nship is a table on a given
subset of variables:

R
1
(
X
,
Y
,…,
Z
)


R
2
(
P
,
Q
,…,
Z
) =
R
3
(
X
,
Y
,
P
,
Q
,…,
Z
)

Typically, the real system get

task
s

like
:

"do something otherwise
would
not
be
very good".

This information has two parts: 1)
what is
required to do, and 2) th
at
could be

wrong if this is not
done.

If we consider the system without self
-
organization, the modality "Do something" and
"otherwise would not be very good"
are necessary
, ie, without
any

choices.

The system, which has elements of self
-
organizati
on, cons
iders all allegations with

modality
"maybe" as well as truths
of
regarding some subjects.

That is, someone offers someone to "do
something",
else
maybe would "not very good"

for someone
.

In developed intelligent systems
the
part "is not very good" is not

u
sed. The system, which has
own

intellect
understands
what exactly is wrong and who needs to do something.

The
system
during
the analysis of

modality "maybe" must unde
rstand why the subject who proposes

this decision
considers it the best among all other po
ssible solutions.


Conclusion

Self
-
organization is the basic concept of the system

development
. Any system (technical or
biological) that is not well
-
organized
goes to crash.

The article makes

an attempt to introduce
the self
-
organization in

software syste
ms formally,
in

the form of additional
sorting

and searching operations

that still makes the programmer, adapting
their product
to changing demands of using.

In the result of

comparison of different algorithmic
systems
-

artificial neural networks and decl
arative programs, we concluded that

the self
-
organization effect

in programs is arose if
these
computational

models are cooperates.

T
he fixed topology of artificial neural network does not allow
making
deep logical
conclusions as it makes declarative progr
am
, but

declarative program must learn to
tendentious
change their code and select
s

the rules by the weight coefficients
.

The
both algori
thmic systems must learn to operate

with approximate, incomplete,
heterogeneous and contradictory information.


1.
Neum
ann von J. Theory of self
-
reproducing automata. (edited by A.W.Burks) Univ. of Illinois
press, Urbana, 1966.
-

400p. 2
.
Ю.Рашкевич, Д.Пелешко, М.Купчак, А.Ковальчук, Виділення
квазістаціонарних ділянок мовного сигналу за спектром матричного

оператора.
-

Вісник НУ
ЛП №710, 2011р., сс. 70
-
74.

3.
Frank Rosenblatt, Audio Signal Pattern Perception Device.
-

US
Patent 3287649, Nov. 22, 1966.

4.
Candida Ferreira. Gene Expression Programming:
Mathematical Modeling by an Artificial Intelligence (Studie
s in Computational Intelligence).
-

Springer, 2006, 498p.