Cellular Neural Networks

Reported by:

Hamada Ahmed Khadragy

1

Abstract:

A novel class of information processing system called cellular

neural network is proposed. Like a neural network, it is a large-

scale non linear analog circuit which processes signals in real

time. Like cellular automata, it is made of a massive aggregate of

regularly spaced circuit clones called cells, which communicate with

each other directly only through its nearest neighbors. Each cell is

made of a linear capacitor, a nonlinear voltage-controlled current

source, and a few resistive linear circuit elements. Cellular neural

networks share the best features of both worlds; its continuous time

feature allows real-time signal processing found wanting in the

digital domain and its local interconnection feature makes it tailor

made for VLSI implementation. Cellular neural networks are uniquely

suited for high speed parallel signal processing.[1]

Introduction:

After the rebirth of the interest for artificial Neural Networks in 1982

[1],

many

different implementations for this concept have been developed. The differences

between these implementations affect the overall characteristics of the network and

thus the applicability for a certain purpose like, for example, real time image

processing. Here, an important feature of neural networks is touched upon; these

circuits can operate very fast since the 'signal processing' is done simultaneously in

parallel neurons, rather than in sequential time steps.

The major disadvantage of most neural network implementations however is the

number of interconnections between neurons.

In

order to reduce the number of

interconnections but keep the advantages of parallel processing, Chua and Yang

proposed a so-called Cellular Neural Network (CNN) in 1988 where neurons were

only connected to other neurons within a certain neighborhood.

In

1992, Rodriguez-Vazquez proposed a modified model for neurons; the so-called

Full Range (PR) model and an implementation for a CNN in the current domain based

on this FR-model.

In

1996 P. Bruin suggested an implementation for a FR-CNN in the voltage domain.

However, after realization of the implementation, it was found to display instabilities.

Every neuron in a cellular neural network can be described by a certain state-

equation. If this network consists of two interconnected cells, the behavior that is

displayed by those cells can be plotted in a so-called state space.

It

is interesting to be

2

able to 'predict' to what final state the CNN will converge. For this, the state-space

will be partitioned into 'basins of attraction', separated by certain boundaries. In this

report a method to find the boundaries of the basins of attraction is proposed.

A Cellular Neural Network

A Cellular Neural Network (CNN) is an artificial neural network consisting of

separate neurons or cells. It has several properties that may be advantageous

compared to other neural networks. The dynamic range of a CNN for example is

bounded. Also a CNN can easily be extended without having to re-adjust the entire

network because a cell is not connected to every other cell in the network but rather to

cells within a certain neighborhood. Albeit its cellular structure it still displays the

complex dynamic behavior as seen with other neural networks. Due to this complex

behavior it can be used in image processing (e.g. noise-removal, connected

component detection (CCD), 'thinning' etc.) It can also be used to simulate certain

equations or be used as an associative memory. Another advantage is that although

the factor with which the output of a cell affects the behavior of other cells (template

parameter) may be different for spatially different neighboring cells, the template thus

formed is translational invariant ('cloning templates'). As stated above, these cells can

interact with other cells within a certain neighborhood. A rectangular two-

dimensional CNN consisting of 16 cells where every cell interacts with only its

directly neighboring cells would look like the circuit shown in Figure 1.

If C(i,j) is the cell on the ilh row and jth column then cell C(2,2) is connected to

C(O,I), C(O,2), C(O,3), C(2,I), C(2,2), C(2,3), C(3,l), C(3,2) and C(3,3).

3

Architecture of Cellular Neural Networks

Any cell in a CNN is connected only to its neighbor cells. The adjacent cells can

interact directly with each other. Cells not directly connected together may affect each

other indirectly because of the propagation effects of the dynamics of CNNs. An

example of a two-dimensional CNN is shown below.

Every cell is influenced by a limited number of cells in its environment. This locality

of connections between the units is the main difference between CNNs and other

neural networks. Large CNN chips can be implemented using VLSI techniques.

The figure above shows the emphasized cell (black) connected to the nearest

neighbors (gray). The cells marked in gray represent the neighborhood cells of the

black cell. The neighborhood includes the black cell itself. This is called a "3*3-

neighborhood".

Similarly, we could define a "5*5-neighborhood", a "7*7-neigborhood" and so on.

The basic circuit unit of CNNs is called a cell. It contains linear and nonlinear

circuit elements, which typically are linear capacitors, linear resistors, linear and

nonlinear controlled sources, and independent sources. All the cells of a CNN have

the same circuit structure and element values. A typical circuit of a single cell is

shown in the figure below.

4

Each cell contains one independent voltage source E

uij

(Input), one independent

current source I (Bias), several voltage controlled current sources I

n

u ij

, I

n

y ij

, and

one voltage controlled voltage source E

y ij

(Output). The controlled current

sources I

n

u ij

are coupled to neighbor cells via the control input voltage of each

neighbor cell. Similarly, the controlled current sources I

n

y ij

are coupled to their

neighbor cells via the feedback from the output voltage of each neighbor cell.

The cell C(i,j) has direct connections to its neighbors through two kinds of weights:

the feedback weights a(k,l;i,j) and the control weights b(k,l;i,j), where the index pair

(k,l;i,j) represents the direction of signal from C(i,j) to C(k,l). The coefficients

a(k,l;i,j) are arranged in the feedback-Template or A-Template. The coefficients

b(k,l;i,j) are arranged in the control-Template or B-Template.

The A-Template and the B-Template are assumed to be the same for all the cells in

the network. The global behavior of a CNN is characterized by a Template Set

containing the A-Template, the B-Template, and the Bias I.

If we assume a "3*3-neighborhood", the Template Set consists of 19 coefficients.

The external input to the cell is typically assumed to be constant over a certain

operation interval. Therefore, the total input current to the cell is given by the

weighted sum of control inputs and weighted sum of feedback outputs. In addition, a

constant bias term (I) is added to the cell. Due to the capacitance C and resistance R,

the state voltage x(i,j) satisfies the following differential equation:

k denotes the neighborhood of the specific cell

Without loss of generality, the time constant T = R*C can be set to 1.

The only nonlinear element in each cell is a piecewise-linear voltage controlled

voltage source with characteristic

y(i,j) = f(x(i,j)).

A widely used nonlinearity is the piecewise-linear function as given by:

y(i,j) = f(x(i,j)) = 0.5*(|x + 1| - |x - 1|)

5

The block diagram of a cell C(i,j) is shown in the figure below.

Modeling and Simulation of the CNN

architecture

-Simulation plays an important role in the design of the CNN cloning templates.

-Therefore, it has to be fast enough to allow the design phase of various templates

be accomplished in reasonable time.

-At the same time, the simulation has to be accurate enough, to reflect the

behavior of the analog circuitry correctly.

-In practice, the simulation of the CNN involves a trade-off between accuracy and

computation time

-The true processing capabilities of CNNs for high-speed parallel processing are

only fully exploited by dedicated VLSI hardware realizations.

-Typical CNN chips may contain up to 200 transistors per pixel.

-At the same time, industrial applications require large enough grid sizes (around

100 x 100).

6

-Thus, CNN chip designers must confront complexity levels larger than 106

transistors, most of them operating in analogue mode

-On the one hand, high-level simulation, which is focused on emulating the

functional behaviour, cannot reflect realistically the underlying electronic

circuitry. Their lack of detail makes them ill-suited for reliable IC simulation.

-On the other hand, the SPICE-type transistor-level simulators, although very

accurate, are barely capable of handling more than about 105 transistors and may

take several days of CPU time for circuit netlists containing about 106 transistors.

-Hence, these low-level tools are ill-suited for simulating large CNN chips.

-Therefore, it would be necessary to bridge the gap between these approaches,

which would give very accurate results in reasonable (but not real-) time.

-However, our main concern now is fast simulation

Global behavior of Cellular Neural Networks

In image processing, n-by-m rectangular grid arrays are often used. n and m are the

numbers of rows and columns, respectively. Each cell in a CNN corresponds to an

element of the array.

Assuming that each cell is connected to its nearest neighbors only ("3*3-

neighborhood") and that the local connections of a cell do not depend on the cell's

position, the Template set contains 19 coefficients (A-Template: a1 .. a9, B-Template:

b1 .. b9, Bias I). The behavior of the CNN is completely determined by this Template

set.

New CNN-Templates for arbitrary tasks may be found using a training algorithm, or

by defining local rules for a given global task. The local rules describe a cell's

equilibrium state depending on the inputs and outputs of the neighbor cells. The

inputs and the outputs of the neighbor cells are assumed the be constant. The

dynamics of the cell is not specified.

If Template values for the local rules are found, simulations are very helpful to test

the dynamic global behavior of the entire clone of cells.

Optimal coefficient calculation leads to solutions which converge after short time.

This means that the output of every cell reaches its final output y=+1 or y=-1 after

short time.

Characteristics of a simple CNN Template

Template set

We assume "3*3-neighborhood". A simple Template set for edge extraction is

7

Global task: Binary edge detection

If the input image is a binary image (black and white), the output of the CNN

will be a binary image showing the edges of the input image only. If the input

image has intermediate (gray) values, the operation of the CNN with this

simple Template set is not well defined..

Input: U(t) = static binary image

Initial state: X(0) = arbitrary (reason: Feedback Template = 0)

Output: Y(t) converges toward a binary image showing all edges of the input

image.

Local rules

Regarding the global tasks, we can define local rules for a single cell. The

rules are deduced from the following reflections:

1. A white pixel never turns black.

2. A black pixel turns white if it is surrounded by black pixels

3. A black pixel never turns white, when at least one neighbor cell is white. In

this case, this cell belongs to the edge of the object

Input u(i, j) Output y(i, j) (t = infinity)

1. white pixel white, independent of neighbors

2. black pixel white, if all nearest neighbors are black

3. black pixel black, if at least one nearest neighbor is white

8

Global behavior

The screenshots below show the correct behavior of the Template set.

The input picture above is the input picture of the CNN. It does not change in

this small time period. The four pictures below show the dynamcal behavior of

the entire grid. Starting with the initial state on the left side, the pictures show

the state of the CNN cluster at t = 0.2, t = 0.5 and t = 2.

After two time units (t = 2), the output of the CNN shows the edges of the

input image. The Template set produces the desired output.

Application

The philosophy, interests, and methodologies of CNN researchers are varied. Due to

the potential of the CNN architecture, this platform has attracted people from a variety

of backgrounds and disciplines. Some are exploring practical implementations of

CNN processors, others are using CNN processors to model physical phenomena, and

there are even researchers exploring theoretical mathematical, computational, and

philosophical ideas through CNN processors. This chapter briefly explores what are

9

CNN processors and what they are being used for. Some applications are engineering

related, where some known, understood behavior of CNN processors is exploited to

perform a specific task, and some are scientific, where CNN processors are used to

explore new and different phenomenon. Hopefully this section will illuminate that

CNN processors are versatile platforms that are being used for a variety of

applications.

CNN processors were designed to perform image processing; specifically, the original

application of CNN processors was to perform real-time ultra-high frame-rate

(>10,000 frame/s) processing unachievable by digital processors needed for

applications like particle detection in jet engine fluids and spark-plug detection.

Currently, CNN processors can achieve up to 50,000 frames per second, and for

certain applications such as missile tracking, flash detection, and spark-plug

diagnostics these microprocessors have outperformed a conventional supercomputer.

CNN processors lend themselves to local, low-level, processor intensive operations

and have been used in feature extraction, level and gain adjustments, color constancy

detection, contrast enhancement, deconvolution, image compression, motion

estimation, image encoding, image decoding, image segmentation, orientation

preference maps, pattern learning/recognition, multi-target tracking, image

stabilization, resolution enhancement, image deformations and mapping, image

inpainting, optical flow, contouring, moving object detection, axis of symmetry

detection, and image fusion.

Due to their processing capabilities and flexibility, CNN processors have been used or

has been prototyped for novel field applications such as flame analysis for monitoring

combustion at a waste incinerator, mine-detection using infrared imagery, calorimeter

cluster peak for high energy physics, anomaly detection in potential field maps for

geophysics, laser dot detection, metal inspection for detecting manufacturing defects,

and seismic horizon picking. They have also been used to perform biometric functions

such as fingerprint recognition, vein feature extraction, face tracking, and generating

visual stimuli via emergent patterns to gauge perceptual resonances. CNN processors

have been used for medical and biological research in performing automated

nucleated cell counting for detecting hyperplasia, segment images into anatomically

and pathologically meaningful regions, measuring and quantifying cardiac function,

measure the timing of neurons, detecting brain abnormalities that would lead to

seizers. One potential future application of CNN microprocessors is to combine them

with the DNA microarrays to allow for a near-real time DNA analysis of hundreds of

thousands of different DNA sequences. Currently, the major bottleneck of this DNA

microarray analysis is the amount of time needed to process data in the form of

10

images, and using a CNN microprocessor, researchers have reduced the amount of

time needed to perform this calculation to 7ms.

CNN processors have also been used to generate and analyze patterns and textures.

One motivation was to use CNN processors to understand pattern generation in

natural systems. They were used to generate Turing patterns in order to understand the

situations in which they form, the different types of patterns which can emerge, and

the presence of defects or asymmetries. Also, CNN processors were used to

approximate pattern generation systems that create stationary fronts, spatio-temporal

patterns oscillating in time, hysteresis, memory, and hetergenity. Furthermore, pattern

generation was used to aid high-performance image generation and compression via

real-time generation of stochastic and coarse-grained biological patterns, texture

boundary detection, and pattern and texture recognition and classification. Control

and Actuator Systems

There is an ongoing effort to incorporate, CNN processors into sensory-computing-

actuating machines as part of the emerging field of Cellular Machines. The basic

premise is to create an integrated system that uses CNN processors for the sensory

signal processing and potentially the decision making and control. The reason is that

CNN processors can provide a low power, small size, and eventually low cost

computing and actuating system suited for that type of system. These Cellular

Machine will eventually create a Sensor-Actuator Network (SAN), a type of Mobile

Ad Hoc Networks (MANET) which can be used for military intelligence gathering,

surveillance of inhospitable environments, maintenance of large areas, planetary

exploration, etc.

CNN processors have been proven versatile enough for some control functions. The

have been used as associative memories, optimize function via genetic algorithm,

measuring distances, optimal path finding in a complex, dynamic environment, and

theoretically be used to learn and associate complex stimuli. They have also been used

to create antonymous gaits and low-level motor for robotic nematodes, spiders, and

lamprey gaits using a Central Pattern Generator (CPG). They were able to function

using only feedback from the environment, allowing for a robust, flexible,

biologically inspired robot motor system. CNN-based systems were able to operate in

different environments and still function if some of the processing units were

disabled.

The variety of dynamical behavior seen in CNN processors make them intriguing for

communication systems. Chaotic communications using CNN processors is being

11

researched due their potential low power consumption, robustness and spread

spectrum features. The premise behind chaotic communication is to used a chaotic

signal for the carrier way and to use chaotic phase synchronization to reconstruct the

message. CNN processors can be used on both the transmitter and receiver end to

encode and decode the message. They can also be used for data encryption and

decryption, source authentication through watermarking, detecting of complex

patterns in spectrogram images (sound processing), and transient spectral signals

detection.

CNN processors are neuromorphic processors, meaning that they emulate certain

aspects of biological neural networks. The original CNN processors were based on

mammalian retinas, which consist of a layer of photodetectors connected to several

layers of locally coupled neurons. This makes CNN processors part of an

interdisciplinary research area whose goal is to design systems that leverage

knowledge and ideas from neuroscience and contribute back via real-world validation

of theories. CNN processors have implemented a real-time system that replicates

mammalian retinas, validating that the original CNN architecture modeled the correct

aspects of biological neural networks used to perform. However, CNN processors are

not only limited to verifying biological neural networks associated with vision

processing; they have been used to simulate dynamic activity seen in mammalian

neural networks found in the olfactory bulb and locust antennal lobe, responsible for

pre-processing sensory information to detect differences in repeating patterns.

CNN processors are being used to understand systems that can be modeled using

simple, coupled units, such as living cells, biological networks, physiological systems,

and ecosystems. The CNN architecture captures some of the dynamics often seen in

nature and is simple enough to analyze and conduct experiments. They are also being

for stochastic simulation techniques, which allows scientists to explore spin problems,

population dynamics modes, lattice gas models, percolation, etc. Other simulation

includes heat transfer, mechanical vibrating systems, protein production, Josephson

Transmission Line (JTL), seismic wave propagation, and geothermal structures. The

3D (Three Dimensional) CNN have been used to prove as complex shapes are

emergent phenomena establishing a link between art, dynamical systems and VLSI

technology. CNN processors have been used to research a variety of mathematical

concepts, such as researching non-equilibrium systems, constructing non-linear

systems of arbitrary complexity using a collection of simple, well-understood

dynamic systems, studying emergent chaotic dynamics, generating chaotic signals,

and in general discovering new dynamic behavior. They are often used in researching

systemics, a trandisiplinary, scientific field that researches natural systems. Their goal

12

13

is to develop a conceptual and mathematical framework necessary to analyze, model,

and understand systems, including, but are not limited to, atomic, mechanical,

molecular, chemical, biological, ecological, social and economic systems. Topics

explore are emergences, collective behavior, local activity and its impact on global

behavior, and quantifying the complexity of an approximately spatially and

topologically invariant system. Although another measure of complexity many not

make some people enthusiastic (Seth Lloyd, a professor from Massachusetts Institute

of Technology (MIT), has identified 32 different definitions of complexity), it can be

potentially be mathematically analyze systems such as economic and social systems.

References

[1] L.O. Chuo and L.Yang, "Cellular neural networks: Applications," IEEE Trans.

Circuits Syst., pp. 1273-1290, this issue

[2] J.J Hopfield, "Neural networks and physical system with emergent computational

abilities," proc. Natl. Acad. Sci. USA.

[3] D. Balya, G, Tímar, G. Cserey, and T. Roska, "A New Computational Model for

CNN-Ums and its Computational Complexity", Int’l Workshop on Cellular Neural

Networks and Their Applications, 2004

[4] L. Chua, T. Roska, Cellular Neural Networks and Visual Computing: Foundations

and Applications, 2005.

[5] V. Cimagalli, M. Balsi, "Cellular Neural Networks: A Review", Neural Nets

WIRN Vierti, 1993.

[6] H. Harrer and J.Nossek, "Discrete-Time Cellular Neural Networks", Int'l Journal of

Circuit Theory and Applications, 20:453-467, 1992.

[7] S. Majorana and L. Chua, "A Unified Framework for Multilayer High Order CNN",

Int’l Journal of Circuit Theory and Applications, 26:567-592, 1998.

[8] T. Roska, L. Chua, "The CNN Universal Machine: An Analogic Array Computer",

IEEE Trans. on Circuits and Systems-II, 40(3): 163-172, 1993

## Comments 0

Log in to post a comment