# Neural Network

AI and Robotics

Oct 19, 2013 (4 years and 8 months ago)

75 views

Neural Network

Ming
-
Feng Yeh (

)

Department of Electrical Engineering

Lunghwa University of Science and Technology

E
-
mail: mfyeh@mail.lhu.edu.tw

Office: F412
-
III Tel: #5518

Ming
-
Feng Yeh

2

COURSE OBJECTIVE

This course gives an introduction to
basic
neural network architectures

and
learning
rules
.

Emphasis is placed on the mathematical
analysis of these networks, on methods of
training them and on their application to
practical engineering problems

in such
areas as
pattern recognition
,
signal
processing

and
control systems
.

Ming
-
Feng Yeh

3

SYLLABUS

Textbook
:
Hagan, Demuth, Beale
,

Neural Network Design
,

PWS Publishing Company

Midterm Exam
: 30%

Final Exam
: 30%

Projects:

40%

Ming
-
Feng Yeh

4

CONTENTS

Ch 1.

Introduction

Ch 2.

Neuron Model & Neural Architecture

Ch 3&4.

Perceptron
(

)

Learning Rule

Ch 7.

Supervised
(

)

Hebbian Learning

Ch 10.

Widrow
-
Hoff

Learning

Ch 11&12.

Back
-
propagation
(

)

Ch 13.

Associative
(

)

Learning

Ch 14.

Competitive
(

)

Networks

Ch 15.

Grossberg

Networks

Ch 16.

(

)

Theory

Ch 18.

Hopfield

Network

Ming
-
Feng Yeh

5

Information

Review

Ch 5

Signal and Weight Vector Spaces

Ch 6

Linear Transformations for

Neural Networks

Ming
-
Feng Yeh

6

CHAPTER 1

Introduction

Ming
-
Feng Yeh

7

Objectives

As you read these words you are using a
complex
biological neural network
. You
have a highly interconnected set of
10
11

neurons

to facilitate your
,
breathing
,
motion

and
thinking
.

In the
artificial neural network
, the neurons
are
not

biological. They are extremely simple
abstractions of biological neurons, realized as
elements

in a program or perhaps as
circuits

Ming
-
Feng Yeh

8

History
-
1

Pre
-
1940: von Hemholtz, Mach & Pavlov

General theories of learning, vision, conditioning

No specific mathematical models of neuron operation

1940s: Hebb, McCulloch & Pitts

Mechanism for learning in biological neurons (Hebb)

Neural
-
like networks

can compute any arithmetic or logical
function (McCulloch & Pitts)

1950s: Rosenblatt, Widrow & Hoff

First practical networks and learning rules:
the perception
network

and associated learning rule (Rosenblatt) &
Widrow
-
Hoff learning rule

Can not successfully modify their learning rules to train the
more complex networks.

Ming
-
Feng Yeh

9

History
-
2

1960s: Minsky & Papert

Demonstrated limitations of existing neural networks

Neural network research was largely suspended

1970s: Kohonen, Anderson & Grossberg

Kohonen and Anderson independently and separately
developed neural networks that could as
memories

Self
-
organizing networks

(Grossberg)

1980s: Hopfield, Rumelhart & McClelland

The use of statistical mechanics to explain the operation of
recurrent network: an
associative memory

(Hopfield)

Backpropagation algorithm

(Rumelhart & McClelland)

Ming
-
Feng Yeh

10

Applications

The applications are expanding
because neural networks are good at
solving problems, not just in
engineering
,
science

and
mathematics
, but in
medicine
,
,
finance

and
literature
as
well.

Ming
-
Feng Yeh

11

Biological Inspiration

Human brain

consists of a large number
10
11
) of highly interconnected
10
4

connections per
element) called
neurons (

)
.

Three principle components are the
dendrites
, the
cell body

and the
axon
.

The point of contact is called a
synapse
.

Ming
-
Feng Yeh

12

Biological Neurons

Dendrites
(

⤺)
carry

electrical
into

the cell body

Dendrites

Cell Body

Cell Body
(

⤺)
sums

and
thresholds

these incoming
signals

Axon
(

⤺)
carry

the signal
from the cell body
out

to other
neurons

Axon

Synapse
(

contact

between an axon of one cell
and a dendrites of another cell

Synapse

Soma

Ming
-
Feng Yeh

13

Neural Networks

Neural Networks
: a promising new
generation of
information processing systems
,
usually operate in parallel
, that demonstrate
the ability to
learn
,
recall
, and
generalize

from training patterns or data.

Basic models
,
learning rules
, and
distributed
representations

of neural networks will be
discussed.

Ming
-
Feng Yeh

14

Artificial neural network

，是指

，包括

，它從

，並輸出其結果到

Ming
-
Feng Yeh

15

Fuzzy Logic

Fuzzy set theory

was first proposed by

A mathematical way to represent
vagueness

in
linguistics

A generalization of
classical set theory

Ming
-
Feng Yeh

16

Fuzzy Systems v.s.
Neural Networks

Fuzzy logic

is based on the way the
brain

deals with
inexact information
.

Neural networks

are modeled after the
physical architecture

of the
brain
.

Fuzzy systems and neural networks are both
numerical model
-
free estimator

and
dynamical systems
.

They share the common ability to improve the
intelligence

of systems working in an
uncertain
,
imprecise

and
noisy environment
.

Ming
-
Feng Yeh

17

Machine Intelligence

Neural networks

provide fuzzy
systems with
learning ability
.

Fuzzy systems

provide neural
networks with a structure framework
with high
-
level
fuzzy

IF
-
THEN rule

thinking

and
reasoning
.

Ming
-
Feng Yeh

18

Fuzzy Neural

Integrated System

Neural fuzzy systems
: use of neural
networks as tools in
fuzzy models
.

Fuzzy neural networks
: fuzzification of
conventional neural
network models
.

Fuzzy
-
neural hybrid systems
:
incorporation of fuzzy logic technology
and neural networks into
hybrid
systems
.

Ming
-
Feng Yeh

19

Soft
/
Hard Computing

Hard computing

whose prime desiderata are
precision
,
certainty
, and
rigor
.

Soft computing

is tolerant of
imprecision
,
uncertainty
, and
partial truth
.

The primary aim of
soft computing

is to exploit
such tolerance to achieve
tractability
,
robustness
,
a
high level

of
machine intelligence
, and a
low
cost

in practical applications.

Fuzzy logic
,
neural networks
(including
CMAC
),
probabilistic reasoning

(
genetic

algorithm,
evolutionary

programming, and
chaotic

systems)

Ming
-
Feng Yeh

20

Soft Computing

Methodology

Strength

Neural network

Fuzzy set theory

Knowledge representation
via fuzzy if
-
then rule

Genetic algorithm
and simulated
annealing

Systematic random search

Conventional AI

Symbolic manipulation

Ming
-
Feng Yeh

21

Computational Intelligence

Fuzzy logic
,
neural network
,
genetic algorithm
,
and
evolutionary programming

are also
considered the building blocks of
computational intelligence
.
(James Bezdek)

Computational intelligence is
low
-
level
cognition

in the style of human brain and is
contrast to conventional (symbolic)
artificial
intelligence (AI)
.