IEEM
5119
Genetic Algorithms
and Other Nature

Inspired
Metaheuristic Algorithms
Maw

Sheng Chern
http://chern.ie.nthu.edu.tw/gen/gen.htm
This course is tended to introduce the stochastic and other
extended local search methods (genetic algorithms, simulated
annealing, tabu search, ant algorithms, particle swarm
optimization, bee algorithm, etc.) and their applications to
difficult

to

solve optimization problems in industrial
engineering and manufacturing systems design.
Genetic algorithms
is based on the concepts from population
genetics and evolution theory. The algorithm are constructed
to optimize fitness of a population of elements through
crossover (recombination) and mutation (perturbation)
operations on their genes.
1
Simulated annealing
is based on an analog of cooling the
material in a heat bath
–
a process known as annealing. A solid
material is heated in a heat bath until it melts, then cooling it
down slowly until it crystallizes into a solid state (low

energy
state). The atoms in the material have high energies at high
temperatures and more freedom to arrange themselves. As the
temperature is reduced, the atom energies decrease. The structural
properties of the solid depend on the rate of cooling. From the
point of view of search methods for optimization problems,
simulated annealing is a stochastic local search method. It always
accepts a selected better

cost local solution and it may also accept
a worse

cost local solution with a probability which is gradually
decreased in the course of algorithm’s execution.
1
Tabu search
is a deterministic iterative improvement local
search method with a possibility to accept worse

cost local
solution in order to escape from local optimum. The set of legal
local solutions are restricted by a tabu list which is designed to
present from going back to the recently visited solutions. The set
of solutions in tabe list are not accepted in the next iteration.
The Ant algorithm
is based on the observation of real ant’s
behavior. Ants can coordinate their activities via stigmergy, a
way of indirect communication through the modification of the
environment. The main idea of ant algorithm is to use self

organizing principles of artificial agents which collaborate to
solve the problems.
1
Particle Swarm Optimization
(PSO) algorithm
is a population

based stochastic optimization method proposed by James
Kennedy and R. C. Eberhart in 1995. It is motivated by social
behavior of organisms such as bird flocking and fish schooling.
In the PSO algorithm, the potential solutions called
particles
, are
flown in the problem hyperspace. Change of position of a
particle is called
velocity
. The particle changes their position
with time. During flight, particle’s velocity is stochastically
accelerated toward its previous best position and toward a
neighborhood best solution. POS has been successfully applied
to solve various optimization problems, artificial neural network
training, fuzzy system control, and others
.
1
The
Bees Algorithm
is a new population

based search algorithm,
first developed in 2005 by
Pham DT etc. [1]
and
Karaboga.D [2]
independently
. The algorithm mimics the food foraging
behaviour of swarms of honey bees. In its basic version, the
algorithm performs a kind of neighbourhood search combined
with random search and can be used for optimization problems.
5.
Course homepage
http://chern.ie.nthu.edu.tw/gen/gen.htm
4. Word

Wide

Web
3
.
Lecturer Slides
1.
Geng, Mitsuo and Cheng, Runwei,
Genetic Algorithms & Engineering
Design
, John Wiley & Sons, New York, 1997.
2.
Geng, Mitsuo and Cheng, Runwei,
Genetic Algorithms & Engineering
Optimization
, John Wiley & Sons, New York, 2000.
3.
Mitsuo Gen, Runwei Cheng, Lin Lin,
,
Network Models and
Optimization
[electronic resource] :
Multiobjective Genetic Algorithm
Approach,
London :
Springer

Verlag,
2008.
Textbook & References:
References:
Genetic Algorithms:
# Michalewicz , Zbigniew,
Genetic Algorithms + Data Structures = Evolution
Programs
, 1994, Third Revised and Extended Edition, Springer, New York, 1999.
Goldberg, David E.,
Genetic Algorithms: in Search, Optimization & Machine
Learning
, Addison

Wesley Publishing Company, Inc. New York, 1989.
Scott Robert Ladd.,
Genetic Algorithms in C++
, M&T Books, New York, 1996.
Bagchi, Tapan P.,
Multiobjective Scheduling by Genetic Algorithms
, Kluwer
Academic Publishers, Boston,
Golderberg, David E.,
Genetic Algorithms: in Search, Optimization & Machine
Learning
, Addison

Wesley Publishing Company, Inc., New York, 1989.
Mitsuo Gen, Runwei Cheng, Lin Lin,
,
Network Models and
Optimization
[electronic resource] :
Multiobjective Genetic Algorithm Approach,
London :
Springer

Verlag,
2008. .
Bauer, J.,
Genetic Algorithms and Investment Strategies,
John Wiley & Sons, New
York, 1994.
Michell, M.,
An Introduction to Genetic Algorithms
, MIT Press, Cambridge, MA,
1996.
Duc Truong Pham and Dervis Karaboga,
Intelligent Algorithms, tabu search,
simulated annealing and neural networks
, Springer, New York, 1998.
Charles, L. Karr and L. Michael Freeman (ed.),
Industrial Applications of Genetic
Algorithms
, CRC Press, New York, 1998.
Man, K. F., K.S. Tang and S. Kwong,
Genetic Algorithms
, Springer, New York,
1999. (Refer a genetic game in this book)
Erick Cantu

Paz , Efficient and Accurate Parallel Genetic Algorithms (
Genetic
Algorithms and Evolutionary Computation
Volume 1)
Randy L. Haupt, Sue Ellen Haupt, Practical Genetic Algorithms
George Lawton,
A Practical Guide to Genetic Algorithms in C++
/Book and
Disk
Andrzej Osyczka,
Evolution Algorithms for single and Multicriteria Design
Optimization
, Physica

Verlag, Heidelberg, 2002.
Thomas Bäck,
Evolution Algorithms in Theory and Practice: Evolution
Strategies, Evolution Programming, Genetic Algorithms,
Oxford University
Press, Oxford, 1996.
Colin R. Reeves, Jonathan E. Rowe,
Evolution Algorithms: Principles and
Perspectives,
A Guide to GA Theory
, Kluwer Academic Publishers, Boston,
2003.
David Corne, Marco Dorigo, and Fred Glover (ed.),
New Ideas in Optimization
,
The McGraw

Hill Companies, NY, 1999.
1
Ant Algorithms:
# Marco Dorigo, Thomas Stützle,
Ant colony optimization
, Cambridge, Mass.,
MIT Press, c2004
Eric Bonabeau, Marco Dorigo, and Guy Theraulaz,
Swarm Intelligence: From
Natural to Artificial Systems,
Oxford University Press, NY, 1999.
David Corne, Marco Dorigo, and Fred Glover (ed.),
New Ideas in Optimization
,
The McGraw

Hill Companies, NY, 1999.
James Kennedy, Russell C. Eberhart, and Yuhui Shi,
Swarm Intelligence,
Morgan Kaufmann Publishers, San Francisco, 2001.
Simulated Annealing:
Emile Aarts and Jan Korst,
Simulate Annealing and Boltzmann Machines,
John Wiley & Sons, Inc. NY,
1989
.
1
General Stochastic Search Methods:
Emile H. L. Aarts and Jan Karel Lenstra,
Local Search in Combinatorial
Optimization
, John Wiley and Sons, NY, 1997.
Holger H. Hoos and Thomas Stűtzle,
Stochastic Local Search: Fundamentals
and Applications
, Morgan Kaufman Publishers, NY, 2005.
Tabu Search:
# F. Glover and M.Laguna,
Tabu Search
, Kluwer Academic Publishers, Boston,
MA, U.S.A, 1997.
# R.H.J.M. Otten and L.P.P.P. van Ginneken, The Annealing Algorithm, Kluwer
Academic Publishers, Boston, MA, U.S.A, 1989.
1
Randomized Algorithms
Rajeev Motwani and Prabhakar Raghavan,
Randomized Alogorithms
,
Cambridge University Press, NY, 1995.
J. C. Spall,
Introduction to Stochastic Search and Optimization: Solving
Estimation, Simulation, and Control,
John Wiley and Sons Inc., NY, 2003.
George S. Tarasenko,
Stochastic Optimization in the Soviet Union Random
Search Algorithms
, Delphic Associates, Inc., VA, 1985.
Colin R. Reeves (ed.),
Mordern Heuristic Techniques for Combinatorial
Problems
, John Wiley & Sons, Inc. NY, 1993.
Sadiq M. Sait and Habib Youssef,
Iterative Computer Algorithms with
Application to Engineering: Solving Combinatorial Optimization Problems,
IEEE Computer Society, LA, 1999.
Xin

She Yang,
Nature

Inspired Metaheuristic Algorithms, Luniver Press 2008.
Raymond Chiong (ed.),
Nature

inspired algorithms
for optimisation
[electronic resource], Berlin, Heidelberg :
Springer Berlin Heidelberg,
2009.
Contents
1
Introduction
2
Genetic Algorithms
3
Non

linear Programming Problems
4
Foundation of Genetic Algorithms
5
Metaheuristic Algorithms
6.
Simulated Annealing
7.
Tabu Search
8.
Ant Algorithms
9.
Particle Swarm Optimization
10.
Bee Algorithms
11.
The Other Bio

Inspired Metaheuristic Algorithms
12.
Traveling Salesman Problems
13.
Knapsack Problems
14.
Set Covering Problems
15.
Minimum Spanning Tree Problems
16.
Flow Shop Scheduling Problems
17.
Job Shop and Open Shop Scheduling Problems
18.
Bin Packing Problems
19.
Data Mining
20.
Quadratic Assignment Problems
21.
Reliability Optimization Problems
22.
Layout and Location Problems
1. The Computational complexity
We do not expect
NP

hard (and
NP

complete) problems to be
solved in polynomial steps.
Ambati et al. (1991): An evolution algorithm that can achieve
heuristic solutions 25% worse than the expected optimal solution on
random
Traveling Salesman Problem
in O(
N
log
N
) time.
Fogel (1993): An evolution algorithm that can achieve heuristic
solutions 10% worse than the expected optimal solution on random
Traveling Salesman Problem
in O(
N
2
) time.
The length of the minimal tour is 21134 km. [Aarts 1989]
1
The sizes of the Traveling Salesman Problem
100,000 = 10
5
people in a stadium.
5,500,000,000 = 5.5
10
9
people on earth.
1,000,000,000,000,000,000,000 = 10
21
liters of water on the earth.
10
10
years = 3
10
17
seconds =
The age of the universe
# of cities
n
possible solutions (
n

1)!
= # of cyclic permutations
10
181,000
20
10,000,000,000,000,000
= 10
16
50
100,000,000,000,000,000,000,000,000,000,000,000,000,000,
000,000,000,000,000,000,000
= 10
62
( 179 digits)
# of possible solutions =
The shortest roundtrip through
120
German cities. The length of this tour is
6942
km.
0!
= 1
1!
= 1
2!
= 2
3!
= 6
4!
= 24
5!
= 120
6!
= 720
7!
= 5,040
8!
= 40, 320
9!
= 962,880
10!
= 3, 628,800
11!
=
39
,
916
,
800
12!
=
479
,
001
,
600
13!
=
6
,
227
,
020
,
800
14!
=
87
,
178
,
291
,
200
15!
=
1
,
307
,
674
,
368
,
000
16!
=
20
,
922
,
789
,
888
,
000
17!
=
355
,
687
,
428
,
096
,
000
18!
= 6,402,373,705,728,000
19!
= 121,645,100,408,832,000
20!
= 2,432,902,008,176,640,000
n
7
12
18
32
59
81
105
132
228
265
n
!
位數
⠱0
進位
)
4
9
ㄶ
㌶
㠱
ㄲ1
ㄶ1
㈲2
㐴4
㔲5
n
284
304
367
389
435
483
508
697
726
944
n
!
位數
⠱0
進位
)
㔶5
㘲6
㜸7
㠴8
㤶9
㠹
1156
ㄶ㠱
ㄷ㘴
㈴
number of digits
9
Algorithm
–
Problem Solving
2. Search Spaces (Solution Space)
How to do efficient search in the
state space
?
1
1. To obtain a good initial state.
2. Goal identification (identify the optimal condition).
3. Control the search process.
e.g.
Karash

Kuhn

Tucker condition
For some problems, we are not able to identify the goal state.
State Space Representation:
How to search in the
state space?
( a partial solution or a
complete solution )
( the desired solution )
Simplex Method, . . . , Tabu Search, Simulated Annealing, . . .
9
Evolution Process
.
. . .
Initial
Populations
1st
Generation
2nd
Generation
n

th
Generation
.
. . .
Genetic Algorithm, Ant Algorithm, Particle Swarm Optimization,
Bee Algorithm, Firefly Algorithm, . . .
Example 1: Simplex algorithm for linear
program
Goal identification:
primal feasible & dual feasible condition
Goal identification:
1
. The goal state can be identified. (
e.g.
linear program, convex program )
2
. The goal state cannot be identified. (
e.g.
traveling salesman problem, quadratic
assignment problem, knapsack problem, scheduling problems, . . . )
Control the search process:
moving to improved adjacent basic solution
Example 2: Steepest descent algorithm for convex program
Goal identification:
Karash

Kuhn

Tucker condition
Control the search process:
moving to the steepest descent direction
( minimization problem)
Example 4: Ant algorithm for the Traveling Salesman Problem
Goal identification:
The optimal condition
can not be identified efficiently
.
Using heuristic rules to identify approximate solution
.
Control the search process:
pheromone & state transition probability
Example
3
: Genetic algorithm for the Traveling Salesman Problem
Goal identification:
The optimal condition
can not be identified efficiently
.
Using heuristic rules to identify approximate solution.
Control the search process:
Crossover, mutation and selection operations.
1
A state space of a
Traveling Salesman Problem
with
n
= 4.
1
3. Search Methods
1
Exact search methods:
Advantage: To produce an optimal solution and It is able to
detect that a given problem has no feasible solution.
Disadvantage: It is time

consuming. For example, it is infeasible
for real

time problems.
Local search methods:
Advantage: It is time

efficient and easy to write a program.
Disadvantage: It may not produce an optimal solution and is not
able to detect that a given problem has no feasible
solution.
Traversal Search, Backtracking Algorithm, Branch and Bound Algorithm,
Dynamic Programming Method, etc.
Traditional Local search
does not provide a mechanism for the search to escape
from a local optimum. The goal of local search is find a solution which is as close
as possible to the optimum.
1
neighborhood search, simulated annealing
population

based search
I
t makes use the information of a set of solutions.
e.g. genetic algorithms, ant algorithm, . . .
Refinement:
The best solution comes from a process of repeatedly refining and
inventing alternative solutions
.
1
A
J
D
E
F
K
H
I
G
C
B
A
(A J D E F K H I G C B) is a complete solution.
Constructive Search Methods:
To generate a complete solution by iteratively extending partial
solutions.
1
Perturbation Search Methods:
For a complete solution, we can easily change it into new complete
solution by modifying one or more solution components.
For example, in TSP a complete solution (ABCD) is changes into a
new solution (ADCB) by interchange the positions of B and D.
( neighborhood search methods, mutation operations i.e. )
( The Liberty Times, July 27, 2005 )
1
For
a set of complete solutions
, we can easily change them into
new
complete solutions
by modifying one or more solution components
among the solutions. ( crossover and mutation operations in genetic
algorithms. etc. )
Deterministic Algorithms:
In each search step, it progresses
toward the complete solution by making deterministic decision.
e.g. Simplex method, Quasi

Newton algorithms, tabu search and
many other conventional algorithms.
Deterministic algorithm will produce the same solution for a given
problem instance. Even for the same instance, the stochastic
algorithm usually product distinct solutions at each run.
Ackley’s function
1
4. Stochastic Algorithms:
It make a random decision at each search step. e.g. Monte Carlo
algorithms, simulated annealing, genetic algorithms, ant algorithms,
etc.
There are two cases.
(1) The available information
–
the objective function to be
optimized
–
may be considered possible erroneous or corrupted by
random noise.
(2) For the case with perfect information, we may introduce a
random element to guide us when searching for the optimum
solution.
1
[ Hoos 2005]
1
. They are efficient for the practical uses.
2
. They are simple to implement. For many applications, stochastic
algorithm is the simplest algorithm available, or fastest, or both.
3
. They are every general and can be implemented for a wide class of
optimization. For example, no differential function of real valued
parameters is required. It need not be expressible in any particular
constraint language.
4
. They can run in parallel. The quality of solutions may be improved
time by time.
Why stochastic algorithms?
trade

off
computing time
sl畴渠煵qli瑹
Transition probabilities for a deterministic algorithm:
Minimize
f
(
x
)
subject to
x
S
f
(
x
k
) = 2
Configuration Graph
Deterministic algorithm will produce the same solution for a given
problem instance.
Transition probabilities for a stochastic algorithm:
Remark: Transition probabilities may be dependent on the number
of iterations.
Even for the same instance, the stochastic algorithm usually product
distinct solutions at each run.
1
T
= the number of iterations.
There are two ways to avoid getting trapped in a local optimum.
1.
Accommodate nongreedy search move. It is allowed to move to
a neighborhood state with a worse function value. ( Tabu search,
simulated annealing, . . . )
2.
To increase the number of edges in the configuration graph.
However, the denser the configuration graph is, the more
inefficient search step will be.
To enlarge the neighborhood for each state.
21, 15,
36
, 28, 32, 18, 84, 57, 72, 50
21
15
18
32
28
≤
36
≤
84
72
57
50
21, 15, 18, 28, 32
Get one friend to
sort the first half.
84, 57, 72, 50
Get one friend to
sort the first half.
21, 15, 18, 28, 32,
36
, 84, 57, 72, 50
Quicksort
Pick a number as pivot element
.
5. Stochastic Quicksort Algorithm:
15, 18, 21, 28, 32
36
Glue pieces together.
15,18,21,28,32,
36
,50,57,72,84
50
,
57
,
72
,
84
15 18 21 28 32 36 50 57 72 84
1
Deterministic Quick Sort Algorithm:
The pivot element at each step is selected by using deterministic
rules.
Stochastic Quick Sort Algorithm:
The pivot element at each step is selected randomly.
Las Vegas Algorithm
Theorem 5.1 [Motwani 1995]:
The expected number of comparisons in an execution of Stochastic
Quicksort algorithm is O(
n
log
n
) where
n
elements are to be sorted.
1
6. A Stochastic Algorithm for Min

Cut Problem:
For a multigraph
G
, a cut is a set of edges whose removal results in
G
being broken into two or more components. A
min

cut
is a cut with
minimum cardinality.
{
a
,
e
,
g
}, {
f
,
g
}, {
a
,
b
,
d
}, {
c
,
d
}, {
a
,
b
,
e
,
g
} are cuts of
G
.
{
f
,
g
}, {
c
,
d
} are min

cuts of
G
.
Multigraph
G
1
Contracting an edge:
The effect of contracting edge
e
.
Important Observation:
An edge contraction does not reduce the
min

cut size of
G
.
A stochastic min

cut algorithm:
1.
Pick an edge uniformly at random and merge its two vertices.
With each contraction, the number of vertices of
G
decreased by one.
2. It continues the contraction process until only two vertices remain. Then, the
set of edges between these two vertices is a cut of
G
.
Monte

Carlo algorithm
1
select
e
select
a
select
g
{
c
,
d
} is a cut.
Theorem 6.1 [Motwani 1995]:
The probability of discovering a particular min

cut is larger than 2/
n
2
where
n
is
the number of nodes.
Theorem 6.2 [Motwani 1995]:
If we run the algorithm
n
2
/2 times, making independent random choices each time,
then the probability that a min

cut is not found in any of the
n
2
/2 attempts is at
most
< 1/
e
= 1/3.1416.
Comments 0
Log in to post a comment