Metaheuristic Algorithms I: Genetic Algorithm

Review of Traditional optimization

Review of the general form of optimization problems

Minimize or Maximize f ( x) Objective Function

subject to
g ( x) = 0 Constraints
x x x

- x denotes the vector (x1, x2, … , xn) decision variables (n)

- g(x) denotes the vector constraints (m)
- x and x denote the range of decision variables
-2-
Metaheuristic Algorithms I: Genetic Algorithm

Traditional optimization

Types of Optimization Approaches for Engineering Problems

⚫ Traditional trial-and-error approaches based on ‘engineering sense’

✓ Traditional approaches cannot guarantee ‘optimal’ or ‘near-optimal’ solutions.

⚫ Mathematical approaches (LP, NLP, DP)

✓ Too ideal (all linear functions)

✓ Not for discrete variables or complex functions

⚫ Meta-heuristic algorithms (GA, SA, TS, ACO, PSO, HS, etc.)

✓ The techniques have been successfully applied to a number of complex

problems.
→ More effective in finding optimal solutions compared to traditional and
mathematical approaches.
-3-
 Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

Kennedy and
Holland Kirkpatrick Eberhart Storn and Geem and
(1975) Dorigo (1992)
Glover (1977) et al. (1983) (1995) Price (1996) Kim (2001)
Genetic Ant Colony
Tabu Search Simulated Particle Differential Harmony
Algorithm Optimization
Annealing Swarm Evolution Search
Optimization

1970s 2001
-4-
 Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

Geem and Nakrani and Yang and Eskandar et Sadollah et al.

Yang (2007) Yang (2010)
Kim (2001) Tovey (2004) Deb (2008) al. (2012) (2012)
Firefly Bat
Harmony Honeybee Cuckoo Water Cycle Mine Blast
Algorithm Algorithm
Search Algorithm Search Algorithm Algorithm

2000s -5- 2013

Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Holland’s original GA (1975) is now known as the Simple

Genetic Algorithm (SGA)

• SGA is inspired by the survive of the fitness in “DARWIN'S

THEORIES”.

• Main mechanisms of SGA is..

– Representations
– Mutations
– Crossovers
– Selection

-6-
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

-7-
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

Phenotype space Genotype space =

Encoding {0,1}L
(representation) 10010001
10010010
010001001
011101001
Decoding
(inverse representation)

-8-
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

Representation Binary strings

Crossover N-point or uniform

Randomly switching the

Mutation
chromosome with a fixed probability

Parent selection Fitness-Proportion

Survivor selection All children replace parents

-9-
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

- 10 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

1. Select parents in the mating pool (in population)

(size of mating pool = population size)

2. Shuffle the mating pool

3. For each pair apply crossover formulation with probability Pc,

otherwise copy parents

4. For each offspring apply mutation (with probability Pm

independently for each chromosome)

5. Replace the whole population with the resulting offspring

- 11 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Choose a random point on the two parents

• Split parents at this crossover point
• Create children by exchanging tails
• Pc is generally in the range of (0.6, 0.9)

- 12 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Each gene independently applies a probability Pm

• Pm is called the mutation rate

- Typically between 1/pop_size and 1/ chromosome_length.
- It might be from 0 to 0.3

• If the selected chromosome in the parent is 1, the value of

chromosome is changed by 0 after mutation mechanism.

- 13 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Main idea: better individuals get higher chance to select

- Chances proportional to fitness
- Method: Roulette wheel technique
✓ Assign to each proportion of individual a part of the
roulette wheel
✓ Spin the wheel n times to select n individuals

1/6 = 17%

A B fitness(A) = 3
C
3/6 = 50% 2/6 = 33%
fitness(B) = 1
fitness(C) = 2
- 14 -
 Metaheuristic Algorithms I: Genetic Algorithm
Process of SGA

- 15 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

• Objective function : Min. f (x1, x2)=x12+x22-12x1-4x2+44

• Search range: 0 ≤ x1 ≤ 63 , 0 ≤ x2 ≤ 15 (integer values)
• Global optimum: x1=6, x2=2, f (6, 2)=4

Global optimum
f(6 , 2) = 4

- 16 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Population size: 10 • Max. no. of function evaluation : 1,000

• Pc: 0.85 • Total no. of individual runs: 100
• Pm: 0.1

String
x1 x2
No.
1 0 0 1 0 0 1 0 0 1 1
2 1 0 1 0 1 1 1 1 1 1
3 0 1 1 1 0 0 1 1 0 1
4 0 1 1 0 0 0 1 1 1 0
5 1 1 0 1 1 0 0 0 0 0
6 0 0 1 0 1 1 1 0 0 0
7 0 1 1 1 0 0 0 1 0 0
8 0 1 0 0 0 1 0 1 1 0
9 1 0 1 0 1 0 0 0 0 0
10 1 0 1 1 - 17 -1 1 0 0 1 1
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

String No. x1 x2 Fitness Prob. Expected count Cumulative

1 9 3 14 0.111 1.109 1.109
2 43 15 1542 0.091 0.913 2.022
3 28 13 609 0.103 1.032 3.054
4 24 14 472 0.105 1.051 4.105
5 54 0 2312 0.081 0.814 4.919
6 11 8 65 0.110 1.103 6.022
7 28 4 492 0.105 1.047 7.069
8 17 6 141 0.109 1.093 8.162
9 42 0 1304 0.094 0.944 9.106
10 47 3 1686 0.089 0.894 10
sum 8637 1 10
best 14

( f ) − f
n n fn
Min. problem :Prob n = n =1 Max. problem :Prob n = T
 
f
T T

  n
n =1  n =1
( f ) − f n
 n =1
n

- 18 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

String No. x1 x2 Fitness Prob. Expected count Cumulative

1 9 3 14 0.111 1.109 1.109
2 43 15 1542 0.091 0.913 2.022
3 28 13 609 0.103 1.032 3.054
4 24 14 472 0.105 1.051 4.105
5 54 0 2312 0.081 0.814 4.919
6 11 8 65 0.110 1.103 6.022
7 28 4 492 0.105 1.047 7.069
8 17 6 141 0.109 1.093 8.162
9 42 0 1304 0.094 0.944 9.106
10 47 3 1686 0.089 0.894 10
sum 8637 1 10
best 14
String No. x1 x2
10 1 0 1 1 1 1 0 0 1 1
1 0 0 1 0 0 1 0 0 1 1
10 1 0 1 1 1 1 0 0 1 1
8 0 1 0 0 0 1 0 1 1 0
1 0 0 1 0 0 1 0 0 1 1
9 1 0 1 0 1 0 0 0 0 0
1 0 0 1 0 0 1 0 0 1 1
4 0 1 1 0 0 0 1 1 1 0
3 0 1 1 1 0 0 1 1 0 1
9 1 0 1 0 - 19 -1 0 0 0 0 0
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

Crossover
String No. x1 x2 Crossover team
point
3 0 1 1 1 0 0 1 1 0 1 A 1
1 0 0 1 0 0 1 0 0 1 1 B 9
10 1 0 1 1 1 1 0 0 1 1 C 7
1 0 0 1 0 0 1 0 0 1 1 D 9
8 0 1 0 0 0 1 0 1 1 0 E 3
10 1 0 1 1 1 1 0 0 1 1 E 3
4 0 1 1 0 0 0 1 1 1 0 D 9
1 0 0 1 0 0 1 0 0 1 1 C 7
9 1 0 1 0 1 0 0 0 0 0 B 9
9 1 0 1 0 1 0 0 0 0 0 A 1
point 1 2 3 4 5 6 7 8 9

String No. x1 x2
3 0 0 1 0 1 0 0 0 0 0
1 0 0 1 0 0 1 0 0 1 0
10 1 0 1 1 1 1 0 0 1 1
1 0 0 1 0 0 1 0 0 1 0
8 0 1 0 1 1 1 0 0 1 1
10 1 0 1 0 0 1 0 1 1 0
4 0 1 1 0 0 0 1 1 1 1
1 0 0 1 0 0 1 0 0 1 1
9 1 0 1 0 1 0 0 0 0 1
9 1 1 1 1 - 20 -0 0 1 1 0 1
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

String No. x1 x2
3 o o
1 o o
10
1
8 o
10
4
1 o
9 o
9 o

String No. x1 x2
3 0 1 1 0 1 0 1 0 0 0
1 0 0 1 0 0 1 0 1 1 1
10 1 0 1 1 1 1 0 0 1 1
1 0 0 1 0 0 1 0 0 1 0
8 0 1 1 1 1 1 0 0 1 1
10 1 0 1 0 0 1 0 1 1 0
4 0 1 1 0 0 0 1 1 1 1
1 0 0 1 0 0 1 1 0 1 1
9 0 0 1 0 1 0 0 0 0 1
9 1 1 1 1 - 21 -0 0 1 1 1 1
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

Parents Children
String
x1 x2 Fitness x1 x2 Fitness
No.
1 9 3 14 26 8 440
2 43 15 1542 9 7 38
3 28 13 609 47 3 1686
4 24 14 472 9 2 13
5 54 0 2312 31 3 630
6 11 8 65 41 6 1245
7 28 4 492 24 15 497
8 17 6 141 9 11 94
9 42 0 1304 10 1 21
10 47 3 1686 60 15 3089
sum 8637 7753
best 14 13

- 22 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

4.00 (Global optimum)

- 23 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Has been subject of many (early) studies

• still often used as benchmark for novel GAs
• Shows many shortcomings, e.g.
• Mutation & crossover can be applied only for binary &
integer representations
• Representation is too restrictive
• Selection mechanism is sensitive depending on the mating
partner.
• Reproduction cycle can be improved with survivor selection.

- 24 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Performance with 1-Point Crossover depends on the order

that variables occur in the representation
• More likely to keep together genes that are near each
other
• Can never keep together genes from opposite ends of
chromosome
• Can be exploited if we know about the structure of our
problem, but this is not usually the case

- 25 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

• Choose n random crossover points

• Split along those points
• Glue the parts, alternating between parents

- 26 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

• Assign 'heads' to one parent, 'tails' to the other

• Flip a coin for each gene of the first child
• Make an inverse copy of the gene for the second child

- 27 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

• Many problems occur as real valued problems in the real-

world problem, e.g. continuous parameter optimization f :
n→
• In case of the real-world problem, the traditional
representations form is difficult to apply.

- 28 -
Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

⚫ Keep improving the solution as iteration continues

⚫ Avoid being trapped in local optimum solution

⚫ In addition, better algorithms should have faster

conversion

- 29 -
Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

⚫ Global Search, Exploration, Diversification

⚫ Local Search, Exploitation, Intensification

- 30 -
Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

• Exploration (Global search): Discovering promising areas in

the search space, (i.e., gaining information on the problem)

• Exploitation (Local search): Optimizing within a promising

area, (i.e., using information on the optimization processing
results)
• There is co-operation AND competition between two search
characteristic
- 31 -
Metaheuristic Algorithms I: Genetic Algorithm
Genetic Algorithm

✓ Crossover is Global search, it makes a big jump to an area

somewhere “in between” two (parent) areas
✓ Mutation is Local search, it creates random small variance,
staying near (in the area of) the parent

• Only crossover can combine information from two parents

• Only mutation can introduce new information (alleles)

• Crossover does not change the frequency of the mutation

• To hit the optimum solution often need a ‘lucky’ mutation

- 32 -
Metaheuristic Algorithms I: Genetic Algorithm

Genetic Algorithm

• Decade long debate: which one is better / necessary / main-

operator

• Answer (at least, rather wide agreement):

✓ it depends on the problem, but
✓ in general, it is good to have both
✓ both have another role

- 33 -
Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

⚫ Diversification – makes sure the algorithm explores the

search space globally

⚫ Intensification – intends to search locally and more

intensively

⚫ A fine balance between these two components is very

important to the overall efficiency and performance of the
algorithm

⚫ Furthermore, needs Survival of the Fittest

- 34 -
Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

Objective : Maximization of Function Value

Searching Space Final Solution Using Algorithm 1
Objective function value

Objective function value

Good Exploration Capability
Bad Exploitation Capability

Variable set Variable set

Final Solution Using Algorithm 2 Final Solution Using Algorithm 3

Bad Exploration Capability
Objective function value
Objective function value

Good Exploitation Capability Good Exploration Capability

Good Exploitation Capability

Variable set - 35 - Variable set

 Metaheuristic Algorithms I: Genetic Algorithm
Exploration Vs. Exploitation

Exploration vs. Exploitation

Heriot-Watt Universtiy
- 36 -
http://www.streamsim.com/sites/default/files/explorationvsexploitation.png
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

• Might want to put in EA as part of larger system

• Might be looking to improve on existing techniques

• Might be looking to improve EA search for good solutions (i.e.,

exploration and exploitation)

- 37 -
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

- 38 -
Metaheuristic Algorithms I: Genetic Algorithm
Hybrid Evolutionary Algorithms

• The combination of Evolutionary Algorithms with

Local Search Operators that work within the EA
loop has been termed “Memetic Algorithms”

• Memetic Algorithms have been shown faster and

more accurate than EAs on some problems, and are
the “state of the art” on many problems

- 39 -
Metaheuristic Algorithms I: Genetic Algorithm
Hybrid Evolutionary Algorithms

- 40 -
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

• Bramlette ran experiments with limited time scale

and suggested holding a n-way tournament amongst
randomly created solutions to pick initial population
(It dose not take the same population even we know the
best population size)
• Multi-Start Local Search is another option: pick
popsize points at random to climb from
• Constructive Heuristics often exist

- 41 -
Metaheuristic Algorithms I: Genetic Algorithm
Hybrid Evolutionary Algorithms

• Another common approach would be to initialize population

with solutions already known, or found by another technique
(beware, performance may appear to drop at first if local
optima on different landscapes do not coincide)
• Surry & Radcliffe (1994) studied ways of “Preventing”
population with solutions gained from previous experiments
or other algorithms/heuristics
• It found mean performance increased as population was
biased towards known solutions,
• but best performance came from more random solutions

- 42 -
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

• It is sometimes possible to incorporate problem or

instance specific knowledge within crossover or
mutation operators
• Yadav et al. (2017) developed a new meta-heuristic
technique combined Particle Swarm Optimization and
Gravitational Search and methodology for handling
constraints approach.
• Sadollah et al. (2015) have developed a hybrid genetic
algorithm (HyGA) that combines genetic algorithm (GA)
and rule induction algorithms. It confirmed that the
proposed HyGA is superior to other algorithms in terms of
computation speed and efficiency.
• Many other examples in literature
- 43 -
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

• Can be viewed as a sort of “lifetime learning”

• Lots of researches done using EAs to improve the structure
of Artificial Neural Networks to optimize learn weight
factors
• Recently, the hyper-heuristic algorithm which combined
various metaheuristic and artificial intelligent approach (e.g.,
Machine learning, Artificial Neural Network) developed to
find suitable search operator among the various optimization
algorithm.

- 44 -
Metaheuristic Algorithms I: Genetic Algorithm

Hybrid Evolutionary Algorithms

• It is common practice to hybridise EA’s when using

them in a real world context.
• This may involve the use of operators from other
algorithms which have already been used on the
problem, or the incorporation of domain-specific
knowledge.
• Hybrid algorithms have been shown faster and
more accurate than each optimization algorithm on
some problems, and are the “state of the art” on
many problems.
- 45 -
 Metaheuristic Algorithms I: Genetic Algorithm

• Homework Assignment #1
▣ Due date :
▣ 1. Program a Simple Genetic Algorithm (SGA) to solve <Min f(x) = x12 + x22 -
12x1 - 4x2 + 44> with integer x1 and x2 varying from 0 to 63 for x1 and 0 to 15
for x2.
First, use Population size = 10, Pc = 0.8, Pm = 0.1, Maximum number of
function evaluations = 1,000, Total number of individual runs = 10

1-1. Try various values of crossover rate (0.6, 0.8, 0.9), mutation rate (0.05,
0.1, 0.2, 0.4) in the above problem. Run 10 times for each mutation rate and
show the average f (x) for each mutation rate.

1-2. Also, try the n-point crossover approach and compare with the original
version of Genetic algorithm.

Using already existing software is not recommended. Code the algorithm

yourself (Any language is possible to use but please presents explanation in
your code about). The assignment should
- 46 - include the analysis report and Code.