University of Zagreb, Croatia

Lectures 2
Heuristic Optimization Methods:

Combinatorial optimization; complexity

theory; when and why to use heuristics

Slides prepared by Nina Skorin-Kapov

Academic year 2020/2021

Graduate Studies Programme
 Combinatorial optimization
University of Zagreb, Croatia

 Combinatorial optimization problems:

 Discrete decision variables
 Finite search space
 Objective function can be anything ((non)linear,
nonanalytic, …)

 Many real-world problems fall into this category

Department of Telecommunications 2
 Examples of classical combinatorial
optimization problems
University of Zagreb, Croatia

 Travelling salesman problem (TSP)

 Satisfiability problem (SAT)
 Knapsack problem
 Graph coloring (GC)
 Steiner tree problem (STP)
 Bin packing (BP)

Department of Telecommunications 3
 Traveling Salesman Problem (TSP)
University of Zagreb, Croatia

 Given n cities and their mutual distances, find

the shortest tour which visits each city once,
and only once (Hamiltonian cycle), and returns
to the original starting point.
 (n-1)! possible tours
 (n-1)!/2 possible tours if the orientation doesn’t matter

A 2 B
Some possible tours:
A-B-C-D, distance=9
3 4 1 3
A-B-D-C, distance=10
C D
2
Department of Telecommunications 4
 Satisfiability problem (SAT)
University of Zagreb, Croatia

 SAT: given a Boolean expression, can the

variables be assigned in such a way as to
make the formula evaluate to TRUE
 Decision problem
 Example SAT:
x1  ( x1  x2 )  ( x1  x3 )
 The Maximum Satisfiability problem (MAX-
SAT) is finding an assignment which gives the
maximum number of clauses of a given
Boolean expression that can be satisfied.
 Optimization problem
Department of Telecommunications 5
 Knapsack problem
University of Zagreb, Croatia

 Given are n objects and a knapsack of size C

units.
 Each object has value vi and size ci.
 Objective: Find the subset of objects which fits
in the knapsack and have the maximal total
value
v2=6 c2=5
C=9
v1=8 c1=9 v3=3 c3=3
v2=6 c2=5
v4=2 c4=1
v3=3 c3=3
v4=2 c4=1 Total value: v2+v3+v4=11
Total capacity: c2+c3+c4=9
Department of Telecommunications 6
 Graph coloring (GC)
University of Zagreb, Croatia

 Given a graph, color its nodes with the

minimum number of colors such that no two
adjacent nodes share the same color

5 2

colors = 3
4 3

Department of Telecommunications 7
 Minimum Steiner Tree Problem
University of Zagreb, Croatia

• Steiner tree: Given a graph G=(V,E) and a subset of

nodes D in V, a Steiner tree is a connected and acyclic
subgraph of G which spans all nodes in D
• A minimum Steiner tree is such a subgraph of minimum
weight in a weighted graph
(3) e
b
(4)
(3) (1) (3)
h
a d (2) f

(1) (2)
(4) (1) (2)
c
(4)
g
Department of Telecommunications 8
 Bin packing (BP)
University of Zagreb, Croatia

 Pack n objects of size ci , i=1,...,n into the

minimum number of identical bins of capacity C
 Geometric bin packing: items are of arbitrary shape
 Applications: packing commercials into breaks;
storage; packing boxes into trucks,...

...

Department of Telecommunications 9
 Bin packing (BP)
University of Zagreb, Croatia

 Pack n objects of size ci , i=1,...,n into the

minimum number of identical bins of capacity C
 Geometric bin packing: items are of arbitrary shape
 Applications: packing commercials into breaks;
storage; packing boxes into trucks,...

2 bins
needed
...

Department of Telecommunications 10
 Combinatorial explosion
University of Zagreb, Croatia

 A finite search space implies that an optimal

solution can surely be found through
exhaustive search (brute-force)
 Problem: combinatorial explosion!

TSP (Travelling Salesman Problem)

cities (n) Size of solution Relative time for
space (n!) brute force
20 20! ≈ 2.5*10^18 1 hour
21 21! ≈ 5.1*10^19 20 hours
22 22! ≈ 1.2*10^21 17.5 days
25 25! ≈ 1.5.*10^25 6 centuries

Department of Telecommunications 11
 Complexity theory
University of Zagreb, Croatia

 Algorithm complexity

 Problem complexity

Department of Telecommunications 12
 Algorithm complexity
University of Zagreb, Croatia

 Computational complexity: the number of steps

needed to solve a problem of size n
 Worst-case analysis; scalability issues
 Gives an asymptotic bound on the step count
 Does not specify the practical run-time for a given
instance
 Big O-notation
 An algorithm has complexity f(n)=O(g(n)) if there
exist postive constants n0 and c such that
n  n0 , f (n)  c  g (n)
 Big Ω-notation; Big Θ-notation
Department of Telecommunications 13
 Polynomial time algorithm
University of Zagreb, Croatia

 An algorithm is polynomial if its complexity is

O(p(n)), where p(n) is a polynomial function of n

p(n) = ak n k +  + a j n j +  + a1n + a0
O(nk)
ak  0; a j  0, 1  k  k − 1

 Example:
 Dijkstra’s shortest path algorithm
 O(n2)

 Floyd’s all-pairs shortest path algorithm

 O(n3)

Department of Telecommunications 14
 Exponential time algorithm
University of Zagreb, Croatia

 An algorithm is exponential if its complexity is

O(cn), where c is constant c  , c  1

 Example:
 Exhaustive search for SAT
 O(2n)

 Exhaustive search for TSP

 O(n!)

Department of Telecommunications 15
 Polynomial vs. Exponential Growth
University of Zagreb, Croatia

Complexity Problem size

n=10 n=20 n=30 n=40 n=50
Exponential Polynomial

O(n) 0.01 ms 0.02 ms 0.03 ms 0.04 ms 0.05 ms

growth

O(n2) 0.1 ms 0.4 ms 0.9 ms 1.6 ms 2.5 ms

O(n5) 0.1 s 3.2 s 24.3 s 1.7 min 5.2 min

O(2n) 0.001 s 1.0 s 17.9 min 12.7 days 35.7 yrs

O(3n) 2x108
growth

0.059 s 58.0 min 6.5 yrs 3855

centuries centuries

Values from: M Garey and D. Johnson, “Computers and Intractability: A Guide to the Theory of
NP-completeness”, W.H. Freeman and Co. Publishers, NY, 1979

Department of Telecommunications 16
 University of Zagreb, Croatia

Department of Telecommunications 17
 Problem Complexity
University of Zagreb, Croatia

 Problems are classified according to how

difficult they are to solve

 The complexity of a problem is equivalent to

the complexity of the best algorithm solving it
 Tractable (“easy”) problems – there exists a
polynomial time algorithm to solve it
 Intractable (“difficult”) problems –no polynomial time
algorithm is known which solves it

Department of Telecommunications 18
 Decision problems
University of Zagreb, Croatia

 Complexity problems deal with decision problems

- the solution is yes/no
 Every optimization problem can be reduced to a
decision problem
 TSP
 “Is there a Hamiltonian tour shorter than T?”
 Knapsack problem
 “Is
there a subset of objects whose total size is less than C
and value greater than V?”
 Bin Packing
 “Do the objects fit in less than B bins?”
 SAT
 “Isthere a combination of values that makes the logical
expression true?”
Department of Telecommunications 19
 Complexity classes
University of Zagreb, Croatia

 Class P: decision problems that can be solved

by a deterministic machine in polynomial time
(i.e., worst-case complexity is bounded by a
polynomial function)
 Example: MST, Shortest path problem, max flow
network
 Class NP: decision problems that can be
solved by a nondeterministic machine in
polynomial time
A proposed solution can be verified (checked) in
polynomial time
 Example: all P problems, TSP, MST, …

Department of Telecommunications 20
 Complexity classes
University of Zagreb, Croatia

 Class NP-complete: NP decision problem X is

NP-complete if all other problems in NP can be
reduced to X in polynomial time;
 Ifa polynomial time algorithm is found for an NP-
complete problem, it can solve all problems in NP,
then P=NP
 Is P = NP or P  NP ??? Open question

P
OR P = NP = NP-complete
NP NP -
complete

Department of Telecommunications 21
 Complexity classes
University of Zagreb, Croatia

 Class NP-hard: “at least as hard as NP-

complete problems”
 An algorithm for solving it can also solve NP-
complete problems but an algorithm which solves an
NP-complete problem does not necessarily solve it
 Can be any type: decision, search or optimization
problems
 Alloptimization problems whose associated decision
problems are NP-complete are NP-hard

P = NP =
NP - P
NP -hard
complete
OR NP-complete
NP NP -hard

Department of Telecommunications 22
 NP-hard problems
University of Zagreb, Croatia

 Most real-world optimization problems;

usually solved sub-optimally using heuristics

 Classical examples:
Flow-shop and job scheduling problems
Generalized assignment problem, location facility
Data clustering, graph partitioning, graph coloring
Vehicle routing problem, set covering, Steiner
tree problem
Knapsack problem, bin packing

Department of Telecommunications 23
 Summary: Why are some problems
difficult to solve?
University of Zagreb, Croatia

 The solution space is too big for exhaustive

search
 NP-complete/hard; combinatorial explosion
 The problem is too complicated to facilitate an
answer
 Must use simplified models
 The evaluation function is noisy or changes
with time
 It is so heavily constrained that even one
feasible solution is difficult to find

Department of Telecommunications 24
 Optimization methods
University of Zagreb, Croatia

Exact Approximate
A*
methods methods
Simplex-based Constraint
for LP programming Approximation
algorithms
Heuristic
Branch and Bound, Dynamic
Cut , Price (B&B, programming algorithms
B&C, B&P)

Improvement (meta)heuristics
Constructive heuristics
• Local search
• Greedy algorithms
•Nature inspired
• Tabu search, Genetic
Hybrid methods
algorithms, Simmulated
• GRASP Annealing, Ant colony
•Problem specific
heuristics

Department of Telecommunications 25
 Exact methods:
University of Zagreb, Croatia

 Simplex for LP problems – if variables are

continuous
 Branch and Bound and A*: based on implicit
enumeration of all solutions;
 ‘divide and conquer’ concept
 Dynamic programming: ‘bottom-up’ approach
 Recursive division of the problem
 Constraint programming: uses tree search and
logical implications

Department of Telecommunications 26
 Approximate algorithms
University of Zagreb, Croatia

Approximate algorithms can be divided into

approximation algorithms and heuristic
algorithms

 Approximation algorithms: provide provable

solution quality and provable run-time bounds

 Heuristic algorithms: do not guarantee

optimality of the solution or define how close it
is to the optimal; usually give ‘good’ solutions
quickly
Department of Telecommunications 27
 Approximation algorithms
University of Zagreb, Croatia

 ε-approximation algorithm: its time complexity is

polynomial and the obtained solution s is
s    s, if   1 where s* is the global
  s  s, if   1 optimum (minimization);
ε can be constant or a function of the input size
 Example: Bin packing
 First Fit Decreasing (FFD) – Θ(n log n)
 (11/9)s*+1

 First Fit
 (17/10)s*+2

Department of Telecommunications 28
 Approximation algorithms
University of Zagreb, Croatia

 Goal of designing approximation algorithms:

find tight worst-case bounds
 Gives knowledge on the difficulty of the
problem
 Problem dependent
 In practice, often attain approximations too far
from the global optimum and may thus not be
applicable

Department of Telecommunications 29
 Heuristic algorithms
University of Zagreb, Croatia

 Don’t guarantee how far we are from the

optimum
 How to evaluate the quality of the solutions ?
 Analytical lower/upper bounds
 Compare with opimal solutions for small
test cases
 Average case analysis
 Empirical testing
 Difficult to find a representative sample
 Hard to compare

Department of Telecommunications 30
 Heuristic algorithms
University of Zagreb, Croatia

Heuristics can be constructive or improvement (or

hybrid)

 Constructive: evaluate partial solutions;

terminate when a whole solution is constructed
 Ex: greedy algorithms

 Improvement: iterative; evaluate complete

solutions
 Metaheuristics – general ‘templates’ used to guide
the design of heuristics for specific problems

Department of Telecommunications 31
 Metaheuristics
University of Zagreb, Croatia

 ‘meta’ – Greek for ‘upper layer methodology’

 Term ‘metaheuristic’ – 1986, F. Glover
 Guiding strategies for designing heuristics
 Define how to explore the search space
 Can be applied to any optimization problem

 Often mimic natural phenomena

 evolution– Genetic algorithms (GA)
 memory – Tabu Search (TS)
 annealing – Simulated Annealing (SA)
 ants – Ant Colony Optimization (ACO)
 bees, wasps, immune system, …

Department of Telecommunications 32
 When to use heuristics?
University of Zagreb, Croatia

 Easy problems (class P) but with excessively

large instances
 Easy (or difficult) problems with real-time
constraints
 Difficult problems (NP-hard, NP-complete) of
moderate and large size
 Non-analytical models of optimization problems
 Problems with uncertainty or robustness issues

Department of Telecommunications 33
 Which optimization method to use?
University of Zagreb, Croatia

 Problem type:
 Design problems: long-term decision making,
financial investments
 Solution quality very important; Search time can be months
 Planning problems: medium-term decision making,
network planning
 Solution quality important; Search time can be hours/days
 Control problems: short-term decision making,
operational problem; online problems (routing)
 Search time critical; Solution as good as can be with limited
time

Department of Telecommunications 34
 Trade off: solution quality vs. search time
University of Zagreb, Croatia

Design
problems
Solution quality

Planning
problems NOTE:
Doesn’t
necessarily
Control grow linearly
problems

Search time
Department of Telecommunications 35
 Which optimization method to apply?
University of Zagreb, Croatia

 The trade-off between search time and solution

quality depends on the problem and the
method used but in general:
 Exact algorithms are usually used for smaller-sized
instances for planning and design (and for small
online problems)

 Greedy algorithms are usually fast and used for

online problems (and planning problems if they give
high quality solutions)

 Iterative improvement heuristics often require many

iterations and are best for planning problems (and
difficult design problems)
Department of Telecommunications 36
 Create a new heuristic or use existing?
University of Zagreb, Croatia

 If the problem can be reduced to a classical

NP-hard optimization problem, it may be easier
to apply the State-of-the-Art methods for that
problem or similar approaches

 Examples:
routing → Steiner tree problem
Multicast
Wavelength assignment → Graph coloring

Department of Telecommunications 37