Optimization Techniques

Optimal Problem Formulation

1
 Books
Recommended Books:
1. Kalyanmoy Deb, optimization for engineering design,
algorithms and examples, Prentice Hall (2005).
2. M. Asghar Bhatti Practical Optimization Methods With
Mathematics Applications, Springer-Verlag N.Y, Inc., 2000.

nasir m mirza 2
 Optimization
• The idea behind “optimization” is to find the “best”
solution from a domain of “possible” solutions.
• Optimization methods provide mathematical tools that
allow the search for this “best” solution to carried out in
a rational and efficient way.
• Before these tools can be applied the design problem
needs to be recast in an appropriate form.

nasir m mirza 3
 Optimization in Design
Need Identified

Choose Design Variables

Formulate Constraints

Formulate Objective Function

Set up Variable Bounds

Choose an Optimization Algo

Obtain solutions

nasir m mirza 4
 Optimization
Set of all “workable”
or “functional designs”
x2 H2 : x2 > c2 (Allowed by physics,
orange boarder)

H1 : X1> c1

* Set of all “acceptable”

designs. (allowed by
constraints, yellow
boarder)

Optimal Design
U(x1,x2) = Umax

x1

nasir m mirza 5
 Important definitions
• Objective Function:
• It represents the quantity (U) which is to be optimized
(the “objective”) as a function of one or more independent
variables (x1, x2, x3…)
• The best form to put the objective function in depends on
the optimization technique to be employed.

U = U( x1 , x2 , x3…) → Uopt

nasir m mirza 6
 Important definitions
• Design Variables:
• The independent variables (x1, x2, x3…) that the
objective function depends on.
• It is generally best to minimize the number of design
variables … the more variables the tougher the
optimization will be.

nasir m mirza 7
 Important definitions
• Constraints:
• Relations which limit the possible (physical limitations) or
the permissible (external constraints) solutions to the
objective function.
• Constraints come in two mathematical “flavors.”
• Equality Constraints: Often come from fundamental physics
considerations (e.g. cons. of mass)
• Inequality Constraints: Often from safety, cost, space, material
strength limits etc.
• Generally Equality Constraints are easier to deal with than
inequality constraints.
• it is desirable to reduce the number of constraints.

nasir m mirza 8
 Mathematical Formulation
• Objective Function of n independent design variables:
For U( x1, x2, x3…xn); Find Uopt
• Equality Constraints:
Gi( x1, x2, x3…)=0 ; i=1,2,…,m
• Inequality Constraints:
Hj(x1, x2, x3…) < or > Cj ; j=1, 2 ,…, l.
• If n > m → An Optimization problem results;
• If n = m → A unique solution exists…just solve all
equations simultaneously;
• If n < m → The problem is “overconstrained” no
solution which satisfies all of the constraints is possible.

nasir m mirza 9
 Optimization
x2 Set of all “workable”
H2 : X2> c2 or “functional designs”
(Allowed by physics,
orange boarder)

H1 : X1> c1

* Set of all “acceptable”

designs. (allowed by
constraints, yellow boarder)

Optimal Design
U(x1,x2) = Umax

x1

nasir m mirza 10
 Classification of Optimization Techniques
• Calculus based Techniques
• Lagrange Multipliers
• Search Methods
• Elimination Methods
• Exhaustive
• Fibonacci
• golden section search
• “Hill Climbing” techniques
• Lattice Search
• Steepest ascent
• “Programming” methods
• Linear Programming
• Geometric Programming

nasir m mirza 11
 Exhaustive Search
x2
H2 : X2> c2

H1 : X1> c1

* Note: None of the search

points exactly hits the
optimum. The space
between search points is
known as the “interval of
Optimal Design uncertainty”
U(x1,x2) = Umax

x1

nasir m mirza 12
 Lattice Search
1
3

* *

2
4

* *

nasir m mirza 13
 “Programming” Methods
• These methods have nothing to do with “Programming” in the sense
that you usually think of it!
• Linear Programming: Very powerful, but very limited!
• Applies only when the objective function and all constraints can be
expressed as Linear Functions … Not often the case in thermal/fluid
systems
• Dynamic Programming: Related to optimizing a “process”
• Lets you find the “best” order to do steps in
• Very useful in Project Management and Assembly optimization
• Geometric Programming: Similar to Linear Programming, but now all
functions must be polynomials.
• Reduces the restrictions on linear programming quite a bit
• Very useful where empirical correlations for system behavior are known
• More difficult and computationally intensive than Linear Prog.

nasir m mirza 14
 Functions
Optimization deals with functions. A function is simply a mapping from one
space to another. (that is, a set of instructions describing how to get from
one location to another)

f :A B This is the range

This is the domain

This is a function
f
For any x A and yB A B

Note: A function maps each value of x to one and only one value for y

x y  f x 
y
nasir m mirza 15
 Examples of Single variable Function

For 0 x5 y  3x  5 5  y  20

Domain f x  Range

A  0,5 B  [5,20]

3 x A
f 3  14
14

nasir m mirza 16
 Examples of Single variable Function
20 y A straight line is a function.

y  3x  5
Range

Y =14 For 0 x5

x
Domain
0 5
X =3

nasir m mirza 17
 Optimization of a function
It involves finding the maximum value for y over an allowable range.

20 y max3 x  5
0 x5
Range

Here, the
optimum occurs
at x = 5 (y = 20)
5

x
Domain
0 5
nasir m mirza 18
 What is the solution to this optimization problem?
y
 1 
max  
5  x 
0  x  10

x
5 10

There is no optimum
because f(x) is
discontinuous at x = 5

nasir m mirza 19
 What is the solution to this optimization problem?

y
12
max2 x
0 x6

There is no optimum
because the domain is open
(that is, the maximum
occurs at x = 6, but x = 6 is
NOT in the domain!)

x
0 6

nasir m mirza 20
 What is the solution to this optimization problem?

y
12
max  x x0

There is no optimum
because the domain is
unbounded (x is allowed to
become arbitrarily large)

x
0

nasir m mirza 21
 One-Variable case
The Weierstrass Theorem provides sufficient conditions for an optimum
to exist, the conditions are shown in next slides

Here suppose f(x) is continuous over the domain of x.

Sufficient conditions guarantee a

y solution, but are not required
max f ( x)
Necessary conditions are required
for a solution to exist

x
The domain for f(x) is closed
and bounded.
x
nasir m mirza 22
 Derivatives
Formally, the derivative of a function is defined as

df  f ( x  x)  f ( x) 
f ' x    lim  
dx x  0
 x 
All you need to remember is the derivative represents a slope.
y

f x  x 
f ( x  x)  f ( x)
f x 
 f ( x  x )  f ( x ) 
x Slope =  
  x 
x
x x  x
0
nasir m mirza 23
 2
Example: f ( x)  x
2x
 f ( x  x)  f ( x)  y x 2
 
 x 

 ( x  x)  x 
2 2 x2
 
 x  x x
0

 x 2  2 xx  x 2  x 2   2 xx  x 2 
    2 x  x
 x   x 

lim 2 x  x  2 x
x 0

nasir m mirza 24
 Unconstrained maximization
Strictly speaking, no problem is truly unconstrained. However,
sometimes the constraints don’t influence the maximum.

First Order Necessary Conditions

If x* is a solution to the optimization problem:

Min{ f(x) } or Max{ f(x) } then

f ' ( x*)  0

nasir m mirza 25
 How can we be sure we are at a minimum?
Secondary Order Necessary Conditions
*
If x is a solution to the maximization problem:
max f ( x)
x

then f ' ' ( x*)  0

*
If x is a solution to the minimization problem:
min f ( x)
x

then f ' ' ( x*)  0

nasir m mirza 26
 The second derivative is the rate of change of the first derivative
y y

x x
x* x*

f ' ( x*)  0 f ' ( x*)  0

f ' ' ( x*)  0 f " ( x*)  0
Slope is decreasing Slope is increasing

nasir m mirza 27
 An Example

Suppose that a company owns a corporate jet. The annual

expenses are as follows:
 You pay your flight crew (pilot, co-pilot, and navigator a
combined annual salary of $500,000.
 Annual insurance costs on the jet are $250,000
 Fuel/Supplies cost $1,500 per flight hour
 Per hour maintenance costs on the jet are proportional to the
number of hours flown per year.
 Maintenance costs (per flight hour) = 1.5(Annual Flight Hours)

If you like to minimize the hourly cost of your jet, how many hours
should company use it per year?

nasir m mirza 28
 An Example

Let x = Number of Flight Hours

$750,000
Hourly Cost   $1500  $1.5 x
x

 $750,000 
min   $1500  $1.5 x 
x 0
 x 

First Order Necessary Conditions

 $750,000 750,000
2
 $1.5  0  x   707hrs
x 1.5

nasir m mirza 29
 An Example
Hourly Cost ($)

Annual Flight Hours

nasir m mirza 30
 An Example

Let x = Number of Flight Hours

 $750,000 
min   $1500  $1.5 x 
x
 x 
First Order Necessary Conditions

 $750,000 750,000
2
 $1.5  0  x   707hrs
x 1.5
Second Order Necessary Conditions

$1,500,000
3
0 For X>0
x
nasir m mirza 31