M An.AB is a regislered lrademark ofle MalhWorks Inc.

This book is p led on ncid paper.@

Copyrighl@22 by John Wiley & Sons N e w York. AII righls ed.
Published simultaneously in Cnnadn.
N o pnrt of Ihis publiclllion may be reprodu dstored in a trievsystem or 1nsmitted in nny form or
byanym nseleclronic mechanicaJ pholocopyingcordingscanning or otherwise except as
permittcd under Sections 107 or 108 ofthe 1976 United Stalcs Copyrighl Acr wirhoul either the prior
writtcnprmission of Ihe Publisher or outhori7.3lion through paymenl of the appropriule per-copy fcc 10
Ihe Copyrighl Clearance Cenler 222 Ro W d Drive Dnnver M A 01923 (978) 7508400fnx (978)
7504744. Requ IS 10 Ihe Publisher for permission should be nddressed 10 the Pemlissions Departmenl
John Wilcy & Sons Inc. 605 ird Avenue N e w York N Y 10158-12 (212) 850-11 (212)
850-6008 E-Mail: PERMREQ@WILEY.COM.
spublicalion is designed 10 provide accule nnd authorilalive infomtalion in E 10 esubjecl
mattercoved. It is 80ld with the underslunding Ihal the publisher is nol engaged in rendng
professional services. If professionnl ndvice or othcr expert assislance is quirede services of a
compelent professional pe na should be soughl.
Ubrary ofCongress Catalog;lIg;n P"bl;catiotl D ata:
VenkoluromnP.

Applied oplimiZlllion with Programming I P. Venkalarmlln.

p. com.
A Wiley-Intersciencc publicnlion."
ISBN 0-471 958.5 (c1oth : alk. pnper)
1. MnlhemalicaJ optimization--Dolll procing2. MATLAB. J. Title.
QA402.5.V42 2001
519.3-dc21
Prinled in Ihe Uniled Slales of Americn.
1 0 9 8 7 6 5 4 3 2 1
2001026938

T 0 coexistence of peace and harmony between the different
melodies of people everywhere
6

C O N T E N T S
P R E F A C E
1 Introductlon
1.1 Optimization Fundamentals I 2
1.1.1 Elements of Problem Formulation I 4
1.1.2 Mathematical Modeling I 10
1.1.3 Nature of Solution I 16
1.1.4 Characteristics of the Search Procedure I 20
1.2 Introduction to MA11..As I 25
1.2.1 W h y MATLAB? I 25
1.2.2 MATLAB Installation Issues I 26
1.2.3 Using MATLAB the First Time I 27
1.2.4 Using the Editor / 33
1.2.5 Creating a Code Snippet / 37
1.2.6 Creating a Program I 40
Problems I 44
xiii
1
2 Graphlcal Optlmizatlon 4 5
2.1 blem Definition I 45
2.1.1 Example 2.1 I 46
2.1.2 Format for lhe Graphical Display I 47
2.2 Graphical Solution I 48
2.2.1 MATLAB High-Level Graphics Functions I 48
2.2.2 Example 2.I-Graphical Solution I 50
2.2.3 Displaying the Graphics I 53
2.2.4 Customizing the Figure I 54
2.3 Additional Examples I 56
vll
vlll CONTENTS
2.3.1 Example 2.2 I 56
2.3.2 Example 2.3 I 64
2.3.3 Example 2.4 I 73
2.4 Additional M A11..AB Graphics I 79
2.4.1 Handle Graphics I 80
2.4.2 Graphical User Interface I 81
2.4.3 GUI Code I 84
References I 91
Problems I 92
3 Llnear Programmlng
3.1 Problem Definition I 94
3.1.1 Standard Format / 94
3.1.2 Modeling Issues / 98
3.2 Graphical Solution / 107
3.2.1 Example 3.1 / 110
3.2.2 Characteristics of the Solution / II1
3.2.3 Different Solution Types I 114
3.3 Numerical Solution-the Simplex Method I 115
3.3.1 Features of the Simplex Method / 115
3.3.2 Application of Simplex Method / 117
3.3.3 Solution Using M A11..AB / 120
3.3.4 Solution Using MATLAB'S Optimization Toolbox I 123
3.4 Additional Examples / 124
3.4.1 Example 3.2-Transportation Problem / 124
3.4.2 Example 3.3-Equality Constraints and Unrescted
Variables / 130
3.4.3 Example 3.4- A Four-Variable Problem / 134
3.5 Additional Topics in Linear Programming / 138
3.5.1 Primal and Dual Problem / 138
3.5.2 Sensitivity Analysis / 148
Refernces / 151
Problems / 152
4 Nonlinear Programming
4.1 Problem Definition I 155
93
154
CONTENTS Ix
4.1.1 Problem Formulation-Example 4.1 / 155
4.1.2 Discussion of Consaints / 157
4.2 Mathematical Concepts / 159
4.2.1 Symbolic Computation Using MATLAB / 159
4.2.2 B a ~ i c Mathematical Concepts I 162
4.2.3 Taylor's T heorem/Series / 169
4.3 Graphica1 Solutions / 171
4.3.1 Unconstrained Problem / 171
4.3.2 Equality Constrained Problem / 172
4.3.3 Inequality Constrained Problem / 173
4.3.4 E quality and Inequality Constraints / 174
4.4 Analytica1 Conditions / 175
4.4.1 Unconstrained Problem / 176
4.4.2 E quality Cons ined Problem I 179
4.4.3 Inequality Constrained Optimization I 186
4.4.4 General Optimization Problem / 191
4.5 Exnples / 194
4.5.1 Example4.2/ 194
4.5.2 Example 4.3 / 196
References / 200
Problems / 201
5 Numerical Technlques-T h e O ne-Dlmenslonal Problem
5.1 Problem Definition I 204
5.1.1 Constrained One-Dimensional Problem / 204
5.2 Solution to the Problem / 205
5.2.1 Graphical Solution / 205
5.2.2 Newton-Raphson Technique / 206
5.2.3 Bisection Technique I 209
5.2.4 Polynomial Approximation / 211
5.2.5 G olden Section Method I 214
5.3 Importance of the One-Dimensional Problem / 217
5.4 Additional Examples I 219
203
5.4.1 ample 5.2-Il1ustion of Geneml G olden Section Method / 219
5.4.2 Example 5.3 T wo-Point Boundary Value Problem I 220
5.4.3 Example 5.4-Root Finding with Golden Section I 223
X C O N T E N T S
References I 225
Problems I 225
6 Numerlcal Technlques for Unconstralned Optlmlzatlon
6.1 Problem Definition I 227
6.1.1 Example 6.1 I 228
6.1.2 Necessary and Sufficient Conditions I 228
6.1.3 Elements of a Numerica1 Technique I 229
6.2 Numerical Techniques-Nongradient Methods I 230
6.2.1 Random W a1k I 230
6.2.2 Pattem Search / 234
6.2.3 PoweJ1's Method I 238
6.3 Numerical Thniques-Gradient-Based Methods I 241
'6.3.1 Steepest Descent Method I 241
6.3.2 Conjugate Gradient (F1etcher-Reeves) Method I 244
6.3.3 Davidon-FJetcher-PowelI Method I 246
6.3.4 Broydon-Fletcher-GoldfartShanno Method I 249
6.4 Numerical Techniques-Second Order I 251
6.5 Additional Examples I 253
6.5.1 Example 6.2-Rosenbrock Problem / 253
227
6.5.2 Example 6. 3 -Three-Dimensional A o w n raRo ngDisk I 255
6.5.3 ExampJe 6.4-Fitting Bezier Parametric Curvcs I 258
References I 262
Problems I 263
7 Numerlcal Techniques for Constralned Optlmlzatlon 265
7.1 Problem Definition I 266
7. 1.1 Problem Formulation-Example 7.1 I 266
7.1.2 Necessary Conditions / 267
7.1.3 Elements of a Numerical Technique I 269
7.2 Indirect Methods for Constrained Optimization I 270
7.2.1 Exterior Penalty Function (EPF) M eod I 271
7.2.2 Augmented Lagrange Multiplier (ALM) Method I 276
7.3 Direct Methods for Consained Optimization I 281
7.3.1 Sequentia1 Linear Programming (SLP) I 284
7.3.2 Sequential Quadratic Programming (SQP) I 289
C O N T E N T S xl
7.3.3 Genera1ized Reduced Gradient (GRG) Method I 297
7.3.4 Suential Gradient Restoration AJgorithm (SGRA) I 302
7.4 Additiona1 ExampJes I 307
7.4.1 ExampJe 7.2-FJagpoJe Problem I 307
7.4.2 ExampJe 7.3-Beam Design I 310
7.4.3 ExampJe 7.4Optima1 Control I 313
References I 316
Problems I 316
8 Dlscrete Optlmlzation
8.1 Concepts in Discrete Programming / 320
8.1.1 Problem Relaxation I 321
8.1.2 Discrete Optimal Solution I 322
8.2 Discrete Optimization Techniques I 324
8.2.1 Exhaustive Enumeration I 326
8.2.2 Branch and Bound I 329
8.2.3 Dynamic Programming / 336
8.3 Additional Examples I 341
8.3.1 Exnmple 8.4--1 Beum Design I 341
8.3.2 Zero-One Integer Programming I 343
References I 348
Problems I 348
9 Global Optlmlzation
9.1 Problem Definition I 351
9.1.1 Global Minimum / 351
9.1.2 Nature of the Solution I 354
9.1.3 Elements of a Numerica1 Technique I 356
9.2 Numerical Thniqusand Additional Examples I 357
9.2.1 Simulated Annealing (SA) I 358
9.2.2 Genetic Algorithm (GA) I 366
References I 377
Problems I 378
10 Optlmization Toolbox from M ATlAB
10.1 The Optimization Toolbox I 380
318
350
379
xll CONTENTS
10. 1.1 Programs I 380
10.1.2 Using Programs I 382
10.1.3 Setting Optimization Parameters I 384
10.2 Examples I 385
10.2.1 Linear Programming I 385
10.2.2 Quadratic Programming I 386
10.2.3 Unconstrained Optimization I 388
10.2.4 Constrained Optimization I 389
Reference I 391
Index
393
P R E F A C E
le s u ~ j e c t of optimization is receiving serious attention o m engineers scientists
managers and most everybody else. This is driven by competion quality assurance
cost of production and finally the success of the business entee.lgnoring e
practice of optimization is not an option during current times.
Optimization is practiced through softwar programs and requires significant
computer resources. T he techniques of optimization have not changed significantly in
cent years but tlle areas of applications have mushroomed at a significant rate.
Successfully embedding the use of optimization in professional practice requires at
least three Pquisites.bey include mathematical modeling of the design problem
knowledge of computer programming and knowledge of optimization techniques.
Many special-purpose optimization software packages relax required
knowledge of programming are available today. T o use them efficiently the
remaining two areas still have to be addressed.
lere are several excellent books on the subject of optimization a few of them
released recently. Most of them covrthe subject in depth. which is necessary because
thmathematical models are nonlinear and require special techniqusthat are usually
notpaofany core curriculum. All ofebooks assumeereader is familiar wilh
a programming languageaditionally F O R T R A N and recently C. Another assumplion
hquently made is the optimization chniques w i1l be implemented in a
mainframe computing environmenl. Such a combination is exemely difficu1t for the
self-Iearner even with enormous motivation.
A n explosion of inexpensive desktop computing resources aids engineering design
practi today. Paralleling this development is e availability of extensive sou
on the Internet both for learning and deployment. T his is joined by the availability of
sofsystems that provide opportunityfor handling mathematics graphics. and
programming in a consistent manner. These software systems are significant1y easy to
master compar to the higher-level programming languages of the pvious ye S.
lis book primarily seeks to harness this iangulation of srvices to provide a
practical approach to the study of design optimization. le book uses M ATI.AB @ to
i11ustrate and implement the various techniques of optimization. MATLAB is a product
from MathWorks. Inc.
This book attempts 10 accomplish two important obj tives. The fir
xlv PREFACE
of symbolic. numerical. and graphical f l tures ofMATLABlt integrates is powerful
combination during nslation of many algorithms into app1ied numerical
techniques for design optimization. There is a constant enhancement of the
programming skills throughout thbook. The second 0ective is thprimary task of
book-to communicate and demonstrate various lIumerical techniques that are
currenlused in tlte area oloptimal des.
A H of thc numerical techniquesMEsupported by M M L A B code available
computer files. These files are available on a companion web site
www.wilev.comlvenkat. ltis necessary to visit the web site to download thc files to
fo l l o w a l l of t h e E X tlI n p i es -T h e b oo k w i l l b e e ntiallyin c omplet.ewitho 0u tthcsc files.
At the web site you will also find uscful links 10 other resources a web course on
MATLAB programmingand updated information about Applied Optintimuon with
M A Tl.AB @ Programming.
The decision to locate the files on a web site pmvidEs an avenue to shadow the
dynamic changes in MATLAs sof1ware itsclf which is bcing transformed in a major
way almost every ycar.lt provides a mechanism 10 correct bugs ie code in a timely
manner. It estabishes an opportunity to improve code through reader suggcstions
and provides a way for e author 10 keep thc rcader engaged with new dcvclopments.
It docut down the sizc of the book enormously by locating all of the pag ofcode
outside the book.
A n unusual feature of the book is c inclusion of discrete optimization and global
optimization. Tradilional continuous design optimization techniques do take up a
significant portion of the book. Continuous problems generally allow the algorithms
to be mathematically developed in a convincing manncr and e ideas could be
core of original a1gorithms dcvc10pcd by the readers to address their o w n special
necds. Every chaptcr includes additional nontrivial cxamp1es at oftcn prescnt a
novel use or extcnsion of thc optimization techniques in the chapter. T he mathcmatical
algorithms and cxamples ar accompanied by MA' AB code available at the
companion web sue-Thenumerical techniques usually include strong graphical
support to il1ustrate c opcration of thc techniques.
Discrete andglobal optimization techniques are basedon verydifferentalgorithms
and .ch of thcm can easily justify full-fledged independent courscs detcd
exclusively 10 thcir S1udy. In a comprchensivc book such
PREFACE X V
classroom setting withou1 computational cxperience is also feasib1e. In is case the
a1gorithms can be presented and the nurnerical results illustrated. The senior/gradua1e
s1udents in various discip ines. especially engincering are the targc1 audience.
Optimization techniqucs are an importanl 1001 to solve dcsign problems in all
professional areas of study. M a n y illustrations are o m the area of mechanical
engineering reflecting experience ofeauor.
Independent learnel's particuIarly professionals who need to understand the subject
of optimization should also find the book very useful . A reader w h o has access to
MATLAB software c us the book to its best advantage. The book is largely
self-contained and develops all ncccssary mathematical concepts when ncedcd.
Abstract mathematical idcas of optimi tion introduccd graphically and through
iIlustrative examples in thc book. In many instances the operation of thc numcrical
technique is atcd to show h o w the design is changing with iterations. This has
been possible because MATLAB graphics are rich in fcatures and simple to incooratc.
making Ulcm cffcctive for convcying e idcas. It is for this reason e reader is
cxposed to graphical programming early on incIuding a chapter on graphical
optimization. T o balance thc development of programming skills and thpresentation
of optimization concepts programming issues are incrementally explored and
implemented. Investment of time in dveloping programming skills is e most
effective way to imbibe them. Thc author is nOl aware of shortcuts. Studcnts arc
expected to program independently. make mistakes debug errors. and incorporate
improvements as p tof their learning experience delivered through is book.
The book proceeds at a brisk pace to provide experiencc in MATLA8 programming
and communicate ideas in optimization in a reasonable number of pages. T his could
not be possible without taking advantage of Ihe accompanying web site 10 locate much
of the code for algorithms and examples. Merely running the code will n01
significantly ece the reader's programming skills. It is important for the student
to understand the code also. T o sist inis process the code carries Iiberal comments
often corresponding to the algorithm that is being implemcnted. The auor expects
e reader to have read e comments. The accompanying code allows the instructor
to assign both programming assignments as well as so
xvl PREFACE
approaches the instructor can choose to include only a subset of continuous
optimization algorithms to allow the inclusion of ese popular techniques. Ife
student has access to the Optimization Toolbox from MATlABen it can be integrated
into e course for handling the final design project . T he last chapter provides an
overview on the use of the toolbox.
This book started out using M ATI.AB Version 5. About halfway through the book
e transition to MATlAB Version 5.3 was complete. At the completion of the
m uscript MATLAB Version 6 wshipping for some time. A s of this tirne Version
6.1 is available. Most institutional MATlAB licenses are usually based on subscription
so they will have latest version of e software. In a sense and is is true of
all books that depend on particular software systems the book rnay appear out of date
with respect toe software. This is not really so. While the usage ofMATLAB functions
rnay change between versions there should be suitable wamings and instruction about
e different usageA n important insurance is built into the book particularly to
handle such changes if they arise. Sprinkled throughout the book is the habit of e
using e online help. In most instances is was primarily done to expose the reader
to alternative use of the same command. Another significant change the manuscript
had to endure was e loss of access to a U N I X version because of thuniversity's
change to distributed computing on P C clusters. The author's experience wie
early chapters suggs18 that this should not m at1er. Except for external files and
resources use of M AT l AB as illustrated in the book is indifferent to the various
operating sy(WindowsM a c U N I X or Linux).
T he topics chosen for inclusion and the simplicity of presentation ofe topics are
directly related to the experience of teaching the course on optimization at the
senior/graduate level in e Dpartmentof Mechanical Engineering for over ten years.
Experience proved at comprehension irnproved with simple iIIustrations however
complicated the mathematical ideas. O n the other hand straighorward mathernatical
expressions elicited no interest or understanding. The emphasis on application was
important to keep the students' attention. In this connection the author would like to
1 k all his studen18 for pushing hirn for simple explanations for providing positive
interaction and for their contribution of original and useful ideas in the discuss
PREFACE xvii

d
l
l
p
d
L
Y

The book was made possible through support from John Wiley dSonsInc.d
MathWorks Inc. Sincere thanks are owed to Bob Argentieri senior editor at Jobn
Wiley for accepting the proposaland w h o all along displayed a lot of patiencein
getting the book moving forward.Same is due to Bob Hilbed-associate managing
editor at John Wiley for his impssive work at c1eaning up the manuscript . Brian
Snapp N e w Media editor at John Wiley created companion web site
(www.wiley.comlvenkat) and w also be rnaintaining it. Naomi Femandes o m
MathWorks Inc. saw to it at 1 had the latest version of MATLAB as soon as it was
available. M y regard for Dr. Angelo Miele Professor Emeritus at Rice University is
more than can be expressed in these Iines. It was he who introduced m e to 1hsubjecl
of optimization and demonstrated the effectiveness of simple presenution-I will
always regard him as a great teacher. O f course m y family deserves special rnention
for putting up with all the Not now"Later"H o w about tornorrow?" din2
debugging e code. Sp ial ks are to A ana d Vinayak rny offspring t i
their patience understanding and encouragement. The auor apologizes for any
shortcornings on the presentation and welcomes comments criticisms and
suggestions for improvement at all times.
P. VENKATARAMhN
Rocheste N e w York
1-a. .
t
J
F
J
J
-

p
d
r
h
w

A
b
r

r
=
3
5
3
t
h
t
Y
4
y
I N T R O D U C T I O N
Optimization has become a necessary part of design activity in all major
disciplines. These disciplines are not restricted to engineering. T he motivation to
produce economically relevant products or services with embedded quality is the
principal reason for is inclusion. Improved production and design tools wi a
synergistic thrust through inexpensive computational resources have aided the
consideration of optimization methods in new developments particularly
engineering products. Even in the absencc of a tangible product optimization
ideas provide the ability to define and explore problems while focusing on
solutions at subscribe to some measure of usefulness. Generally the use of the
word optimization implis thc best result under thcircumstances. This includes
the particular set of constraints on the development rcsources current knowledge
market conditions and so on. Every one of us has probably used thc term at some
time to describe e primary quality of our work or endeavor. It is probably e
mosl used or abused lerm in advertising and presentations. Neverthclessc
ability to make the best choice is a perpelual desire among us all.
Optimization is frequently associated with design be it a product service or
strategy. Aerospace design was among earliestdisciplines to embrace optimization
in a significant way driven by a natura1 need to lower the tremendous cost associated
with carrying unnecessy weight in aerospace vehicles. Minimum mass structures arl
norm. Optimization forms part of the psyche of every aerospace designer. Saving
on fuel through trjectory design was another problem that suggested itself. Very soon
enti engineering community could recognize c need to define solutions based
on merit. Recognizing the desire for optimization and actua1ly implementing we two
different issues.
2 INTRODUCTION
Until recently. for much of the time. optimization was usually attempted only in
those situations where there were significant penalties for generic designs. The
application of optimi tion demanded large computational resources. In the nascent
years of digital computation these were available only to large national laboratories
and research programs. These resources were necessary to handle th nonlinear
problems at associated with engineering optimi .tion. As a result of these
constraints most of the everyday products were designed without regard to
optimization. This includes everything you sound you or use in your daily life. It
is inconceivable at the new generation of replacement products. likc the car. the
house. the desk. or e pencil. redesigned and manufactured today arc not designed
optimally in one sense or anothcr.
Today. you would defmitIy explore procedures to optimize your inveslents by
tailoring your portfolio. Y o u would optimize your business travel time by
appropriaIy choosing your dcstinations. Y o u can optimize your commuting timby
choosing your time and route. Y o u can optimize your necessary expenditure for Iiving
by choosing your day and store for shopping. Y o u can optimizc the useful time you
connectto Intemet by detcrmining your time of connection. Y o u can buy software
will optimize your conncction to thc lntemct. Y o u can buy books or read articles
tcll you h o w to perform these various optimizations. The above activitics
primarily relatc to :;crvIces or strategy. It is n o w apparcnt at every activity except
aSeticprovides the scope for optimization. lis justifies looking at e study of
ootimizon as a tool can be applied 10 a variety of disciplines. If so the myria
d
ofo p tim iz at io n prob le m ns s o 11 I
T hisis eempha sis ofthe book.
The partnership between dsign and optimization activity is often found in
engineering. This book recognizes that connection and many of thc problems used for
illustrations and practice are from engin ringprimariIy mechanical civil and
aerospace design. Ncvertheless the study of optimization. particularly applied
optimization.is not m exclusive pmperty o f m y sptcinc discipurle.It involves the
dECOVEryanddesignof utions through appropriate techniques ociated with the
formulation of the problem in a sp ific manner. This can be done for example in
economics chemistry and business m a n
1.1 OPTIMIZATION F U N D A M E N T A L S
Optimization can be applied to all disciplincs. Qualitatively this assertion implies
multiple decision choices; implicitly recognizing the ncessity of choosing among
a1tematives. T his books deals with optirnization in a quantitative way. This means that
an outcome of applying optimization to e problem design. or service must yield
numberat will define the solution. or in other words numbers or values will
characterize the particular design or service. Quantitative dcscription of lution
requires a quantitative description of the problem itself. T his description is called a
; : . ~
".
"
1.1 OPTIMIZATION FUNDAMENTALS 3
mathematical model . bedi. icharactcrization. d its circumstances must be
expressed mathematically. Consider the design activity in the following cases:
N e w consumcr research indicatthat people like 10 drink about 0.5 litcr of soda
pop al a time during the summer months. The fabrication COSI of the redesigned
soda can is proportional to surface area. and can be estimatcd at $1.00 pcr
square meter of the materiaI used. A circular cross section is the most plausible
given current tooling available for manufacture. For aesthetic reasonse
height must be at least twice the diametcr. Studies indicate at holding comfort
requires a diameter between 6 and 9 cm.
A cantilever beam nceds to be dcsignd to carry a point load F at the cnd of a
beam of length L. The cross section of the beam w iII be in the shupe of thltter
I(e d a s I-b m). The beam should meet prescribed failure criteria.
ere is also a limit on idefItion. A beam of minimum mass is requir to
be designed.
M y P C Company has decidcd to invcst $12 million in acquiring several nW
Componenl Placemnl Machines to manufacture differcnt kinds of
mothcrboards for a ncw gcncration of personal compulers. Three models of
thcsc machines are under consideration. Total numbcr of operators available is
100 becausc of the local labor market . A fIoor spacc constraint needs 10 bc
satisfied because of the diffcrent dimcnsions of thcse machines. Additional
information rclating to each of the machines is given in Table 1.1. The company
wishes to determine how many of each kind is appropriate 10 maximize the
number of boards manufactured per day.
The above list rcpr ents three problems at w iII be used 10 definc formal elements
of an optimization problem. Each problem quires information from the specific arca
or discipline to which it refers. T o rccognize or design these problems assumes at
designcr is conversant wie particular subj tmatter. le problems are kept
simple to focus on optimization issucs. Problems of such variely are important today.
Recent advertisements in general consumer magazines i1Iustrate at Alcoa (an
aluminum m ufaclurer) is quithappy to have rcducedL hc weight of the standard
soda pop can by over 3 0 % in recenl years. A similar c1aim by the plastics industry with
respect to the standard milk jug (gaIlon) is al50 evident in these magazines although
in is case the number is 409. Roof coUapses in c Northeast due 10 excessive snow
Table 1.1 Component Placement Machir
Machine Operablc
Model BoardTypes BoardslHour Opemtors/Shift HoursIDay CostIMachine
A 10 55 18 400000
B 20 50 2 18 600000
C 18 50 2 21 700000
4 INTRODUCTION
during the 1998-1999 winter will have structura1 designers and homebuilders
exploring second problem. T hvibrant stock market in these times has made
ba1ancing the investment portfolio more challenging. T hc third case may suggest a
mathematica1 model appropriate for such decision making.
1.1.1 Elements of Problem Formulatlon
In this section w e will in duce e forma1 elements of thc optimization problem. In
this book the tcrm product a1so fers to a servic or a strategy. It should be
understood at opzation presupposes the knowlcdge of e design rules for the
sp ific problem primarily the ability to describe the design in mathematica1 terms.
T hese terms cJude design variables design parameters and design functions.
Traditiona1 design practiat is design without regard to optimization incJudes a11
of these elements ugh ey were not formally recognized as such. This also
justifies e prerequisi that you must be capable of dcsigning the 0tif you are
planning to apply e techniqucs of optimi tion. It is al80 a good idea 10 recognizc
that optimization is a procedurc for scarching 1he best design among candidatscach
of which can produce an acceptable product . The oced for thc objec1 or product is not
questioned here. but this may be due to a decision based 00 optimization applied in
another disciplinc.
Deslgn Varlables: Design variables are entities that identify a particular design. In
e search for optimal designese entities will change over a prcscribed range.
The values of a complete set of these variables charactcrize a specific design. Th
number and type of cntities bclonging to this set are vcry important in identifying and
setting up equ titative design problem. It is essential is choice captue
essence of object being desied d at e samc time provide a quantitative
characterization of the design problem. In applicd mathematical terminology. design
variables serve as the unknowns of the problem being solvcd. Borrowing an ana10gy
from the area of system dynami and control theoryey are equiva1ent to defining
the state of the systcm in is case thc statc of design. Typically design variables Can
be associatcd with describing object's size such as its length and height. In other
cesthey may represen1 the number of items. The choice of design variables is the
responsibility ofe designer guidcd by intuition expeseand knowledge. le is
a fundamenta1 requirement to be mct by is set of design variables namelyey must
be I inearly independent . This means a1 you cannot cstablish thvalue of onc ofe
design variables from e values of e remaining variables through basic arithmetic
(scaling or addition) operations. For example in a design having a rcctangular cross
section you cannot have e variables reprenting elcngwidth and area. If the
first two ar prescribede third is automatica1ly established. In complex designs
these relationships may not be very apparent . Nevertheless the choi of the set of
design variables must meet criterion of linear independence for applying the
techniques of optimization. From a practicnl pcrspcc1ivc thc property of I inear
independence identifies a minimum set of variables that cun complctely describe the
r::.-:
l
'"

1.1 OPTIMIZATION FUNDAMENTALS 5
design. T his is significant because the effort in obtnining the solution varies as an
integer power of the number of variables and this power is 1ypica11y greater an 2.
M ting the requirement ensures reduced difficulty in mathematically exploring thc
solution.
le set of design variables is identified as the design vector. 1Is v tor will be
consider a column vector in this book. In fac1 a11 vectors are column vectors E
text. The length of is vector which is n is the number of design variles io E
problem. T he design variables can cxpress different dimensional qu tities in
problem but in the malhematical model they are distinguished by charac1erx. A lI
of the techniques of optimization in this book nre bascd on eabstract mathematical
model. le subscript on x for cxamplc X 3rcpresents the third design variable which
may be the height of an objec1 in e chucterization of the product. This abstract
model is ne ssary for mathematical convenience. This book will refer to e design
variables in one of following ways:
(1) [-refcing to the vcc10r 01 ' dsign variablcs
(2) X or x-referring to thc vcctor again omilling 1hc square brackets for
convcnicncc if appropriate
(3) [xj x2' ... XIl]T_indicating thc vcclor lhrough its elements. Nole e
superscript transposition symbol to identify it as a colunm vector.
(4) X;. i = 1 2. . . . n-referring to all of the elements of the design vctor.
Thc above notationa1 convenience is extcndcd to all vcctors in the book.
Deslgn Parameters: In is bookese identify constan1will not change
dierent designs are compared. Many 1exts use term design parameters 10
prescnt dcsign variables w e defined earlier and do not formally recognize design
parameters as defincd hc. The principnl reason is that parameters have no role 10 play
detenning optima1 design. T hey are significant in the discussion of modeling
issues. Examples of parameters include matcria1 propcyapplied loads. and choicc
of shape. The paramctcrs in the abstract mathematical model are rep sentcd in a
similar form as the design vec1or excep1 tw e use the charactcr p. T herefore. [P]
P [P"P2' .. .Pq1 rep sent e parameters of the problem. Note eleng ofthe
parnervector is q. Except in c discussion of modelingeparneters will not
be explicitly rred to as they are primarily predetrmincd constants in design.
D e s'n Functlons: Design functions define meaningful information about the
design. They are evaluated using the design variables and design parameters
discussed earlier. They establish the mathematical model of the design problem. These
functions can represent design objective(s) andlor constraillts. As iname implies
design 0ective drives the search for the optimal design. The satisfaction of the
traillts establishes the validity of the design. If not explicit1y statede designer
is responsiblc for idntifying thc objectivc and constraints. Minimize the mass of the
6 I N T R O D U C T I O N
structure will translate to objectiveU 1ction. The sin the material must be less
than the yieId strength w i1l translate to a constraintfunction. In many problems it is
possible for thc same function to switch roles to provide diffcrcnt design scenarios.
ObJectlve Function The traditional design optimization problem is dcfind
using a single objective function. le format of this statemcnt is usually to minimizc
or maximize some quantity is calculated using somc dcsign function. This
function must depend explicitly or implicitly on the design variables. In the
literaturcis problcm is cxpssed exc1usively without loss of generaIity as a
minimum problem. A maximum problem can brccast as a minimization problcm
using the ncgativc or reciprocal of thc function used for the objectivc function. ln
the first example in duced earlierc oective is to minimize cost. lerefore
desi function represg cost w ilI be the objective function. In e second se
e objective is to minimize mass. In the third case the objective is to maximizc
machine utilization. le ar of single objective desi is considered mature today.
N o new solution techniques for classical problcms havbecn advanced for somc time
now. Today. much of the work in applied optimi tion is dirccted at expanding
applications to practical problems. ln many cases. this has involved crcative use of the
solution techniques. In abstract mathematical model. the objective function is
reprosented by the symbolf T o indicate its dependence on the design variables. it is
f quently cxpressed f(x" x2.. . . x
n
). A more concisc prcsentation is j{X).
Single objective problcms have only one function dcnoted by f It is a scaIar (not a
V tor). Notc at although the objcctive function depends on P (parameter vector). it
is not explicitly inc1uded in the format .
Multiobjective and multidisciplinary dsigns arc important developments today.
Multiobjtive design or multiple objecti ve design. refersusing sevcraI dierentdesign
functions to e sh for opnal dign. GeneraIlyey are expecd b e
conflicting 0cctives. T hey could also be opcrating objectives. le current approach
to e solution of these problems involves standard optimization proced applicd to a
single ( onstructed objective optimization problcm based on the different multiple
objectives. A popar approach is to use a suitably weighdlin r combination of
multiplc objectives. A practicallimitation wiis approach is .echoi ofweights us
in the model . This approach has not been cmbraced widely. A n altemative approach of
recognizing a prcnectiveand solving a singlc objectivc problem with additional
constraints based onningovenctions can usually geneaccep le
solution. In multiobjective problen
l
l
Y
2
4
4

y
f
J
1.1 OPTIMIZATION F U N D A M E N T A L S 7
Constraint Functlons: As design functions these will be influenced by the
design variables. T he format of these functions requires them to be compared to some
numerically limiting value at is established by design requirement or the designer.
T his value remains constant during thc optimization of the problcm. A w cl1-describcd
dcsign problem is expected to inc1ude scveral such functions. which can be
represented as a vcctor. T he comparison is usual1y set up using the threc standard
onal operators: = S and . Consider our first example. LetfimJ(X) represent
e function calculates the volume of the new soda can w e are designing. le
constrnt on the dsign can bc expresscd as
IIJ(X) = 500 c m
J
In tlIe second example.letm2(X) bthe function that calculates the deflection of the
beam under the applied load. T he constraint can be statcd as
ft11l2(X) :S; I m m
Thccons int functions can be c1assified as eqllality constraints [likefunl(X) aboveJ
or inequality consnts [likefun2(X)].
Problems without constraints are tcrmcd unconstrained problems. If constraints are
present then meeting them is more pmount an optimizalion. Constraint
satisfaction is necessary beforc the dcsign established by currcnt value of the
desi variables is considered valid and acceptable. If constraints are not ssfied
then there is no solution. A feasible design is one in which all of the constraints are
satisfied. A n optimal solution is one at has met the design objective. A n optimal
dcsign must be fcasiblc. T he design spacc cncloscd by thc constraints is called the
feasible domain. Dcsign space is described a few paragraphs bclow.
Equallty Constralnts: Equality constraints are mathematically neat and easy to
handlc. Numericallyey J1uire more effort to satisfy. They are also more restrictive
on tlIe designthey limit the region from which the solution can be obtained. The
symbol representing equality constraints in the abstract model is h. T here may be
more than one equality constraint in the design problcm. A vector representation
for uality constraints is introduced through the following representation. [8] [h
l

h
2
. . . . h]. and h
k
: k = 12 . . . .1 are ways of identifying e equality constraints. le
dependence on the design variables X is omitted for convcnience. Notc at the length
of the vector is 1. A n important on for distinguishing the equality and inequality
constraints is at they arc mnnipulated differently in the senrch for the optimal
solution. The number n of design variables in c problcm must be greatcr the
number of equality constraints 1 for optimization to take place. lf n is cqual to 1 . then
problem will be solved without reference to the objective. ln mathematical terms
the number of equations matches e number of unknowns. If n is less than 1 then you
have an overdetermined scl of relations which could result in an inconsistent problcm
definition. T he sct of cquality constraints must be linearly indcpendent . Broadly this
8 INTAODUCTION
implies you cannot obtain one of the consaints from elementary arithmetic
operations on the rcmaining constraints. T his serves 10 ensure that the mathematical
search for solution will not fail. These tcchniques are based on methods from Iincar
algebra. In the standard format for optimization problems the ua1ity consaints are
written with a 0 on the right-hand side. This means e equality constraint in thc
first example will be exp ssed as
hl(X):funl(X) - 500 = 0
In practica1 problems equality constraints are rarIy encountcred.
Inequallty Constraints: Inua1ity consaints app rmore naturally in problcm
formulation. Inequality constraints also provide more flexibility in dcsign selection.
The symbol represcnting inequality cons'aints in the abstract model is 8. There may
be more onc inua1ity constraint in the design problem. The vector
reprsentation for inequality constraints is similar to what w e havseen before. T hus
[G) [8182 . . . 811/]and : j = 1 2 . . . m arc ways of identifying Ihe inequality
consaints. m reprents e number of inequa1ity constraints. A Il design functions
explicitly or implicitly depend on the design (or independent) variable X. 8 is used to
descriho bo loss thun or cqual 10 ) d grealcr 01" equlll 10 () constralnts.
Strictly greater ( > ) and strictly Icss th( < ) are not used much in optimization
because solutions are ua11y expected to lie at e constraint boundary. In the
standard format all problems arc expressed wie S relationship. Moreover the
right-hand side of e S sign isO. le inequality constraint from the second example
ftm2(X) is set up as
81(X):n2(X)-1 O
In case of inequality constraints a distinction is madc as to whether the dcsign
variables lie on the constraint boundary or in interiorof the region bounded by E
constraint. If the set of dsign variables lie on the boundary of e constraint
mathematically this expresses e fact that constraint is satisfied with strict equality
isg = O. T he constraint acts like an equality constraint. In optimization
terminologyis particular consaint is referred to as an active constraint. If the set
of design variables do not lie on the boundary that is they liinside e region of thc
constraints they arc conside d inactive constraints. Mathematica1lye constraint
satisfies e rclation g < O. A n inequality consU'aint can thercfore be either active or
lllactJve.
Slde Constralnts: Side consaints are a necessary part of the solution techniques
escia1ly numerical oncs. They express the range for the design variablcs. Each
design variable must be bound by numcrical va1ues for its lower and upper Iimit. T he
designer makes this choice based on his anticipation of an acceptable design.
4
F
1.1 OPTIMIZATION FUNDAMENTALS 9
Design Space: le design spacee space at w il be searched for optimal
design is the Euclidean or Cartesian n-dimensional space generated by the n
independent design variables X. This is a genera1ization of the three-dimensional
physical space with which w e are familiar. For ten dsign variables it is a
ten-dimensional space. This is not easy to imagine. Il is a1so not casy to express this
information rough a figure or graph because of the Iimitation of the
three-dimensional world. However if e d:sign variables are independenten the
l1-dimensional considerations are mere extrapolations of thee-dimensiona1reality.
Of course w e cannot geometrica1ly definthem though we will be working with thc
numbers. The side constraints Iimit the search gionimplying that only solutions at
lie within a crtain region will be acceptable. They define an n-dimensional
rectangular region (hypercube) from whichefeasible and optimal solutions must be
chosen. Later we w il sat the mathematica1 models in optimization are usually
described by nonlinear relationships.11e solutions to such problems cannot be
ana1ytically predicted they are typica1ly govemed by e underlying numerical
tec ique used to solve them. It is neccssary to ( ict the solutions to an acceptable
region. The side constraints provide s ready mechanism for implementing this limit.
Care must be taken that thcse limits are not imposed overzealously. Thcre must be
reasonable space for numerical techniques 10 operate.
The Standard Format: T he above definitions a1low us to assemble Ihe general
abs .ct mathernatical model as
Minimize f(x. x2 . . . x
n
)
Subject to: hl(x" X2XII)=O
h2(x. x2 . . . x
n
) = 0 (1.2)
. 1


l

h/(XI' X2.. XI I) = 0
8.(X" X2.. Xn) O
82(XI' X2 Xn) S 0 (1.3)
gm(XIX2.. Xn) S O
xl SXj S x i i = 12...n (1.4)
The same problem can be expressed concisely using the following notation:
Minimize f(x. X2 . . . X
n
) (1.5)
Subject to: hk(xt X2X
n
) = 0 k = 1 2 . . . I (1.6)
(X" X2 . . X
I I
) S 0 j = 12. . . . m (1.7)

10 INTROOUCT10N
xlsxjSx i= 12... n
Exploiting vector notation the mathematical model is
Minimize j(X) [X].
Subject to: [h(X)]1 = 0
( X)]tn S 0
X10wSX S Xup
(1.8)
(1.9)
(1.10)
( 1.11)
(1.12)
T he above mathematical model expresses following standard format of the
optimizon problem expressed in naturallanguage:
Minimize the objective functionj subject to 1 equaity constraints m incquality
constraints with e n design variables Iying between pr cribed lower and
upper limits.
le techniqucs in is book will apply to thc problem describcd in c abovc format.
T o solve any specific design problem it is requircd 10 refunnulatc the problcm in the
above manner so methods can be applied directly. Also in this book the
techniques are developed progressively by considering the standard model wi
duced elements. For example thunconstrained problem w il be explod first. The
equality cons ints a considered next followed by the inequality constraints. d
finally the complete model . This represents a natural progression as prior knowledge
is used to develop additional conditions need to be satisfied by the solution in
e instances.
1.1.2 Mathematlcal Modellng
In this sectione three design problems i' Oduced earlier will be anslated to e
standard format. The mathematical m o de1 will first be identified. The second problem
requires information from a course in mechanics and materials. This should be within
.e purview of most engineering students.
Example 1.1 N e w consumer research indicates at people like to drink about 0.5
liter of soda pop at a time during the summer months. The fabrication cost of the
redesigned soda can is proportional 10 the surface area and can be estimaled al $1.00
per square meter of lhe matrial used. A circular cross section is the most plausible
given current looling available for manufaclure. For aesthetic reasons tl1e height must
be at least twice tl1e diameter. Studies indicate at holding comfort requires a
diameter between 6 and 9 cm.
Figurc 1.1 shows a sketch of tl1e can. In most product designs particularly in
engineringthe model is easier to develop using a figure. T he diameter d and the
height h are sufficient to dcscribe tl1e soda can. What about the tl1ickness t of e
~
p
J
:<:
d
1.1 OPTIMIZATION FUNOAMENTAlS 11
d
h
Flgure 1.1 Example 1.1- Design 01 a new beverage can.
matcrial of the can? What arc lhe assumptions for thc dcsign problcm? One of the
assumplions could be lhat t is small cnough to be ignored in thc calculation of thc
volume of soda in e can. Another assumption is that e material quired for lhe can is
only the cylindrical surfac. 'hetop and botlom of thc can will be fitted with cnd caps
atw provide e mechanism by which the soda can bc p o w. is not part ofis
dcsign problem. In the tattempt at dcveloping the mathematical model w e could start
out by considering equ tities identified ove as des1variables:
Design variables: d h t
Reviewing the stalement of the design problem one of the parameters is the cost of
material per unit area is given as l(t per sqU8I centimeter.bis is idenlified C.
During the search for the optimal solution is quantity will be hcld constant at e
given value. Note that if this valuc changes lhen e cost of thc can will
correspondingly change. T his is what w e mean by a design p ameter. Typically
change in parameters w il cause the solution to computed.
Design parameter: C
The design functions will include e computation of the volume enclosed by the can
and the surface area of the cylindrical section. The volume in the can iscf2h14. le
surface area is 1tdh. T he aesthctic constraint requires at h ~ 2d. The side consaints
on diameter are prescrib in e problem. For complelness the side constraints
on other variables have to be prescribed by the designer. W e can formally set up
the optimization problem
Minimize f(d h t): C7rd1z (1.13)
SubjecllO: h Mht): 7rd
2
/1/4 - 500 = 0 (1.14)

12 INTRODUCTION

/
1.1 OPTIMIZATION F U N D A M E N T A L S 13
Intuitively therc is some concem with the problem as expressed by Equations
(1.13(1.15) even though description is valid. H o w can the value of the design
variable t be estab1ished? The variation in t does not a'ect the design. Changing the
value of t does not changef h
l
or glHenceitcannot be a design variable. (Note: the
variation in t may affect the value of c.) lf is werc a practical design problcm then
cans have to be designed for impact and stacking strength. In C et will
probably be a critical design variable. This will require severadditional s clural
constraints in the problem. It could serve as an interesting extension to this problem
for homework or project le optimization problem after dropping t d expressing
[d. h] [XIX2] becomes:

4

I
V
-
1
-
-
gM.h.t): 2d- h S O
6 S d s 9 ; 5 s /t s20; O.OOIStsO.Ol (1.15)
Flgure 1.2 Example 1.2- Canlllever beam.
Minimize f(XIX2): c : ( 1.16)
(1.17)
gl(lXU: 2x1 - X 2 S 0 (1.18)
lntuitivelye larger the value of F. the greater iscro s8-Secl lio na alare = anecsv
t .0 ha nd le it. 11 c a n b e i l y concluded ati f F w a d si
bsel at its lower limit a8 lhe mass of the beam will be direclly proportional 10 e
area of croS8 8cc(Iun. N o techniques are ruired f. or lhis conclusi. on. B y . our definition
F is a go. od candidate as a parneter rather as a design variable. Similarly the
larger the value of L e g ateris the momentF generates about the cantilevered
end. lis will require an increase in e area . of cross-section. T herefore L is not a
good choice f. or e design variable either as the best choice f. or it will bc il. ower
I imit. T o represcnt material as a design variable. w e necd to use its SlruClural propcrties
in e formulalion ofthe problem. Typically. a malerial is characlerized by its specc
Subjecllo: hl(xlXv:ru/ 4 - 5 0 0 = 0
6 SX) S 9; 5 SX2 S 2 0
The problem presenled by Equati. ons (1.16)(1.18) is mathematical m o de f. or
thdesign problem expressed in the slandard format. For this problem no extranc. ous
informali. on was nd to set up e problem except for some geometrical relations.
E xample 1.2 A cantilever beam needs 10 be designed to cry a point load F al the
end . of the beam . of lenglh L. T he cross section of the beam w iII be in the shape of the
letter 1 (ferred to as 1b m). le beam should meet prescribed failure criteria.
Thcre is also a I imit on its deflecti. on. A beam of minimum mass is quired 10 be
digncd.
Figure 1.2 shows a side view . of the beam carrying the load and Figure 1.3 a typical
cr. oss snofthben. T he I-shaped cross section is symmetric. T he symbols d tw.
b
r
and t rc. orrespond to the dep.ofthe bcam thickness of the web width of thflange
and thickness . of the flange respectively. Thesquantities are sufficient to define E
cr. oss section. A handbo. ok or textb. ook on strcngth of materials can aid e
dvelopment . of thdesign functions. A n important assumption for this problem is
w e will be w . orking within the elastic I imit of thc material where there is a I inear
relati. onship between e stress and the strain. A lI . of the variables identified in Figure
1.3 will s ngly affect the soluti. on. T hey are design variables. T he applied f. orce F
will also directly affect the problem. So will its locati. on L. H . ow about material?
Stecl is dcfinitely supcri. or 1.0 c. opper for the beam. Should F L and the material
pr. operties be included as design variables?
bt
I ~
' l
4 tW -

Flguro 1.3 Example 1.2- Cross-sectlonal shape.

14 INTRODUCTION
weight y its value ofthe modulus of elasticity. E. its modulus of rigidity. G. its yield
limit in tension and compr sion.yie'd. iyield in sh yield. its ultimate streng
in tension.ult. and its ultimate strength in shear.ullOptimi lion techniques identify
diffe nt va1ues of the design variables. If E is the variable used to present E
material design variable in a problem. then it is possible the solution can quire a
materia1at does not existat is. a va1ue of E still undiscoverd. For a11 intents and
p poses. is is beyond the scope of the designer. Again. materia1 as a parameter
makes a lot of.sense and e optimization problem should be reinvestigated if F
or the material changes. Concluding from is discussion on modeling. the following
I ist can be established for this example:
Design p1eters F . L . {y. G. Y'eld'1:yi eld' u11' 1: E
Design variables: d t
w
' b r ' I
f
In the devlopment of the mathematical model standard technical definitions
used. Much of the information can be obtained from a mechanical engineering
handbook. T hfirst design function is weight of the beam. This is the product of
y L and Ac where Ac is e area of cross section. The maximum slress due to bending
""nd can be calculated as the product of FLdI21cwhereis thc momenl of inertia
about thc ccntroid of the cross section along the axis parallel to the f1ange. The
m i m u m shear stress in cross section is expressed as FQcllc1w wherc Qc is first
moment of area about the cenid parallcl to thc flange. The maximum deflection
of the beam w i11 be at the end of the beam calculated by the expression F L
3
/3El
c
' Ac
of the cross section of the beam is 2brtr + Iw (d - 2tf). The first momcnt of area Qc can
bc calculated as 0.5b
f
tf (d - tr) + 0.51
w
(0.5d - tf)2. Finally. c momcnt of inertiais
obtained from (b112) ((b
r
- tw )(d - 2tf)3/12). In followingfor convcnience
w e will continue to use Ae Qc' andinstcad of detai1ing their dependence on e
design va1ues. Associating x with d X2 with t
w
' X3 with b
r
and X4 with lr so a t X =
[xX2X3' x41e problem in standard format is
Minimize .ftX): YLAC ( 1.19)
Subject to: g(X): FL x /21c- yie'd 0
g2(x ): FQJ1.x: x2 - 1:yie e
g3(X):

0.0 1 x 0.25; 0.00 1 X2 0.05;
0.0 1 X3 0.25; 0.00 1 X4 0.05
(1.20)
T o solve is problem F must bc prescribed (10000 N) L must be givcn (3 m).
Matcria1 must bc selected (stee1 := 7860 kglm
3
; O"yield = 250 E+06 N/m
2
;yicld = 145
E+06 N/m
2
). A1so the maximum dct1ection is prcscribcd = 0.005 m).

h
F
d
f

J
d
J
i
J
d
h
h
b
M
U
F
F

1.1 OPTIMIZA TION FUNDAMENTALS 15

This is an example with four design variables three inequality constraints. and
no equality constraints. Othcr vcrsions of this problem can easily be formulated.
It can be reduced to a two-variablc problcm. Standard failure criteria with respect
to combined stresscs or principa1 strcsses can bc formulated. If the cantilevered
end is bolted then additiona1 design functions regarding bolt failure need to be
exarned.
Example 1.3 M y P C Company has decided to invest $12 million in acquiring severa1
ncw Component P1acement Machines to manufacture dierentkinds of m oeoards
for a new generation of personal computers. Three models of these machines are under
consideration. T ota1 number of operators available is 100 becausc of e local labor
market. A floor space constraint needs to be satisfied because of e different
dimensions of se machines. Additiona1 information relating to each of e
machines is given in Table 1.1. e company wishes to determine how m y ofeach
kind is appropriate to maximizc the number of boards manufactud perday.
lt is difficu1t to use a figure in this problem to sct up our mathematical model . The
number of machines of cach model necds to be determined. T his will serve as our
design variables. Let XI represent thnumber of Component Placement Machines of
Model A. Similarly X2 w i1l be associatcd with Model B and X3 with Model C.
i
!
i

D igll variubles: xX2X3
T he infonnation in Table 1.1 is used to set up the design functions in terms of e
design variables. A n assumplion is made that all machines are run for three shs. T he
cost of acquisition of the machines is the sum of e cost per machine mu1tiplied by
numberof machines (g 1)' Thmachines must satisfy e floor spacc constraint (82)
which is defined without details in Eq. (1.23). The constraint on the number of
opemtors is three timcs sum of e product of the number of machines of ch
model and the operators per shift (g3)' T he utilization of each machine is the number
of boards per hour times the number of hours the machine operates per day. T hc
optimization problem can be assembled in following form:

f
s
M aimize f(X):18 (1.21)
or
T Minimize f(X):- 18 x 55 x x -18 x 50 X x
2
- 21 X 50 x X
3
Suct to: g(X): 400OOOx + 600.000X2 + 700000X3 120 000 (1.22)
g2(X): 3x -X2 + X3 30 (1.23)
83(X): 3xI + 6X2 + 6x
3
100 (1.24)

XI 0; X2 0; X) 0
E
r
i
-
5
2
2
J
J
y
d

-

17
9
OPTrMIZATION FUNDAMENTALS
6
1.1
5
INTRODUCTION
Equations (1.21)(1.24) express mathematical model of the problem. Note tin
this problem there is no product bcing dcsigned. Here a strategy for placing order for
numberof machines is being dctermined. Equation (1.21) illusthetranslation
of the m imization objective into an uivalent minimizing one. T he inequality
constraints in Equations (1 .22)(1.24) are different from the previous two exnples.
He the right-hand side of the S operator is nonzero. Similarly the side constraints
are bound on the lower side only. This is done deliberately. T here is a significanl
difference between isproblem and the previous two. All of the design functions here
are linear. This problem is c1assified as a linear prograrnming problem. The solutions
10 these typcs of problems are very different than thc solution 10 thc first two
cxamplswhich are recognized as nonlinar programming problms.
1.1.3 Nature of Solution
le e cxarnpIsin cprevious section arc used to discuss e nature of solutions
the optimization problem. Exarnples 1.1 and 1.2 dcscribe e nonlin rprogramming
problcm which forms the major part ofis book. Exarnple 1.3 is a lincar programming
problem which is very significant in dccision scicnccs but ra in product design. Its
inc1usion here while necessary for complctcness is also important for understanding
contemporary optimization techniques for nonlinear programming problcms. Exarnples
1.1 and 1.2a dealtwith ftrSt. Between the lwo we notice that Exarnple 1.1 isquite simple
relative to Exarnple 1.2. Second the four variables in Example 1.2 make itdifficult to use
i1lustration to establish some of thc discussion.
16
representation 01 Example 1.1.
boof thesc functions. le uality constraint gl is lincar. ln Equation (1.18) is is
evidentbause the desi variablesappcar byemselveswithout being raised 10 a powcr
o1 . As we develop e techniques for applying optimizationis distinction is
important to keep in mind. It is also significant graphical representation is
typically restricted to two variables. Forree variables we need a fourth dimension
10 resolve the informalIon and three-dimensional conlour plots are not easy to
iIIustrate. The power of imagination is necessary to overcome se hurdles.
1problemreprcsented in Figure 1.5 providcs opportunity to identify the graphical
solution. First the feasible region is identified. In Figure 1.5 is is Eualitycons int
above the incquality consint . Bothfeible and oplimal must occur from this :gion.
A ny solution from is region is an acceplable ( sible) solution. There are usually a
large number of such solutions infinile solutions. If optimization is involved thcn
thesc choices must be rcduced to the best one with spcct to some criteria the
objeclive. 1n this problemlfigure the smallcst value offis desired. T he lowest valuc
of f is just less than the contour value of 308. lt is at e intersection of the two
constraints. While the value of f needs to be calculated the optimal values of the
design variables read from the figure are about6.75 and 13. 5 respectively. Another
significant item of information obtained from Figure 1.5 is gl is an active
constraint . While there are infinite feasible solutions 10 thc problem the optimal
solution is Ultique.
Flgure 1.4

E
Z

5
SOLUT10N T O E X A M P L E 1.1 T hc simplcst determination of nonlinearity is
through a graphical P sentation of c dcsign functions involved in thc problem.
lis can be donc sily for ODe or two variables. If the fUDCtiOD does not plot as a
straight line or a planeen it is nonlincar. Figu 1.4 shows a three-dimcnsional plot
of Examplc 1.1. The figure is obtained using MATLAB. Chapter 2 will providdctailcd
instructions for drawing such plots for graphical optimization. T he threc-dimcnsional
pr entationof Exarnple 1.1 in Figure 1.4 does not real1y enhance our understanding
of the problem or the solution. Figure 1.5 is an altemate repscntation of the problem.
This is thone encouraged in this book. Thc horizontal is rep scnts the diameter
(XI) and vertical axis e height (xu. 1n Figure 1.5 the equaliconstraint is
markcd appropriately as h 1. Any prof valucs on is line will give a volume of 500
cm
3
The inequa/ity constraint is shown as g 1. The pair of values on this line exact 1y
satisfies thc aesetic requi ment. Hash lines on the side of the inequality constraint
cstablish the disallowed gion for the design variables. le constraints are drawn
thicker for emph is. T he scaled 0 'eclivenClion is repr end through sevcral
labelcd contours. Each contour is associated with a xed value of the objcctive
function and these values are shown on the figure. le range of the two axes
establishes the side COllStraints. The objective functionf and the equalily constraint h 1
'c nonlinear sincc they do not plot IIS straight lines. Rcferring to Equations (1.16) and
(1.17) this is substantiated by the pl'OduCIS of the two unknowns (design varillbles) in
19
A n interesting observation in the above set of equations is at there are only
two design variables but three constraints. This is a valid problem. If the
constraints were equality constraintsen this would not be an acceptable
problem definition as it would violate e relationship between the number of
variables and the number of equality constraints. Therefore the number of
inequaliconstraints are not related to the numbrof design variables used. This
fact also applies to nonlinear constrnints. Figure 1.6 is a scaled plot of c
functions (1.25)(1.28). Choosing the first quadrant satisfies the side constraints.
'he linearity of all of the functions is indicated by the straight-line plots on the
figure. Again several contours offare shown to aid the recognition ofthe solution.
All of the inequalities are drawn with hash marks indicating the nonfeasib1e side
of the constraint . The feasible region is enclosed by the two axes along with
constraints g 2 and g3' The objective is to increasthe value off(which is indicated
by the contour levels) without leaving the feasible region. The solution can be
(1.28)
1.1 OPTIMIZATION FUNOAMENTALS
83(X): 3xI + 6x2 ~ 70
X 2 ~ 0 X
1
~ O ;
J
t
u
b
/
/
d
r

v
d
r
f
L
b
f
f
/
y
J
X
INTROOUCTION 18
20
{
E
H
U
}
-
Z

3
4
25 20
Conlour plol 01 modllled Example 1.3.
1S
number 01 Model A
10
Flgure 1.6
5
9
S O L U T I O N T O E X A M P L E 1.3 In contrast to Example 1.1 all of the relationships
in mathematical model expressed by Equations (1.21)(1.24)are linear. T he
word equation is generally used to describe the equivalence of two quantities on either
side of the = sign. In optirnization w e come across mostly inequalities. e word
equatioll will be ed to include both of these situations. In Example 1.1e na
of solution was explained using Figu1 .5. In Example 1.3 the p sence of thr
design variables denies is approach. Instead of designing a new problem suppose
the company p sident has d ided to buy five machines of the third type. T he
mathematical model with two design variables Xl and X2' is reconstructed using the
information that X3 is now a parameter with the value of 5. Substituting for x3 in the
model for Example 1.3 and cleaning up the first cons int :
M imize
Subject to:
(1.25)
(1.26)
( 1.27)
H137
8.5
j(X): 990 X X I + 9x X 2 + 5250

Conlour plol 01 Example 1.1.

gl): O.4xl + 0.6x2 ~ 8.5
7.5
dlameler (cm)
g2(X): 3xI - X2 ~ 25
7
Agure 1.5
6.5
5
6
20 lNTRODUCT10N
spottcd at the intersection of constraints g2 d g3. The values of the design variIcs
at the solution can be read off from thc plot. It is not over yet. Before proceeding
further il is neccssary to acknowledgc that a solution of 6.5 machines for Model B
is unacceptable. The solution must be adjusted to the ncarest integer values. In
aClual practice. this adjustmcnt has to bc made without violating any of the
constraints. The need for integer variables is an important consideration in dcsign.
V ables at arquired to have only integer values belong to e type of variables
called discrete variables.'his book most1y considers continuous variables. Almost a11
of the mathematics an engin r encounters especially academically belong to E
domain of continuous variables. Thc assumption of continuity providcs very fast and
cfficicnt techniques for the search of constrained optimum solutions as w e will
develop in this text . Many real-I ifc designs especially those that use off-the-shelf
materials and components determine discrete models. Discrete programming and
integer programming are disciplines that address these models. The nature of
search methodsat is methods used for searching e optimal solution in discrctc
prograrnming is very different from at in continuous programming. M a n y of th
methods are based on the scanning and replacement of the solutions through
exhauslive search of the design space in the design region. In this book disctc
problems arc typically handled as if they were continuous problems. The conversion
to e discrcte solution is the final part of the design effort and is usually performed
by the designer outsid of any optimization technique w e develop in this book.
Chapter 8 discusses discrete optimization with examples.
Getting back to e linear programming problem just discussed it should also be
noted that only the boundary of the feasible region will affect the solution. H e is a
way that will make is apparcnt . Note that the objctive function contours arstraight
lines. A different objective function will be displayed by parallellines with a different
slope. Imagine these Iines on the figure instead of the actllal objective. Note at the
solution always appears to lie at thintcrsection of the constraints that is thcorners
ofthedsign spacc. Solution can nevcr c o mfrom inside of the fcasible gionunlike
in the nonlinear programming problem which accommodates solution on the
boundary and from within the feasible region. Recognizing thc natu ofsolutions is
important in developing a search procedure. For example in linear prograrnming
problems a technique employcd to search for the optimal dsign need only search thc
boundaries particularly the intcrsection of the boundaries whereas nonlinear
problems the search procedure cannot ignore interior values.
1.1.4 Characterlstlcs of the Search Procedure
le search for the optimal solution will depend on e nature of the problem being
solved. For nonlinear problems it must consider the fact that solution could lie inside
the feasible region. Example 1.1 throllgh Figurc 1.5 will provide e context for
discussing the scarch procedure for nonlincar problems. Simi 1arIy the characteristics
1.1 OPT1M1ZAT10N FUNDAMENTALS 21
of the search process for linear problcms will be explored using Ex lple 1.3 and
Figure 1.6.

p
d

y
d

d
y
d
p
d
d

h
u
u
k

x
y
d
y
M

d
r

i
Nonllnear Prob/ems: Except for a few c1asses of problcms solutions to non1inear
problems are obtained through numerical analysis. Through computr code
numerica1 analysis becomes numcrical techniques. The methods or techniques for
finding the solution to optimization prob1ems are called search methods. In applied
mathematics and numerical t hniques these referred to as iterative techniques.
Implicit1y this means seveml tries will be necessary beforc the solution can be
obtained. A1so implied is that cach try or search is executed in a consistent manner.
Information from previous iteration is utilized in the computation of values in the
present sequence. This consistcnt manner or press by which the search is carried out
and solution determined is ca1ed algorithm. This term a1so refers to the
tr slation of the particular search procedure into an ordered suence of step-bystep
actions. The advantage is derived mostly in the conversion of these steps through
some computer 1anguage into code at w i1l execute on ecomputcr. Developing E
a1gorithm and trans1ating it into code that w il execute in MATLAB w il be a standard
approach in is book.
The search process is started typica1y by trying to guess thinitialdesign solution.
The designer can base this selection on his experience. O n sevcm1 occasions the
success of optimization may hinge on his ability to choose a good initial solution. The
search method even when used consistent1y responds different1y to the particular
problem at is being solved. The degree and type of nonlinearity m a y frequent1y
cause e method to fail assurning there are no inconsistencies in problem
formu1ation. Oftennumber of design variablese number of cons intsas we1
as a bad initial guess p1ay a part in unexpected behavior under standard
imDlementation of the method. This has led to the creation of seveml differcnt
techniques for optimization. Practica1 experience bas a1so suggested at a class of
problems respond well to certain a1gorithms.
The mcthods for e search or itmtive methods. are current1y the only way to
implement numerical solutions to nonlinear problems irrespective of the discipline in
which these problems app r. 1 e problems are p1entifu1 in design optimization.
optimal contro1. structural and t1uid mechanics. quantum physics. and astronomy.
W hi1e optimization techniques are applied all of the noted disciplines. in ditional
design

23
at a distan sS.. This point satisfies the constraint h.. A ird sY would be to
choosc X.This lies on the constraint boundary of g.. lt is to be noted that all E
choices are not feasible. Which point is chosen is dccided by the rules of a particular
algorithm. Assume point X) is the next point in is discussion.
Point X) now becomes new point xoA n e w h dircction is dennin.
Point X
2
is 10d. A noth Suenceof calculations

next iteration yields

e solution. The essence of updating e desi during suc ssive iteration lies in two
major computations thc sea h directioll 8/ and stepsize . lC change in e design
vector definIas .6Xwhich represents veclor diffcrence for exnpleX
2
- X. is
obtained asS/. For each iteration the following suence of activity can be identified:
The above is the basic structure of a typical optimization algorithm. A complete
one will indicate m ner in which S and are computed. Il w il1 incJude tests
which will determinc if solution has bcen reached (convergence) or if the
procedure needs to be suspended or restarted. The diffcrent techniques to bexplorcd
in this book mostly differ in eway S is established. Take a look at Figure 1.7 again.
At X
o
there are infinite choices for the search direction. To reach the solution in
onable time an acceptable algorithm will try not to search along directions do
not improve the current design. lere are many ways to accomplish is strategy.
1.1 OPTIMIZATION F U N D A M E N T A L S
ChooseX
o
Identify S
Determine
A X = S
XM W = Xo+.6X
Set Xo X nen
GoToStepO
8tep 0:
Step 1:
:r;-:.
I N T R O D U C T I O N
FJgure 1.7
g 8.5
A typical ssarch method.
7.5
dlamoler (cm)
22
20
5
6
{
E
S
E
m
g
z

Linear Programming Problem: It was previously mentioned the solution to
se problems would be on the boundary of feasible gioo (also called the
jible domain). In Figure 1.6e es and some of the cons ints detennine this
region. The lalter is distinguished also by epoints of interscction of the constrainLs
among themselveseorigin and e intersection of the binding (active) coostraints
with the axes. These are vertices or corners of the quadrilateral. Essentially the design
improves by moving from one of these vertices to the next one through improving
feasible designs. To do that you have to start at a feasiblcomer.This will be explored
more fully in the chapter on linear programming. It is to be emphasizedatonly with
two variables can we actually see the geometric iIIustration of the technique is
the selection of vertices as the solution converges.
This part of Chapter 1 has dealt with the introduction of the option problem.
While it was approached from an engineering design perspcctive the mathematical
model in abstract terms is not sensitive to any pticular discipline. A few design
rules with resp tto specific examples were examined. TIle standard mathematical
model for optimization problems was established. T w o broad cJasses of problems
initial starting ess is e point marked xoRemember when solving
optimization problem. such a figure does not ist. If it diden we can easily obtain
the solution by inspection. lis statement is espccially true ifere are more than two
variables. In the event th are two variables is no reason not to obtain a
graphical solution (Chapter 2). Thc mcthods in optimization rely on thc dircct
ex polation of the concepts and ideas develod for two variables to any number of
design variables. That is the reason we will use two variables to discuss and design
most of the algorithms and methods. Moreover using two variablesconcepts can
also be reinfor d using geometry. The design at the point X
o
is not feasible. lt is not
useful to discuss whether it is an optimal solution. At least one more iteration is
necessary to obn the solution. Consider e next solution at X
I
. Most methods in
optimization will attempt to move to X I by first identifying a search direction S 1This
is a vector (same dimension as design variable vcctor X) poim Xo loX
Ie
actual distance from xo to X
I

call it XoX.. is some multiple of SI' s SIX

1
0nce
again is not feasible altbough it is an improvement on xo since it satisE
incqualily constrainl gl (lying in lhe inlerior of e constraint). Altemately using lhe
same scal'ch dircction S 1 another techniqueJm eodmay choose for its next point X 1'
.1""':-......1:

n ' '
24 INTRODUCTION
linear and nonlinear. were introduc. Thegeomeical characteristics of the solution
for thesc problems were shown rough graphical description ofeproblems. A n
overview of e search techniques for obtaining the solution to these problems w
also iIIustrated. It was a1so mentioned at e solutions were to be obtained
numerical . primarily through digital computation.
The numerical methods used in is area are iterative methods. Numerical solutions
involve two components: algorithm that will stablish iterative set of
calculations. and second. the translation of the algorithm into computer codes. using
aprogramming lang 'ge. This finished code is referred to software. Until recently
in engineering education as well as practice. the compurlanguage of choice was
predominantly FORTRAN with a I ittle Pascal and C. T he computational environment in
which it was delivered and practiced wemai'ame computing systems. Today
e scene is very different. Individual personal computers Cs)running Windows
Macintosh LINUX or other flavors of UNIX operating systems are now the means
for technical computing. T hey possess more power than the old mainame computers
and they are inexpensive. T he power of the graphical user interface (GUI)e way
e user interacts wi the computer and e software h revolutionized the creation
of software today. N e w programming paradigmsa 11y 0 'ecl oriented
programming have led to new and improved ways to develop software. Today
cnginccring softwarc is crcatcd by Buch gcncl'al programming languagcs IlS C C++
Java Pascal and Visual Basic. Many software vendors develop their own extensiol1s
to the standard language providing a comple1environmentfor software devlopment
by e user. This makes it very convenient for users who are not necessarily
programmers to get eirjob done in a reasonab1y short time.
One such vendor is M aW oswitheir flagship product M A TLAB a10ng with a
complementary collection of toolboxes at deliver specific discipline-oriented
information. In isbkMATLAB is used exclusively for numerical support. N o prior
familiarity with MATLAB is expected though familiarity with the P C is required. Like
all higher-1cvcllanguages effectiveness with this new programming resource comes
with effort frequency of use. original work and a consistent manner of application.
By the cnd of this book it is hoped the ader will be very familiar with thc u

1.2 INTRODUCTION TO MA11.AII 25

1.2 INTRODUCTlO N T O MATLAB
J
A

2
2

/
/

J
1

MATLAB is introduccd by MathWorks Inc. as the language for technical computing.

Borrowing from the description in y one of its manuals MATLAB integrates
mputation visualizatiod programming in an easy-touse environment where
problems and solulions are expressed in farniliar mathematical notation. The typical
uses for M ATI..AD includ
Math and computation
A1goritllm development
Modeling simulation and prototyping
Data analysis exploration. and visualization
Scientific and engineering graphics
Application development including GUI building
T he use ofMATI..AB has exploded recently and so have ifeatures. Tbis continuous
enhancement makes M ATI..AB an ideal vehicle for exploring problems in all disciplines
that manipu1ate mathematical content . This abi1ity is multiplied by application-
specific solutions through toolboxes. To learn more about the latest M ATI..AB. its
features the toolboxes collection of user-supported archives information about
Usenet activities other forums information about books and 80 on visit MATLAIl a1
hup:/lwww.ma1hworks.com.
1.2.1 W h y M A11.AB?
MATLAB is a standard tool for in'oductory and advanced courses in mathematics
engineering and science in many universities around world. In industry itis a too1
of choice for reshdevelopment and analysis. In is book MATLAB'S basic array
element is exploited to manipulate vectors and matrices that are natural to numerical
techniques. In the next chapter its powerful visualization features are used for
graphical optimization. In e rest of this book the interactive featurld built-in
support of MATLAB are utilized to translate a1gorithms into functioning code in a
fraction of the time needed in other languages like C or FORTRAN. Toward eendwe
will inoduce the use of its Optimi tion Toolbox.
Most books on optimization provide excellnt coverage on the ~ h n i q u e s and
a1gorithms and some a1so provide a printed version of the code. T he code is qui d
to be used in mainfmmEor legacy environments that must support a compiler in the
language e code is written which is predominantly FORTRAN. Each student en
anslates the orinted code into a forrn suitable for computation. le next step is to
compile the code debug compilation errors run programand troublhoot
execution crrors before useful sults can be obtained. A n important consideration is
the student is expected to be conversant withe programming language. While
FORTRAN was formally taught to engin rs several y rs agois is no longer true.
7 -
26 INTRODUCTION
Many electrical engineering departments have started to embrace C and C++. PCs are
great for the use of ready-made applications especially those with GUI but its use for
programming in FORTRAN has yet to take off.
M ATI..AB suffers very few of these disadvantages. Its de is compact d itisinitive
to leam. To make problems work requires few lines of code compared to traditional
languages. Instead of programming from eground up stanrd M A TI..AB pieces of code
canbe adedtogeerimplement the new algorithm. It avoids the sndard :pru:
mpilation and link suence by using inte.reter. A nother important ason iscode
po bility. le de written foreP C version ofMATI.AB d snot have to be changed
for the Macintosh or the UNIX systcms.lis will not be true ifsystem-dependent resources
m e d i n ece. As an ilIustration of M ATI.AB'S leaming agility bo M ATI.ABd
applied optimization arevered in this book something very diffict aomplishwi
say FORTRAN and optimization or C++ d optimization. FlermoreIA'..ABlike C or
FORTRAN is a globally relevant product .
1.2.2 MATLAB Installalion Issues
This book does not Care if you are running MATLAB on an individual P C or Mac a
networked P C or Mac or individual or networked Unix workstatioJls. MATLAB takcs
care of this issu. The book assumes you have a working and appropriately Iicensed
copy of MATLAB in the machine in ont of you. O n your personal machine for
installing MATI.AB follow the instructions that accompanid the box. Typically e
system administrator handles network installations.
This book uses M ATt.AB Version 5.2 (until Chapter 4) and Version 5.3 (e rest).
Version 6 has been shipping since the fall of 2000. M A AB version 5.0 and above
should work. If M ATI.AB has changed any built-in commands in newer versions
the one ed in the book it should provide information on the availability of newer
version of commands (or lhrow an error a rare case). The MATLAB code and
examples were created on an Inlel Pentium Pro 200 running the Windows N T 4
operating syslem. The earlier codes were also tested on a D E C Alpha server running
OSFl V4. The M ATI..AB version on the server was 5.1.
In organizations and instilutions M ATt.AB software is usually licensed on a
subscription basis. Users therefore havthe 1atest version of the software in these
situations. In such a case the new version (at time the book is first published) will
be Version 6 (or Release R 12). This book is based on Version 5 (Release RI1). le
code d programming instructions in the book shou1d execute withoUl problem
in the new version. In the new Release R12 on the P C p1atform the default
appearance of M ATI..AB is a composite of several windows presenting different pieces
of information. This is shown in Figure 1.8. This c be customized through the menu
b by View Desktop Layout Command Window Only which also coponds
to e previous 1ease. It is shown in Figure 1.9. lis view is simple and less
distracting for getting familiar wiM ATl..AB. Note e M ATt.AB command prompt
is >>. This window delivers standard operating system commands for saving
printing opening files and so on lhrough menus and a command bar. In order to
T
h'
m.
f.
1.2 INTRODucnON TO M ATl.AB 27
F1gure 1.8 MATLAB opening view.
create files (or code) another graphica1 window is neded. This is eMATLAB M file
editor/debugger window and is shown in Figu1.10. The editor uses co10r for
enhancing code readability. Each of t hese windows spawn additiona1 dia10g boxes (if
nec) f io r. s ctttin 1gfie ature esd u I ri ir
workspace browser.
While this is not so much as an installation issue it w ou1d be appropriale lo discuss e
way k plaI1s to wri andexec M ATI.AB C Gen yere are two ways to
workM ATI.AB: interactivelyand thro'pts. In interactive mode. r n n 1ands
areented duted one at a time in the M ATt.AB cOrnn1and window. Oncesess10n
is over and M ATt.AB exitedthere is no information 1tained on set of commands
were used and what worked (unless a choice was made to save the session). Large
seauen s of cOrnn1aI1ds are usually dicu 1u.o k P ack of inE eracUv sess10n.
4o o p t i
file in M ATI.ABThisis called a script m -ftle. Most MATLAB fillled m -ftles. T he
m is a meextBIESion usually mserved for MA-I-LAB.Any code changEsmemade thmush
eeditor. the changes to efile saved and e filcexecud in M ATt.AB again. E x cutin
MATt.A Btl ghsc dp t mle e si spredo m 1 lin
tore vert to interactive mode any time.
1.2.3 Using MATLAB the First Tim e
E g
M A T wil be used inte ive1y in is section. T he au.thor very ny
recornn1ends a hands-on approach to understanding M ATt.ABThis means typmg tt1e
codes yourself. It a1so inc1udes underslanding the syntax rrorsand debugging and
28 INTRODUCTION
is is a Classtoom License fOt insttuctional use only.
Reseatch and cometcial use is ptohib1ted.
To qet statted select " H ATLAB Help" ftom .the Help menu.
Flgure 1.9 MATLAB Command wlndow.
Optim1zation 11
'iI John
Dt. P. VenkataJ;amenl
F1gure 1.10 MaUab Editor/Debugger window.
1.2 INTRODUCTION TO M Al UB 29
rectiring e code. The author is not aware of a single example where programming
was learned without the reader typing de.ln is bk MATLAB code segments are
courier font with e bold style lIsed for emphasizing commands or other pieces of
information. Anything else is recommendations suggestions or exercises. T he
placeholder for command is in italic.
l
Before W e Start: The folowing punctuation d special characters are worth
noting:
A l text after the % sign is considered a
comment. MATLAB ignores anything to
the right of the % sign. Comments are
used liberally in this book to indicate
why's and what's of command or code
usage. D o not overlook them. Y o u do not
have to code them in however probably
two weeks later you w il not remember
why you developed that particular piece
of code.
This is the default MATLAB prompt.
A semicolon at the end of a line prevents
M A AB from echoing the information you
enter on the screen.
A comma will cause the information to
A

.
J
h.
U
> >

F
t

J
-

echo. Default is a comma.

A succession of three periods at the end of
the line informs MATLAB code will
continue the next line. Y o u cannot split
a variable name across two lines.You
cannot continue a comment on another
Iin.
You can s to 'p MATLAB execution and get
back t hc o mm !m d promp t b y typing ^c
(Cl-C)by h 01din gdown
tog eer.
W ilI display information on the various
ways the command can be used. This is the
quickest way to use help.
Opens a help text window that provides
more information on all of the MATLAB
resources installed on your system.
Provides help using a browser window.
v
l

r
E
Y
H
H

E
J
r
p
L

i
A
c
> > h e l p command_name
> > helpwin
> > helpdesk
30 INTAODUCTION
For those of you programming for the first time the equal to sign (=) has a very special
meaning in all programming languages incJuding MATLAB. It is called the assignment
operalor. le variable on thc Icft-hand side of the sign is assigned the value of e
right-hand side. Thc actual eqllal lO operation in MATLAB is usually accomplished by
a double cqual to sign (==).
S o m e Additionsl Features
M A AB is case sensitive. A n a is different than A. This is difficult for
persons w h o arc uscd to FORTRAN. All built-in MATLAB commands arc in
lowercase.
MATLAB does not necd a type definition or a dimension statement to introduce
variables. It automatically cr tes one on first encounter in code. T he type is
assigned in context
Variablnames start with a letter and contain up to 31 cbaractcrs (only Icttrs
digits and underscorc).
MATLAB uscs some built-in variable names. Avoid using built-in variable nam.
All numbers arc storcd internally using the long format spccificd by IEEE
f1oating-point standard. T hese numbers have roughly a precision of 16 decimal
digits. They range roughly between 10E-308 and 10E+308. Howeveyare
displayed differcntly depending on the context.
MATLAB uses conventional decimal notation for numbers using decimal
points and leading signs optionally for example 1 -9 +9.0 0.001
99.9999.
Scientific notation is cxprcssed with the letter e for cxample 2.0e-03 1.07e23
1.732e+03.
lmaginary numbers use either i or j a suffix for example li -3.l4j 3e5i .
Operators: T he following arc the arithmetic operators in MATLAB.
+ Addition (wben adding matriccs/arrays size must match)
Subaction (same as above)
* Multiplication (the subscripts of arrays must be consistent when
multiplying them)
/ Division
Power
Complcx cougattranspose (0 ay transpose)
In the case of arrays each of these operators can be used wiaperiod prefixed to e
operator for cxample ( . * ) or ( ^ ) or (1).has a special meaning in MATLAB. It
imp1 ies lement-by-element operation. It is uscful for quick computation. It has no
"
1.2 INTAODUCTlON TO MATlAD 31
relevance in mathematics or anywhere else. W e w iJ1 usc it a lot in the next chapter for
gClIerating data for graphical opzation.
A n Interactlve S esslon: SM A AB.On P C prcss the Start button. Press the
Program button. Prcss thc MATLAB 'gram Group. Press M A TLAB. The MATLAB
Command window opens up and after somc introductory messagesE willbe a MATLAB
promptindid by double forward arrow. lis is indication MATLAB is ready to
aeptyourmmands. O n the networked stionstyping m atJab at the window prompt
should do the same. Note that the MATLAB program must be in the paas defmed in e
.Iogin or.h file. Ask the lab assistant for help. T he standard U N I X proccd
trn at the end ofe linc or befoe comment . The commcnts rclatc to what is
being typed d what is displayed in the MATLAB command window. FI 1frccyour
own variatiosS. In fact to undedandreinfocommandsit iscommended
you make up your own examples often.
> > a 1.0; b 2.0; c 3.0 d 4.0; e 5.0
> > w h y d i d o n l y c a n d e e c h o o n the screen?
>> w h o lists all the variables i n the workspace
>> a gives the value stored i n a
> > A 1. 5; variable A
>> a A % case matters
> > o n e a; two b; three C;
> > assigning values t o n e w variables
> > f o u r d; five e; six pi; value o f p i available
>> A 1 [a b c ;' d e f] A 1 is a 2 b y 3 matrix
>> % space o r comma separates columns
>> % semi-colon separates rows
>> A1(2 2) accesses t h e M a t r i x element o n the
>> % second r o w a n d second column
>> size (A1l gives y o u the size of the matrix
> > (rowcolum I lsl
> > A A 1 size(A1l What should happen here?
> > from previous satement the size o f A 1
> > contains two numbers organized a s a row
> > matrix. This is assigned t o A A 1
>> size(AA1l A A 1 is a o n e b y t w o matrix
>> A1' this transposes the matrix A 1
> > B 1 A1' the transpose of matrix A 1
> > is assigned t o B1. B 1 is a three b y two matrix
>> C1 ' " A 1 * B 1 Since A 1 a n d B 1 are matrices this
> > i8 a matrix multiplication
> > Should this multiplication b e allowed?
> > consider matrix multiplication in C o r
32 INTRODUCTION
>> % Fortran
> > the power of MATLAB derives from its
> > ability to handle matrices like numbers
> > C 2 81 * A1 H o w about this?
>> C1 * C2 What about this?
>> % read the error message
> > it is quite informative
> > D 1 [1 2]' D1 is a column vector
> > C 3 [C1 D1] C1 is augmented b y a n extra column
> > C 3 [C3 C2(3 : ) ] Note is a n assignment
> > means d o the right hand side and overwrite the
> > old information in C3 with the result
> > of the right
> > hand side calculation
> > O n the right you are adding a row to current
> > matrix C3. This r o w has the value of the third
> > row of C2 - Notice the procedure of
> > identifying the third row. The colon
>> % represents all the columns
> > C 4 C2 * C3 permissible multiplication
> > Note the presence of a scaling factor
> > in the displayed output
> > C S C2 * C3 seems to multiplyl
> > Is there a difference between C4 a n d CS?
>> % The * represents the product of each element
> > of C2 multiplied with the corresponding
> > element of C3
> > C 6 inverse(C2) % find the inverse of C2
> > apparently inverse is not a command in MATLAB
> > if command name is known it is easy to o bain
> > help
>> lookfor inverse this command will find all files
> > where it comes across the word "inverse" in
> > the initial comment lines
> > The command we need appears to b e INV which
> > says Inverse of a Matrix
>> % The actual command is in lower case. T o find
> > out how to use it - N o w
>> help inv % shows h o w t o use the command
>> inv(C2) inverse of C2
>> for i ' " 1:20
f(i) i"'2;
end
> > This is a n example o f a for loop
1.2 INTRODUCnON TO M An.AD 33
> > the index ranges from 1 to 20 in steps of
> > 1 (default)
> > the loop is terminated with "end"
> > the prompt does not appear until "end" is
> > entered
>> plot(sin(0.01*f) cos(0.03 f) )
>> xlabel('sin(O.Ol*f)') strings appear in single
> > quotes
>> ylabel('cos(0.03*f)')
>> legend ('Example')
>> title ('A Plot Example')
>> grid
> > The previous set of commands will create plot
> > label axes write a legend title a n d gr i d the
> > plot
>> exit finished wih MATLAB
This completes the first session with MATLAB. Additiona1 commands and features will
be encountered throughout the book. In is session it is evident M A. AB alows
easy manipulation of matrices definitely in relation to other programming languages.
Plotting is not difficult either. These advantages arquite substantial in the subject of
optimization. 1n next session we w il1 use the editor 10 I1ccomplish e same
through scripts. This session introduced
MATLAB Command window and Workspace
Variable assignment
Basic matrix operations
Accessing rows and columns
Suppressing echoes
who inverse commands
. * multiplication
Basic plotting commands
1.2.4 Using the Editor
In this section we will use the editor to te d run a MATLAB a script file. Norma1ly
eeditor is used to generate two kinds ofMA. ABfiles. 1 e files are termed scriptftles
andfunctionftles. AIough boofthse fiIes conn MATLAB commands like eones
we have used the second type of fiIes needs to be organized in a specified format . Bo
file tys should have the extension .m. Although these files are ASCII text fiIe
generic .m should be edbeca eMATLABshes forisex sion.his extension is
quetoMA. A B . 1 c script file contains a Iist of M A TLAB commands areexed
in sequence. T his is different m the interactive session of the previous section where
34 INTRODUCTION
MA1 . A B nded to ch conunand immediately. The script file is more useful when
there armany conun ds need to be ex uted to acmplish some objective like
running an optimization technique. It is important to remember that MATLAB aIlows you
to switch back interactive mode at any time by just typing commands in eworkspace
window like in e pvious section.
MATLAB provides an editor for creating these files on the P C platform. This editor
uses color to identify MATLAB statements and elements. It provides e current vaIues
ofthe variables (after ey are available in the workspace) when the mouse is over
the variable name in the editor. T hel two ways to access e editor through the
MATLAB C o m m a n d window on the PC. 5tart MATLAB. This will open a MATLAB
C o m m a n d or Workspace window. In this window the editor can be started by using
the menu or e toolbar. O n the File menu click on N e w and choose M -fiIc.
Altemately click on the Ieftmost icon on the toolbar (e tooltip reads N e w File). The
icon for the editor can aIso be placed on the desktop in which case the editor can be
started by double-clicking the icon. In this event a M ATlAB C o m m a n d window will
not be opened. The editor provides its o w n window for entering script statements.
At this point w e are ready to use the editor. Make sure you ad comments and
understand them as these procedu s will be used often. Using the editor implies w e
will be working with m-files.
The commands are same as in 1hinteractive session except there is no MATLAB
prompt prefixing the expressions. T o execute these commands you w iIl have to save
them toafil. Let us call the filescriptl.m. The.m extension need not be typed ifyou
are using MATLAB editor. Y o u can save the file using the Save or Save As
command from most editors. It is import tto know the full pato is file. Let us
assume the path for e file is C:OpCbookChlscriptl.m. Note e pa here is
specified as a P C path description. T he reason w e need is information is to inform
MATLAB where to find e file. W e do is in e MATLAB C o m m a n d window.
This implies w e should have e MATLAB Command window open. O n PCs w e start
MATLAB through the icons. In this book w e will accomplish most tasks by typing in
the C o m m a n d window or through programming. ln many instances ere are altemate
ways using menu items or e toolbar. This is left to the reader to experiment and
discover. In the MATLAB C o m m a n d window use addpath command to inform
MATLAB of the location of the file:
> > a d d p a t h C:Opt_bookCh1
The script th81 will be created and saved in S C T tl.m can be run by typing (note that
e extension is omitted)
> > scriptl
T o understand and associate with the prograrnming concepts embedded in. the script
particularly for persons with Iimited prograrnming experience it is recommended to
run the script after a block of statements have been written raer than typing e file
1.2 INTRODUCTION TO MATLAD 35
in its entirety. Prior to running the script you will have to save the script each time 5 0
the changes in f i Je are recorded and the changes are current. Another
recommendation is to deliberately mis'pe some statements and attempt to debug the
error generated by e MATLAs debugger during ution.
Creating the Scr tM-.flle (The following will be typedlsav in a fil.)
example o f u s i n g script
A 1 [1 2 31;
A 2 [4 5 61;
the commands n o t terminated w i t h semi-colon w i l l
d i s p l a y information o n t h e s c r e e n
A = [A1; A2]
B [A1' A2']
C A * B
n o w re-create t h e m a t r i x a n d p e r f o r m m a t r i x
multiplication a s i n o t h e r programming languages
example o f f o r l o o p
f o r i 1 3 v a r i a b l e i r a n g e s from 1 t o 3 i n
steps o f 1 (default)
a 1 (1 i) i;
e n d loops m u s t b e c l o s e d w i t h e n d
a 1
f o r i 6:-1:4 n o t e l o o p is decremented
e n d
a 2
a2(1 i-3) i; f i l l i n g v e c t o r f ro m r e a r
creating m a t r i x A a n d B (called A A a n d B B here)
f o r i 1:3
A A ( l i) a 1 (1 i) ; a s s i g n a 1 t o AA.
A A ( 2 i) a 2 (1 i) ;
B B (i 1) a 1 (1 i) ;
B B ( i 2 ) a 2 (1 i) ;
e n d
t h e s a m e c a n b e a c c o m p l i s h e d b y A A (1 :) a 1
without t h e f o r l o o p
A A p r i n t t h e v a l u e o f A A i n the w i n d o w
B B
w h o l i s t a l l t h e v a r i a b l e s i n the workspace
consider c o d e f o r M a t r i x multiplication
% w h i c h M a t l a b d o e s s o e a s i l y
36 INTRODUCTION
multiply two matrices (column of first m arix must
match r o w o f second matrix)
szAA size(AA) size o f A A
szBB size (BB) ;
i f (szAA(l 2) szBB(l l))
on ly i n column of A A match the rows of B B
for i ~ l:szAA(l l)
for j 1:szBB(l 2 )
C C ( i j ) 0 . 0 ; initialize value to zero
for k 1:szAA(l 2 )
e n d
e n d
CC(i j) c CC(i j) + AA(i k)*BB(kj ) ;
k - loop
j - loop
e n d
e n d
C C
i - loop
i f - loop
Note the power o f M A T ~ B derives from its ability to
handle matrices v e r y easily
this completes the script session
Save the above fi)e (scriptl.m). Add the directory to the MATLAB path indicated
before. Run the script file by typing scriptJ at the command prompt . Thc commands
should all execute and you should fmally see the MATLAB prompt in thc Command
window.
Note: You can always rcvert to the intcractive mode by directly entering
commandsin e MATLAB window after the prompt . In the Command window:
> > w h o
> > clear C % discards the variable C from the workspace
> > use with caution. Values cannot b e recovered
> > help clear
> > exit
T his session il1ustrated:
Usc oftheditor
Creating a script
Running a scIipt
Error debugging (recommcndcd activity)
Programming concepts
. Loop constructs if and for loops
L oop variable and increments
Array access

1.2 INTRODUCTION TO M AA D 37
Clear stalement
1.2.5 Creatlng a C o d e Snlppet
In this section w e will examine the 0er typc of m-filwhich is cal1ed thefunction
m-file. For those familiar wilh other programming languages Iike C Java or
F'ORTRANese files represent functions or subroutines. They are primarily uscd t'O
handle s'Om e specific types of calculations. Thcy also pr'Ovide a way for the modular
development of code as well code reuse. These code modules ar used by being
callcd or refcrred in other sections of the codc say through a script file w e looked at
earlier. The code that calls thc function m-file is called the calling progra m /code. The
essential paramctcrs in dcvel'Oping the function m-filc arc (1) what inpu1 is necessary
for the caIculations (2) what specific calculations must take place and (3) what
information must be retumed to the calling pr'Ogram. MATLAB requires the Slructure
ofe function m file to follow a prescribed format.
W e will use 1he edi10r 10 devel'OP a func1ion m-file tha1 w iI1 pcrform a polynomial
curvefit . II requires a set of datarepresen1ing a curve that needs to be fi1 10ge1her
wi the order of the polynomial to be fit. This exercisc is callcd curve fiuing. In
Chaptcr 6 such a problem will be iden1ificd as a problem in unconstrained
optimization. For now the calcula1ions necessary to accomplish the exercise are
considered known. It involves solving a linear equation with enomtal matrlx and a
right-h d vcctor obtained using the data points.'he output from the m-file will be
ecoeicients representing e polynornial.
Before w e start to develop the code the first line of this file must be formatted as
specified by MATLAB. In the first line the first w 'Ord startirig u m e first column is
c wordfunction. It is followed by the set of relum parameters (retumval). Next
ename (mypolyfit) 'Of the function with the parameters passed to thc function within
peses (XY N). le file must be saved as name.m (mypolyfit.m). The comme
be1ween the firs1 line and the firs1 executablc statement w iI1 appc if you 1ype help
lIame (help mypolyfit) in thc Command window. The reason for thc narne
mypolyfit .m is that MATLAB has a bui1t in funclIon polyfit. Open the editor 10 create
the file containing the following information:
function returnval mypolyfit(XY N)

These comments will appear wh en the user types

help mypolyfit i n the Command window
This space is intended t o inform the user h o w t o
interact with the program what it does
what are the input a n d output parameters
Least square error fit of polynomial of order N
x y - Data found in X Y
returns the vector o f coefficients starting from
% the constant term
. i : j'J S I
. .
: 1 '.
39
e n d e n d i 100p
if the x-points are distinct then inverse is not a
prob1em
1.2 INTROOUCTION TO MAl1.AB
lNTRODucnON

o r i 1:N+1
a(i) = 0.0; initia1ize the coefficient to zero
e n d
38
returnva1 inv(A)*b';
Save the file as mypolyfu.m. To use the function w e will need to create some xy data
d thcn call the mypolyfit function. S M A1 L A B in c directory tmypolyfit.m
resides in. or add the directory to the path. In the M A11.AB Command window: T he
fol1owing codis typed in the command window.

J
A

X

*

3
x

*

x

x

*

n
U

2
J

+

.

o
X
2

4
=
=

1
0

J
)

u
z

1
1

a
-

4
r
y
y

o
x
x
x

a

e
sz size (XY) ;
N D A T A sZ(l l); number of data points - rows 0
x y matrix
if NDATA == 0
fprintf('There is n o data to fit');
r eurnva1 a; zero va1ue returned
return; % return b ack t o ca11ing program
e n d
> > c o e f f mypolyfit(XY1 3)
> > a cubic po1ynomia1 was de1iberate1y created to
>> % check the results. Y o u shou1d get back
> > the coefficients y o u u sed to generate the curve
> > this is a good test of the program
> > Let u s create another examp1e
i f NDATA < 2 * N
fprintf'( T o o few data points for fi t' )
returnva1 a;
r eu r n
e n d
>> XY2 ( : 1) X Yl ( : 1) ; same first co1umn
>> XY2 ( : 2) 2 0 + exp (XY2 ( : 1) )
> > note the power of Mat1ab - a vector fi11
> > a n e w set of xy-data
T h e processing starts here.
T h e coefficients are obtained as solution t o the
Linear A1gebra prob1em [A] [c] [b]
Matrix [A] is the Norma1 Matrix which contains
the s u m of the powers of the independent variab1e
[A] is symmetric

u
s
m
u
a
d
u

> > c o e f f 1 mypolyfit(XY2

>> help mypolyit
> > y o u shou1d see the statements y o u set u p
> > In the next exercise w e wi11 create a script
> > fi1e that wi11 run a program for po1ynomia1
> > curve-fitting w e wi11 save XY2 so that it can
> > b e u s e d again
again cubic po1y. 3)
100p over al1 data points
variab1e is "11" (e1) (e)
XY(11 2)*XY(11 1)^(i-1);
11
for i 1:N+1;
b(i) 0.0;
for 11 l:NDATA;
b(i) +
100p
b(i)
e n d
save C:Opt_bookChlXY2.dat -ascii doub1e
this wi11 save the fi1e a s a n ascii text fi1e
with doub1e precision va1ues

This conc1udes e exercise where a code sp W written to lcate E

Cfficients of the polynooual used to fit a curvto some data. e type of filc is thc
for j 1:N+1;
if j >= i ca1cu1ating upper diagona1 terms
power (i-1) + (j 1) ;
A(i j) 0 . 0 initialize
for k l:NDATA; sum over data points
A(i j) = A ( i j) + XY( kl)^power;
e n d k loop
e n d c10se if statement
A i) = A ( i j) exp10iting Matrix symmetry
e n d e n d j 100p
40 INTRODUCTION
fimction m -file. It n b e used in a certain way. le code was ted using a cubic
polynomial. Nonpolynomial data wcre a1so tested. The data were saved for later use.
1.2.6 Creatlng a program
h is section a program will ad dat a curvefit the data using a polynomial and
mpare original and fi ddapru lly will be developed. T heta will be read
using a User I n ( U I ) window and a dialog box. T he code will be developed a
script filc. T his gives us an opportunity to revisit most of the code w e used rlier for
inforcement . W c will also use thc function m-fiIe created in previous exercise.
ler several ways for you or the users to interact with the code you develop. 1
basic method is to prompt users for information at prompt in the Command window.
is e quickest . This is probably what you will use when developing the code. Once
the de has been testeddepencling on usefulness it might be relevant consider using
more sophisticated custom eIements I ike input boxes and file selection boxes. lis book
will continue to use these elements throughout as appropriate. W hile the input elements
us in is code arc ncw commands the rest of the program will mostly use commands
thavc bccn introdu d rlier. In suential order the evcnls in sprogram are: (1) to
read the data saved earlier using a filc sclection box (2) to read e order of fit using
an Input dialog box (3) to use the mypolyfit function dcveloped in the last section to
obtathec fficients(4) to obtain ecina ofthe fittcd c ve(5) to graprucally
compare original d fitt dat a and (6)ton efitted auracyon efim
itselnew script file will be called prog...J1jm.
Start the cditor to create the filc called p'
program for fitting a po1ynomia1 curve to x y data
from
App1ied Optimization using Mat1ab
Dr. P.Venkataraman

Chapter 1 Section 1.2.6

The program 100ks for a fi1e with two co1umn ascii
data with extension .dat. The order of the curve is
obtained from user. The origina1 and fitted data
are compared w i th re1evant information disp1ayed o n
the same figure. The program demonstrates the use
of the fi1e se1ection b o x a n input dia10g b o x
creating specia1 text strings and disp1aying them
[fi1e path] =uigetfi1e('*.dat' 'A11 Fi1es' 200 200);
uigetfi1e opens a fi1e se1ection b o x
checkout he1p uigetfi1e
the string variable fi1e will hold the filename
1.2 INTRODUCTION TO M A T L A B 41
the stri variab1e path wi11 have the path
information the defau1t directory pointed wi11
differ depending o n the p1atform
i f isstr(file) if a fi1e is se1ected
10adpathfi1e ['load' path fi1e];
10adpathfi1e is a sting variab1e concatenated
with three strings "10ad " path and fi1e
note the space after 10ad is important
eval(10adpathfi1e);
eva1uates the string enc10sed - which inc1udes the
Mat1ab command 10ad. This wi11 import the xy-data
the data wi11 b e avai1ab1e in the workspace as a
variab1e with the same name as the fi1ename
without the extension (this assumes you se1ected
the xy-data using the fi1e se1ection box)
newname strrep(fi1e ' .dat' " ) ;
newname is a string variab1e which contains the
string file stripped of the .daextension. Thi
is a string rep1acement command
x eval(newname); assigns the imported data to x
just for convenience. The above step is not
necessary
NDATA length(x(: 1)); number of data points
clear p a t h 10adpathfi1e newname
get rid of these variab1es to recover memory
end
Note: if a fi1e is not se1ected nothing is being
% done
Use of a n input dia10g b o x to get the order
of po1ynomia1 to b e fitted
PROMPT {'Enter the Order o f the Curve'};
PROMPT is a string Array with one e1er
note the curly brackets
TITLE 'Order of the Po1ynomia1 to b e Fitted';
a string variab1e
LINENO 1; a data variab1e
getva1 inputdlg(PROMPT TT L ELINENO);
42 INTRODUCTION
the input dialog captures the user input i n getval
g eval is a string Array
check help inputdlg for more information
n o str2num(getval{l l});
the string is converted to a number- the order
clear PROMPT TTLE LNE NO deleting variables
% call function mypolyfit and obtain the coefficients
coeff mypolyfit(xno);
generate the fitted curve and obtain the squared
error
err2 0.0 i'
for i 1 : NDATA for each data point
for j l:no + 1
a(l j) = x(i l)"(j-l);
e n d
y(i) a*coeff; h e data for h e fite d curve
err2 err2 + (x(i 2) -y(i))*(x(i 2)-y(i));
he square error
e n d
plotting
plot (x ( : 1) x ( : 2) ro' x ( : 1) y b-' ) ;
% original data are red o's
fitted data is blue solid line
xlabel ( x' ) i
ylabel ( y' ) ;
strorder setstr(num2str(no));
convert the order o f curve o a string
same as getval if y o u have not cleared it
setstr assigns the string to strorder
titlestr ['polynomial curvefit o r order'
srorder 'of file' file)i
the three dots at the end are continuation marks
the title will have the order and the file name
title (titlestr)
legend('original data' 'fitted data');
errstrl num2str(err2);
errstr2 ['squared ero r errstrl]
gtext(errstr2) ;
1.2 INTRODUCTION TO MATlA9 43
this places the string errstr2 which is obtained
b y combining the string 'squared error' with
the string representing the value of the error
wherever the mouse is clicked o n the plot.
moving the mouse over the figure y o u should
see location cross-hairs
clear strorder titlestr errstrl errstr2 a y x i j
clear NDATA n o coeff XY2 file err2 getval
grid
This finishes the exercise
Run the progra by first running MATLAB in directory where these es are. or
adding the p to 1ate At e command prompt type proLP. T he
program should EXEcute requiring user input through the tile selection bOLinput
dialog and final1y displaying Figure 1.11. le appe ance may be slightly different
depending on pla'orm MATLAB is being run.
M This finishes the MATLAB section of the chapter.Tlm section lIas introduced
ATLAB in a robust manner. A broad mngof progmmming experience has been
mltlated in this chapter-AH m w commands have h e n ideruincd with a bdef
explanation in comments. It is important you use the OPPOltunity to type in
E CodE yourself;That is the only way the use ofMAT1AB will become familiar.The
practice a1so willlead to fewer syntax erro The writing of code will significantly
improve your abi1ity to debug and troubleshoot. W hile is chapter employed a
Polynomial curve fIt of order 3 of fileXY2.dat
5
x
Flgure 1.11 Origlnal and fllled dala.
t2J!.:;l;'f. li44uu1uik J t : i
44 INTRODUCTION
seoarate section on the use of MATLAB out of nsitysub uent c h a p m M e
C M a ized by a complration ofthe us mofMAT .TLAB
optimization.
P R O B L E M S
uruuz;ztm: ; alr;::;t;
mathematical modeJ.
1.2 Identify several possible optimization problems related a n aircraft. For each
problem identify all e disciplines will help establish the mathematical
model.
1.3 Id nt sev possible 0 ptimiza
pm 0 blide a11 t he d is ciplins will h 1 elp es isl h m a la na atiL
modeJ..
14 Identify severa1 possibloptimization problems related to a microsystern used
for control.ForBach problem identify all thEdiscIPlines that will help establish
e mathematical model .
1.5 Define a problem wit1I respect to your investment in the stock market. Describe
e natureofthemathematical modd-
1.6 Define problem and establish mathematica1 model for the I-beam
holding up an independent single-family home.
1.7 Define the problem and identify the mathematical model for an optimum
overhanging traffic light.
1.8 Define e problem and identify a mathematica1 model for scheduling and
ootimization of the daily routine activity.
Dne e p e m for a lamin
iv given aspe :eic p ump.
110 Define a chemical engineering problem to mix vious mixtures of 1imited
availability to make specified compounds to m tspecified dmands.
2
G R A P H I C A L OPTIMIZATION

P
U
1
2

This book includes a large number of cxamples with two variables. Two-variable
problems can be displayed graphically and thsolution obtained by inspection.
Along with the usefulness of obtaining the solution without applying
mathmatical conditions the graphical representation of the problem provides an
opportunity to explore the geometry of many of the numerical techniques w e
examine later. This is necessary for intuitively understanding the algorithm and its
progress toward the solution. In practical situations which usually involve over
two variables there is no opportunity for neat graphical correlation because of the
Iimitations of graphical representation. It w il1 be necessary to review the numbers
to decide on convergence or the lack thereof. What happens with many variables
is an extension of the geometric features that are observed in problems involving
two variables. A good imagination is an essntial tool for these problems with
many variables.
Contour plots provide the best graphical presentation of the optimization
problem in two variables. The points on any contour (or curve) have the same value
of the function. Several soware packages are available to create and display these
plots. MATLAB. Mathematica Maple and Mathcad are among e general-puqJOse
software packages can draw contour plots. In this book w e will use MATLAB for
graphica1 optimization.

1
9

U
1m

2.1 P R O B L E M DEFINITION
T he standard format for optimization problems was established in Chapter 1. lt is
reinoduc here for convenience:
46 GRAPHICAL OPTIM'ZATION
Minimize fl..x J X2 . Xn)
Subject to: h(x" X2 Xn) = 0
h
2
(xX2.. x ) = O
ht(XIX2.. .X) = 0
gl(X"X2... Xn) o
g2(XI.X2..Xn) S O
glll(X"X2.. .x ) S O
x:gxi X7.
i=I2...n
(2.1)
(2.2)
(2.3)
(2.4)
In is chapter while adhering to the format. the necessity for zero on the right-hand
side is relaxed. is is being done for convenience and 10 aid comprehension. le
right-hand side can also have numerical values other 1han zero. Th first example
chosen for illustration is a simple onEusing elementary functions whose gmp12ical
nature is well known. This simple example will permit exarninntion of the MATLAB
code at will gencrate ecurves and the solution. It will also allow us to define E
format for the disp1ay of solution to the graphical optimization problcm.
2.1.1 Example 2.1
The first example Example 2.1 will have two equality constraints and two inequality
constramts:
Minimize fl..XI'
X2) = (XI -
3)2
+ (X2 -
2)2 (2.5)
Subjcct to: hl(xl. X2): 2x
1 + x2 = 8 (2.6a)
h
2(x"
XU: (XI -
1)2
+ (
-4)2=4 (2.6b)
gl(X" X2): Xl + x2 S 7 (2.7n)
g
2(x . XU:
(2.7b)
o SXI S 10; O S X 2 S 10 (2.8)
In the above dcfmi1ion we have two straight lines. two circles nnd a parabola. Note
that two equality cons ints and two variables imply t E is no scope for
o
zation.le problem will be detcrmined byuali1yconstraints provided the
twocons ints are I inearly independent which is true in this exnrle. T his example
was 1ed 10 help understand codcinMA' A B atwill be used to draw graphical
solutions in this book.

2.1 PAOBLEM DEFINfTlON 47

Figure 2.1 iIIustrates the graphical solution to isproblem.lefigure also displays
additional information related to the inequality consaintsplaced on it after the
principal graphic information was generated and displayed.
2.1.2 Format for the Graphlcal Dlsplay
T he graphical solution to Example 2.1 as seen in Figure 2.1 is generated using
M ATI.AB [1.2] ex:pt for the identifi tionof the inequality constraints. All inequality
constraints are distinguished by hash marks. The hashed side indicates the infeasible
region. In MA'1.AB 5.2ere is no feature to insert these marks through programming.
A n add-on program called Matdraw was used to create the hash I ines in the figure. It
can be downloaded from the Mathworks site. Beginning wiVersion 5.3 there is a
figure (plotedit) ditor will allow you to insert additional graphic elements to E
fig e. le student can also p cil in ehash tines after obtaining the printout It is
to be noted at the graphical solution is incomplete if the inequality constraints are
not distinguished or if the feasible region has not been established in some manner.
10
g
B
3

2 3 4 5 6
x1 valu
7
guro 2.1 Graphical solution for Example 2.1.
s 10
48 GRAPHICAL OPTIMIZATION
In the graphical display of solutions only e objective function is drawn for
several contours so that edirection fore minimum can be identified. Each cquality
constrnt is drawn as a single curve for e numerical value on the right-hand side.
They should be identified. Each inequality constraInt is similarly drawn for a value on
e right-hand side. They need to be identificd too. The hash marks ar lrawn/penciled
in on the figurc or a printout ofefigu. T he extent of the dcsign region is established
and the solution identified on the figure. T he relevant region can be zoomed for better
accuracy. data for e plot are obtained using MATLAB'S matrix operations after
establishing the plolting mesh. T his exploits M ATI.AB'S nalural speed for matrix
operations.
2.2 G R A P H I C A L S O L U T I O N
MATLAB poss s a powerful visualization engine that permits the solution of e
two-variable optimization problem by inspection. There are three ways to lake
advantage of the graphic features of M ATI.AB. e first is lhe use of M ATI.AB'S
high-Ievel graphing routines for data visualization. T his will be primary way to
solve graphical oplimization problems in this book. This w il1 also be e way 10
incorporate graphical exploralion of numerical techniques in e book. For m o
precise control over the display of data MATLAB allows usr intcraction through
programming using an object-oriented system identified in M ATI.AB as Handle
Graphics. The third use ofthe MATLAB graphics engine is to use the Handle Graphics
system to develop a Graphical User Interface (GUI) for the program or m-file. T his
creales a facility for sophislicated user interaction. MOSl of the plotting needs can be
mel by the high-Ievel graphics functions available in MATLAB.
2.2.1 MATLAB High-Level Graphlcs Functlons
'here are three useful windows during a typical MATLAB scssion. T he first is e
MATLAB Command window through which MATLAB receives instruction and displays
alphanumeric information. le second window is the texl-editor window where
m-fiIes are coded. The third is the Figure window where the graphic elements are
displayed. There can be morean one figure window. The figure window is targel
of the high-level graphics functions.
T he graphics functions in MATLAB allow you to plot in 2 D or 3D. T hey allow
contour plots in 2 D and 3 D mesh and suaplotsbar area pie charts histograms
animation and gradient plots. Subplots can also be displayed using these functions.
In addition they permit operation with images and 3 D modeling. They allow basic
control of thc appear ce of the plot through color line style and markersis
ranges and pect ratio of the graph.'hey permit annotation of the graph in several
ways. Some of these functions will be used in the next section when w e devclop the
2.2 GRAPHICAL SOLUTION 49
:.L
m-file for e first example. T he following introduces some more useful Informon
about the use ofMATLAB8 high-Ievcl graphics functions.
The two main graphical elements arc typically controlled using the high-Ievel
graphics functions are the figu and the axes. Using Handle Graphics you can control
most of the other graphical elements which include elemnts used in the G UI. These
other elements are typically childrcn of e axes or figure. The figure function or
command creates a figure window with a number starting at one or will creatanew
figure window incrementing the window count by one. Normally all graphics
functions are targeted to the c rent figur windowwhich is selected by clicking it
with the mouse or executing the command figure (number) where number is the
number of the figure window al will have lhe focus. A Il commands are issued in
command window. Graphics functions or commands will automatically create a
window if none exists.
Invoke help commandname at the MATLAB prompt to know more about the
functions or comrnands. The word handle app on m y plaorms and in many
applications particularly those that deal with graphics. They are widely prevalent in
objectoriented programming practice. M ATI.ABs visualization system is object
oriented. Most graphical elements are considered as objects. The handle" in M ATI.AB
is a syoftware created number can ident'y the specific graphic object . If
this handle is availableen properties of the objecl such as line size marker type
color and so on can be viewcd set or reset if neccssary. In MATLAB Handle Graphics
is e way to custornize e graphical elements. In high-Ievel grhics functions s
is used in a limited way. In this chapter w e will use it in a minimal way to change E
charactcristics of some of the graphical elements on the figure.
T o understand the concept of "handlcs" w e will run e following code
interactively. It deals with creating a plot which was introduced in Chapter 1. T he
boldface comments in the code segment indicate the new fealures that are being
emphasized.
Sta MATLAB and interactiveLy perform the following:
> > X O:pi/40:2*pi;
> > y X. s i n (x) ;
>> plot(xy 'b-');
>> grid;
> > h p l o t ( xy 'b-') h is the handle t o the plot
> > a n e w pl o t is overwritten i n the same figure window
> > a numerical value is assigned t o h
> > W e c a n u s e the variable name h o r its value
> > t o refer t o the plot again
>>
>> set(h 'LineWidth' 2); this should make your plot
> > thicker blue
>> set(h'LineWidth' 3 'LineStyle' ' :' 'Color' 'r')
r

o

c

e

r
h

0
D
t
r

c
o
n

e
t
-
-
v
c

e
x

x
v
s

e
y
v

L
a
e
y

e

r
a
t

c
e
o

r
l

c
p

50 G A A P H I C A L OPTIMIZATION
> > The handle is used t o refer to the object
> > whose property is being changed
> > Usually Property information occurs in pairs of
> > property-namejproperty-value
> > property-value can b e a text string o r number
> >
> > get(gca) this will list the property of the axes
> > of the curren plot. Note there are a significant
> > amount of properties y o u can change t o customize
> > the appearance of the plot
> >
> > set(gca 'ytick' [-5 2.5 0 2.5 5] )
> > y o u have reset the ytick marks o n the graph
> > set(gca 'FontName' 'Arial' 'FontWeight' ...
> > 'bold' 'FontSize' 14)
> >
> > Changes the font used for marking the axes
> > set(gca 'Xcolor' 'blue')
> > changes the x-axis to blue
> > % concludes the demonstration of handles
From the session above it is evident to fine-tune the plot you create you
need to first identify the object handle. You use the handle to access e object
prop y. You change a property by setting its value. W e will see more graphics
function as w e obtain the graphical solution to Example 2.1 in following
sectlon.
2.2.2 Example 2.1-Graphical Solution
Figure 2.1 is the graphical representation of Example 2.1. The range for the plots
matches the side constraints for the problem. The intersection of the two equality
constraints identifies two possiblsolutions to the problem approximately (2.6
2.8) and (1 6). The inequality constraint g2 makcs the point (2.6 2.8)
unacceptable. Point (1 6) is accptable with respect to both constraints. Thc
solution is therefore at (1 6). While the objective function was not used to
determine thc solution contours of the objective function are drawn indicating
direction for e minimum of the objecti vfunction. The solution can be identified
by inspection of the assmbled plots without reference to the terminology or the
techniqucs of optimization. Recall however. that this is only possible with one or
two design variables in the problem.
T he code for is exnple involves six m-files: a script m-file will incude
the M A T L A B statements to create and display e data and five m-fi1one for
cach function involved in Example 2. 1. W e begin with the script m-file-
Ex2_1.m.
2.2 G A A P H I C A L S O L U T I O N 51
File: Ex2 1.m
Chapter 2: Optimization with MATLAB
% Dr. P. Venkataraman
Example 2_1 (Sec 2.1-2.2)

graphical solution using MATLAB (two design variables)

the following script should allow the display
of graphical solution

Minimize f(x1 x2)

h1(x1 x2)
h2(x1 x2)
gl(X1 x2)
gl(x1 x2)

(x13) **2 + (x2 2)**2

2x1 + x2 8
(x1-1)^2 + (X2-4)^2 4
x 1 + x2 < = 7
x 1 - 0.25x2^2 < = 0.0
% 0 < = x1 < = 10 0 < = x2 < = 10

NOTE: The hash marks for the inequality

constraints must b e determined and drawn outside
of this exercise and o n the printout
- - - - - " " - -
x 1 = 0 : 0 . 1 : 1 0 i the semi colon at the end prevents
% the echo
x 2 = 0 : 0 . 1 : 1 0 i these are also the side constraints
x1 and x2 are vectors filled with numbers starting
at 0 and ending at 10.0 with values at intervals
of 0.1
[X1 X2] meshgrid(x1 x2)i
generates matrices X1 and X2 corresponding to
% vectors x1 and x2. It is a mesh of x1 and x2 values
at which the functions will b e evaluated
f1 obj_exl(X1 X2) i the objective function is
evaluated over the entire mesh and stored in f1
MATLAB will compute the values for the objective
througha function m file called obj_exl.m
ineq1 ineconl(Xl X2)i the inequality gl is
evaluated over the mesh
ineq2 inecon2(Xl X2)i the inequality g2 is
evaluated over the mesh
eq1 = eqcon1(Xl X2) i the equality h1 is evaluated
over the mesh
eq2 = eqcon 2 (X1 X2) i the equality h2 is evaluated
over the mesh
52 GAAPHICAL OPTIMIZATION
[C1 han1] contour(1 x 2 ineq1 [7 7) 'r-' ) i
a single contour plot o f g l (or ineql) is drawn for
the value of 7 i n r e d color a s a continuous line
duplication of the contour value is necessary
for drawing a single contour
han1 is the handle t o this plot. This handle
can b e used t o change the plot display characteristics
C l contains the value of the contour
clabel(C1 han1) i
labels the contour wit h the values i n vector C 1
contour(x1 x2 ineq1 [7 7] 'r-') will draw the contour
without labeling the value
h o l d o n allows multiple plots i n the same figure
window
gtext ( g l ' ) i
will place the string 'gl' o n the plot at the spot
selected b y a mouse click. This is a text label
this procedure is repeated for remaining
constraints
[C2 han2] contour(x1 x2 ineq2 [0 0] 'r- ) i
clabel(C2 han2)i
gtext('g2') i
[C3 han3] contour (x1 x2 eq1 [8 8] 'b-' ) i
clabel(C3 han3)i
gtext ( h1' ) i
[C4 han4] contour(x1 x2 eq2 [4 4] 'b--') i
clabel(C4 han4) i
gtext ( h2' ) i
[Chan] contour(x1 x 2 f1 'g')i
contour of 'f' is drawn i n green color. The number
o f contours are decided b y the default value
clabel (C han) i
xlabel('x1 values' 'FontName' 'times' 'FontSize' 12 .
'FontWeigh 'bold') i label for x-axes
ylabel('x2 values' 'FontName' 'times' 'FontSize' 12
'FontWeight' 'bold')i
g r i d
h o l d off
The function m-fiIes are:
2.2 GAAPHICAL SOLUTION 53
o n 1 . m
function retval inecon1(X1 X2)
retval X1 + X 2 i
X 1 X2 are matrices
retval is the value being returned after the
computation
Since X 1 a n d X2 are matrices retva1 is a1so a
matrix
B y this w a y the entire information o n the mesh
is generated b y a sing1e call t o the funcion
inecon1
2 . m
function retva1 inecon2(X1 X2)
retval X l - 0 . 2 S * X 2 . ^ 2 i
Note the use o f ^ operator for element b y e1ement
operation. That is each e1ement of the X2 matrix
is squared. Without the dot the imp1ication is a
matrix multiplication - between matrices whose
inner dimensions must agree. Similar operators are
defined for element b y element muliplication
a n d division
L m
function retval eqcon1(X1 X2)
retval 2.0*Xl + X 2 i
2 . m
function retval eqcon2(X1 X2)
retval (X1 - 1) .*(X1 1) + (X2 - 4). * (X2 - 4) i
U
function retval obj_ex1(X1 X2)
retval = (X1 - 3) .*(X1 - 3) +(X2 - 2). (X2 2) i
2.2.3 Displaylng the Graphics
A l1 of lhc files required for graphical display of the problem have been crcalcd. In the
Command window lype addpatb followed by ecomplele path for thc directory at
holds these files. At the prompt type thc name of thc script file for the example
without the .m eXlension
> > E x2 1
54 G R A P H I C A L O P T I M I Z A n O N 2.2 G R A P H I C A L S O L U T I O N 55
T he first contour plot will appear. Move the mouse over to e plot and a cross hair
appears. Clicking on the plot w i1l placthe string "gl" at the cross hair. The second
contour plot should appear and there is a pause to place text label . After the four
consaintseobjective function is plotted for several contour values. Finally the plot
should app r i n Figure 2.1 without the hash marks (and wiyour choice of B
location for constraint labels). You can insert the hash marks through editing the plot
directly (Version 5.3 onwds). solution for the problem is at ( 1 6) where four
plots intersect . The value of the objective function is 20 e solution.
If the plot is acceptable you can print the informalion on the figure by expong it
(using an appropriate extension) 10 a file and later incorporating it in another
document . You can also send it to e printer from Ihe print command on the File
menu. Typing help print in the Command window should I ist a set of commands you
can use to save e file.
10

2 4 6
X
1
values
s 10
> > p r i n t -depsc2 plot_ex_2_1.eps
w i1l c ate the level 2 color postscript file called plocex_2_l.eps in the working
directory. A complete path name should save the file to e specified directory. You
can then execute standard operating system commands 10 dump the filc 10 e
applicablprintcr. You can also save the figure (ex sion .fig) and later open it in
MATLAB using the filc open command from menu b ortool bar.
2.2.4 Customlzlng the Flgure Flgure 2.2 Customized U forE mple 2.1.
In this section thc basic figure and plots creatcd abov w ilI be custornizcd using the
Handle Graphics commands first explo dinSction 2.2.1. All ofthe plots describing
e constraints will have a linewidth of 2 points. le objective function will have a
linewidth of 1 point. The text labels w i1l be in the "Times" font. It w ilI be in boldface
with a font of size 14 and in e color of the cons int . The x d Y tick marks will be
changed to establish a broader grid. T he x d y labe1s will include a subscript . A
two-row text w i1l be inserted at the point selected through the mousc. The ncw figure
is shown in Figurc 2.2.
First py the file in e previous exercise and rename it.le renamed file will be
used makethen essarych gesto ecode. T hese changes inc1ude both new code
as w eIl as replacement of existing lines of code. In e foIlowing only e changes to
code are i1Iustrated. The original code is shown in the italic style and identified
with three ellipsis points (. . .) at beginning of the line. The code to be inserted or
replacement code is shown in normstyl. If no replacemcnt is apparent then the old
code is used 10 define the locations whee w de is appended.
g t t ( ' g l ' ) ;
k1 g t e x t ( gl' ) ; k1 is the handle t o the text element
set(k1 'FontName' 'Times' 'FontWeight' 'bold'
FontSize' 14 'Color' 'red')
. . c l a b e l ( C 2h a n 2 } i
set(han2 'LineWidth' 2)
k2 = g t e x t ( ' g 2 ' ) ;
set(k2 'FontName' 'Times' 'FontWeight' 'bold'
FontSize' 14 'Color' 'red')
c l a b e l (C3 h a n 3 ) ;
set(han3 'LineWidth' 2)
k3 ~ g t e x t ( ' h 1 ' ) ;
set(k3 'FontName' 'Times' 'FontWeight' 'bold'
'FontSize' 14 'Color' 'blue')
{Cl h a n l } c o n t o u r ( x l x 2i n e q l {77} ' r - ' } ;
set(han1 'LineWidth' 2); sets 1ine width o f the gl
contour to 2 points
c l a b e l ( C 4 h a n 4 ) ;
set(han4 LineWidth' 2)
k4 ~ gtext('h2');
56 GRAPHICAL OPTIMIZATION
2.3 A DDlTIONAL EXAMPLES 57
set(k4 'FontName' 'Times' 'FontWeigh 'bo1d'
FontSize' 14 'Co1or' 'b1ue')
c:label (C han) ;
set(han 'LineWidth' 1)
.. xlabel ( x l val ues' FontName' 'times'
x1abe1(' x 1 va1ues' 'FontName' 'times' FontSize' 12 .
FontWeight' 'bo1d') i
y1abe1(' x 2 values' 'FontName' 'times' '
FontSize' 12 'FontWeight' 'bo1d')i
set (gca xtick' [0 2 4 6 8 10]) set xticks
se(gca 'ytick' [0 2.5 5.0 7.5 10]) set yticks
k5 gtext({'Chapter 2: Examp1e 1'
'pretty graphical display'})
the above gtext describes a string array
string array is defined using curly braces
set(k5 'FontName' 'Times' 'FontSize' 12
Fontweight' 'bo1d')
!(xI' X
2
) cos(qx
2
) -1- c + d
with
a = 1 b = 2 c = 0.3 d = 0.4 p = 3q = 4
(2.9)
Figurcs 2.3-2.7 are graphical display of solutions is section. The figures in
the book being rescted to black and white will notnvey eimpact of color images
you will see on the screen. T hcre are two files associatcd with the plots: the script
m-fiIe (ex2.-2.m) and thc function m-fiIe (obLex2.m).
..z.m
Chapter 2: Optimization w ith MATLAB
Dr. P.Venkataraman
Examp1e 2.2 Sec.2.3.1

(at the end)

c1ear C C1 C2 C3 C4 h h1 h2 h3 h4 k1 k2 k3 k4 k5
get r i d o f variab1es from the workspace
Run the prograrn to sec e figure in Fig 2.2. Once the plot is customIzed to your
satisfaction you can make it standard for other plots you will pr U i n M AA s .
This file could also be a template for general contour plotting. Note at a1l function
information is obtained through function m-fiIes which are coded outside the script
filc. Sctting up ncw problems or examplcs only requires addinglchanging new
function m-fi1.
11IAbclodConl g
defaull cGlcr
2.3 ADDITIONAL E X A M P L E S
T he f01l0wing additional exnples will serve to iIlus tebooptimization problems
well as additional gmphical fcatures of M A T L Aat will be useful in dvelopinggraphical
solutions to optimization problems. lC grphical routines in MATLAB are powerful d
SY to use. T hey can gr phically display the problems vem1 ways wivery simplc
m m ds. T he useful display is however detcnnined by e user. T he first example in
this section Example 2.2 is a problem in uncons ined opttion. second
cxample is a structum1 engineering problcm of reasonablc complcxity. lCexar Ic
demonssoptimization in thc area of heat Iransfcr dcsign.

2.3.1 Example 2.2

This cxamplc iIIustes scvcral diffcrent ways of gmphicaUy displaying a function of
two variables. T he problem was uscd to iIIuslrate global optimization in Reference 3.
T he singlc objcctive function is
-0.5 O
X
J
valu
0.5
gu 2.3 Filled conlours with colorbar . Example 2.2.
59
2.3 A O O m O N A L E X A M P L E S
2 D Con
with Grndi.nl V:10
G R A P H I C A L OPTIMIZATlO N 58
8
2
-
g
J

X
1
values
-1 X:! Vues
0.5 o
X
1
values
-0.5
3D conlour wilh lorbar . Exle2.2.
the objective function is
evaluated over the entire mesh
filled contour w ith default colormap
help graph3d gives y o u the choices for colormap
[C1 han11 contourf(x1 x 2 f1
[0 0.1 0.6 0.8 1.0 1.2 1.5 1.8 2.0 2.4 2.6 2.8 3.01);
Flgure2.5
f1 obj_ex2(X1 X2);
1y't.
ticks
ticks
specific contour levels indicated above
clabel(Clhanl);
colorbar il1ustrates the default color scale
set(gca 'xtick' [-1 -0.5 0.0 0.5 1.01) custom
set (gca 'ytick' [1 -0.5 0.0 0.5 1.01) custom
grid
xlabel(' X_1 values' 'FontName' 'times'
'FontSize' 12);
% label for x-axes
ylabel(' x 2 values' 'FontName' 'times'
graphical solution using MATLAB (two design variables)
Unconstrained function illustrating global minimum
Example will introduce 3 D plots 3 D contours filled
2 D contours with gradient information
- - - - - - - - - - - - - -
x 1 =1:0.01:1; the semi-colon at the e n d prevents
" the echo
x 2 = - 1 : 0 . 0 1 : 1 i these are also the side constraints
x 1 a n d x 2 are vectors fil1ed w ith numbers starting
a t -1 a n d ending at 1.0 w i t h values at intervals o f
0.01
Contour wlth gradlent vectors: Example 2.2. gure 2.4
[Xl x21 = meshgrid(x1 x2) ;
generates matrices X 1 a n d X2 corresponding
vectors x 1 a n d x 2
reminder M A T ~ is case sensitive
60 G R A P H I C A L OPTIMIZATION
CO!U'SC Mesh P101
lormap1
4
3.5
3
2.6
2
1.5
a
u

m
w

0.6
X
2
values
-1 -1
X
1
valu
Flgure 2.6 3D mesh plot wlth colorbar: Exampl2.2.
FontSize' 12);
title({'Filled Labelled Contour' ' ...
default color map'} 'FontName' imes' 'FontSize' 10)
a n e w figure is u s e d t o d r a w the
basic contour plot superimposed w i t h gradient
information
a lso information is generated o n a coarser m e s h t o
k eep the figure tidy. g r i d is removed f o r clarity
figure a n e w figure window is drawn
y 1 -1:0.1:1.0;
y 2 1: 0.1: 1;
[Y1 Y2] meshgrid(y1 y2);
f2 obj_ex2(YlY2);
[C2 han2] contour(y1 y 2 f2
[0 0.5 0.75 1.0 1.5 1.75 2.0 2.5 3.0]);
2.3 A D O m O N A L E X A M P l E S 61
CoaSurfacePlot
lonnnp -jcVdefnult
4
3.5
3
2.5
2
1.5
0.5
0
1
0.5
0.5

5
.0.5
X
2
values
-1 -1

H
M

-
a
v

x

Flgure 2.7 3D su plot with default colorbar : Example 2.2.
2
1.5
clabel(C2 han2)
[GX GY] gradient(f2 0.2);
% generation o f gradient information see help gradient
h o l d o n muliple p l os o n the same figure
quiver(Y1 Y 2 G XGY);
draws the gradient vectors a t g r i d points
see help quiver
h o l d off
set(gca'xtick' [-1 -0.5 0.0 0.5 1. 0])
set(gca 'ytick' [-1 -0.5 0.0 0.5 1.0])
xlabel(' x_1 values' 'FontName' 'times'
FontSize' 12) ;
ylabel(' x 2 values' 'FontName' 'imes'
'FontSize' 12);
title({'2D Contour' 'with Gradient Vectors'}
FontName' 'times' 'FontSize' 10)
62 GRAPHICAL OPTIMIZATlON
A final example of contour information is a 3D
% contour plot - o r a stacked contour plot
figure
colormap(spring)
contour3(x1 x2 f1
[0 0.3 0.6 0.8 1.0 1.5 1.8 2.0 2.4 2.6 2.8 3.0]);
set(gca'xtick' [-1 0.5 0.0 0.5 1.0])
set(gca 'ytick' [-1 -0.5 0.0 0.5 l. 0])
change colormap and set colorbar
changing color for display is easy b y selecting
among some colormaps provided b y MATLAB
colorbar
xlabel(' x 1 values' 'FontName' 'times'
FontSize' 12);
ylabel(' x 2 values' 'FontName' 'times' ...
FontSize' 12);
tile({'Stacked (3D) Contour' :colormap - spring'}
FontName' 'times' 'FontSize' 10)
gr i d
he next two figures wi.ll display 3D plots
he first is a mesh plot of the function
once more coarse data is used for clarity
figure
colormap (cool) another colormap
mesh (y1 y2 f2) using information generated earlier
set(gca 'xtick' [-1 -0.5 0.0 0.5 l. 0])
set(gca'ytick' [-1 0.5 0.0 0.5 1.0])
colorbar
xlabel(' x 1 values' 'FontName' 'times'
FontSize' 12);
ylabel(' x_2 values' 'FontName' 'times' ...
FontSize' 12);
title({'Coarse Mesh Plot' 'colormap - cool'}
FontName' 'times' 'FontSize' 10)
grid
he final plot in this series
surface plot with default colormap
figure
colormap(je}
surf(y1 y2 f2) using old information
colorbar
xlabel(' x 1 values' 'FontName' 'times' .
2.3 AOOrrtONAL EXAMPLES 63
FontSize' 12);
ylabel(' x_2 values' 'FontName' 'times'
FontSize' 12);
set(gca 'xtick' [-1 -0.5 0.0 0.5 l. 0])
set(gca'ytick' 1-1 -05 0.0 0.5 l. 0])
title({'Coarse Surface Plot' colormap - jet/
default'} 'FontName' 'times' 'FontSize' 10)
grid
Q s. m
function retval obj_ex1(X1 X2)
Optimization with MATLAB
Dr. P.Venkataraman
Chapter 2. Example 2.2

f(x1 x2) a*xl""2 + b*x2^2 c cos (aa*x1)

d*cos (bb*x2) . . .
+ c + d
a ' " 1; b 2 c 0.3; d ' " 0.4; aa
3.0*pi; b b 4.0*pi;
note matrix operations need a dot operator
retval a*X1.*X1 + b*X2.*X2 -c*cos(aa*X1)
d*cos(bb*X2) + c + d;
The brief comments in the code should provide an explanation of what you see on
figure. Figures can be further custornized as secn in e previous section. From an
optirnization perspective Figure 2.4 provides thc best infonnation about thc nature of
the problem. T he 2 D contour curves identify the neighborhood of the 10 lrninimum.
le gradient vectors indicatc direction ofe functions steepest rise a1 point
so s and vallcys can bc distinguishcd. T he contou thernselves can be colored
without being fillcd.
T hc quiver plp1 shown in Figure 2.4 also provides a mechanism 10 indicate e
feasible region when dealing with inequality constraints sincc they indicatc e
direction in which constraint function w iII incrse. If several functions are bcing
w nen thc cIutter produ d by e arrows may dfuse e cIarity. T he user is
encouragcd 10 use the.powerful graphical features ofMATLAB to his benefit at all times
without losing sight of e objective of his cffort. MATLAB graphics has many more
features than will be covered in is chaptcr. The expos in this chapter should bc
sufficient forEderconfidently explore many other useful graphimmands.
Thc 3 D mesh and surface plots have limited usefulness. T hese plots can be uscd to
reinforce some of the features found in Figure 2.4. The infonnation in these plots may
be improved by choosing a camera angleemphl Z C S m easpect of the graphi I
description. T his exploration is left to Caderas an exercise. Using help view in the
MATLAB Command window should get you startcd in is diction.
64 GRAPHICAL OPTIMIZATION
2.3.2 Example 2.3
le next example is a complex one from structural engineering design is relevant
in civillmechanicallaerospace cngin ring applications. It appeared as a problcm in
Refercnce 4. It is dcveloped in detail here. The problem is to redesign the basic tall
flagpolc in view of the phenomenal increasc in wind spceds during extreme weather
conditions. In recent catas'Ophic events. the wind sp ds in tomadocs have bccn
measured at over 350 miles per hour. T hese high specds app rto be the norm rather
unusualcvent .
Deslgn Problem: Minimize the mass of a standard IO-m tubular flagpolc to
withstand wind gusts of 350 miles per hour. T he flagpole w il be made of structural
steel . Use a factor of safety of 2.5 for c structuraJ design. lC deflection of the top
of the flagpole should not exceed 5 cm. The problem is described in Figure 2.8.
Mathematical Model: The mathematical modcl is developed in detail for
completencss and to provide a rcview of useful s ctural (5) and aerodynamic
relations [6). The relations are exprcssed in original symbols rather than in standard
format of optimization problems to provide an insight into problem formulation.
Deslgn Parameters: Thc stnlctucal stcel [5] has the fol1owing material constants:
E (modulus of clasticity): 200 E 9 P a
011 (allowablc normaJ s) : 250 E+06 Pa
1:
0
1 1 (allowable shcar stress): 145 E+06 Pa
8 m
A
1 0 m
gure 2.8 Flagpole deslgn: Exampla 2.3.

s8ctlon A A
2.3 ADDITIONAL EXAMPLES 65
'Y (materiaJ density): 7860 kglm
3
FS (factor of safety): 2.5
g (gravitational acceleration) = 9.81 mls2
For the acrodynamic calculations the following are considered:
p (standard air density): 1.225 kglm
3
C
d
( coefficient of cylinder): 1.0
W p (flag wind load at 8 m): 5000 N
Vw (wind speed): 350 m p h (156.46 mls)
Thegeomec parameters ar
Lp: the location of flag wind load (8 m)
L: leng ofthe pol(10 m)
OlIll : permiUed deflection (5 cm)
Deslgn Valibles: The design variables shown in Fig 2.8 are
do: outside diameter (XI) [ Note: x's are not used in e model]
dj: inside diameter (.x2)
Geometric R euons: T he fol1owing relations will be useful in later caJculations:
A: area of cross ction = 0.25 * ( d ~ - d f )
1: diameical moment of inertia =* ( d ~ - dt)/64
Qlt: first momcnt of area above the neutral is divided by thickness
= ( d ~ + dodi + dr)/6
Objective Function: T he objective function is the w ight of the IO-m uniform
flagpole:
Weight:j{xl' X
2
): L * A * 'Y * g
(2.10)
ConstratFunctions:le wind load per unit length (F
D
) on the flagpole is
caIculated as
F o = 0.5 * P * V ~ * C
d
* d
o
Thc bending moment at the base of the pole due to is uniform wind load on the enti
pole is
M w = 0 . 5 * Fo L * L
66 GRAPHICAL OPTIMIZATION
T he bending moment due to the wind load on the f1ag is
Wn"'L_
F - " " " P A JD
Bending (nonnal) stress at the base of e pole is
'bcnd = 0.5 ( M w + M
F
) d il
Nonnal sess due to the weight is
weight = 'Y g L
Tota1 nonna1 sUsses to be resisted for design is e sum of nonna1 stresses
computed above. Incorporating the factor of safety and the a1lowable stress from
materia1 va1ues the first inua1ity nsUint can be set up as
+wcight S nU
IFS (2.11)
The maximum shear load in the cross section is
s= W F + F
o
' " L
The maximum shear stress in the pole is
= S * QI(I t)
T he second inequa1ity constraint bed on handling e rstresses in the f1agpole
18
(2.12)
le third practica1 coostraiot is based 0 0 the def1ectioo of the top of e pole.'his
dflectioo due to a unifonn wind load on the pole is
ow = F o 01< L
4
/(8 E *
The def1ection at the top due to the f1ag wind load atis
O
F
= (2 W
F
'" L
3
_ W
F

L ' " L ' " Lp)/(E '"

The third coostraiot translates to
C3(X
1

X
2
) ow + O
F
S O.
u
(2.13)

f
/
2.3 AODITIONAL EXAMPLES 67
T o discourage solutions where d
o
< dj w e will include a geometric constraint:
84(X
1
X
2
) d 0.001 (2.14)
Side Constraints:lis defines the design region for the search.
2 c m S d
o
S 100 cm; 2 c m S d
i
S 100 c m (2.15)
M A T L A B Code: The m-files for this example are given below. A n important
observation in is problem and structural engineering problems in particular is
order of magnitude of the quantities in the constraining uations. le stress
coosnts are of e order of 108+06 while the displacemeot lenns are ofe order
of 108-02. Most numerica1 techniques s ggle to handle this range. Typica1ly such
problems 0 d to be nonna1ized before beiog soIved' It is essentia1 in applying
numerical techniques used in optimization.
This example is plotted in two p15. In the first part each inequa1ity COIint is
investigated a1one. T w o curves are showlI for each consnt. This avoids clutter.'he
side for the location of the hash mark 0 0 the constraint is detennined by drawing these
C V i n color. le blue color indicates the feasible direction. Altemately quiver
plots can be used. The second part is e consolidated curve shown. The consolidated
curve has to be drawn by removing the comments on some of the code d
commenting the de at is not needed. Since e optima1 solution cannot be c1early
establishedzoom feature ofMATLAB is used to narrow down the solution.
2.:.J....(the m a i n script file)
Chapter 2: Optimizai o n w i t h M A T L A B
Dr. P.Venkataraman
Example 2.3 Sec.2.3

graphical solution u 6 i n g M A T L A B (two d e s i g n variables)

Optimal d e s i g n o f a F l a g Pol e f o r h i g h w i n d s
R ef. 2 . 4
' - - - - - - - - - "
global statement is u s e d t o share s am e information
% b e t w e e n v a r i o u s m-files
global E L A S S I G A L L T A U A L L G A M F S G R A V
global R H O C D F L A G W S P E E D L P L D E L T
- - - - - - - - - - - " - - - - - "
Initialize values
E L A S 2 0 0 e + 0 9 i Modulus o f elasticity -Pa
SIGALL = 250E+06i % allowabe normal stress - P a
T A U A L L = 1 4 5 e + 0 6 i allowable s h e a r stress - P a lI l i 7III 1
68 GRAPHICA l OPTIMIZATION
GAM= 7860;
F S 2.5;
G R A V 9.81;
-
R H O l. 225;
C D l. 0;
FLAGW 5000;
SPEED 156.46;
" -
L P 8;
L 10;
D E L T 0.05;
- -
glval SIGALL/FS
g2val TAUALL/FS
g3val D E L T
g4val 0.001
density o f material kg/m3
% factor of safety
gravitational acceleration
density o f a i r kg/m3
% d r a g coefficient
concentrated load o n flag - N
I/S
location o f drag load o n flag - m
length o f p o l e
allowable deflection - m
right h a n d side values
for the constraints
- - - - ' - - - - - -
x 0.02:0.01:1; the semi-colon at the e n d prevents
the echo
x2=0.025:0.01:1; these a r e a ls o the side consraints
t x 1 a n d x 2 a r e vectors filled w i t h numbers
n ote a w a y t o a v o i d x 1 x 2
[X1 X2] meshgrid{x1 x2);
generates matrices X 1 a n d X 2 corresponding t o
vectors x 1 a n d x 2
f1 obj_ex3(X1 X2);
the objective function is evaluated o v e r the entire
m e s h
Constraints are evaluated
ineq1 ineq1_ex3{X1 X2);
ineq2 ineq2_ex3(X1 X2);
ineq3 ineq3_ex 3 {X1 X2);
ineq4 ineq4_ex3(X1 X2);
[C1 han1] contour{xl x 2 f1 [0 10000 50000
100000 150000 200000 250000 300000] 'g-');
clabel(C1 han1);
set(gca 'xtick' [0 0.2 0.4 0.6 0.8 l. O])
set(gca 'ytick' [0 0.2 0.4 0.6 0.8 1. 0))
xlabel('outside diameter' 'FontName' 'times'
FontSize' 12) ;
E
2.3 ADDITIONAL EXAMPLES 69
ylabel{'inside diameter' 'FontName' 'times'
FontSize' 12)
gr i d
ho l d o n
figure a n e w figure window
contour{x1 x 2 ineq1 [glval glval] 'r-');
h ol d o n
d ra w another contour a t 10 the constraint boundary
contour{x1 x2 ineq1 [O.l*glval O.l*glval] 'b-');
set{gca 'xtick' [0 0.2 0.4 0.6 0.8 l. O])
set{gca 'ytick' [0 0.2 0.4 0.6 0.8 l. O])
xlabel{'outside diameter' 'FontName' 'times' ' ...
FontSize' 12);
ylabel{'inside diameter' 'FontName' 'times' '.
FontSize' 12) ;
ho ld off
grid
the following code m a y b e usefu1 f o r the consolidated
figure. N o t u sed here _ _ uncomment below
[C2 han2] contour{x1 x 2 ineq1 [glval glval] 'r-');
clabel{C2 han2);
set{h2 'Linewidth' 1)
k2 gtext('gl');
set{k2 'FontName' 'Times' 'FontWeight' 'bold' .
FontSize' 14 Color' red')
figure
contour(x1 x 2 ineq2 [g2val g2val] 'r-');
h o l d o n
contour (x1 x 2 ineq2 [0 .1*g2val 0.1 *g2val] b-' ) ;
set{gca 'xtick' [O 0.2 0.4 0.6 0.8 l. 0))
set{gca 'ytick' [0 0.2 0.4 0.6 0.8 l. 0))
xlabel{'outside diameter' 'FontName' 'times'
FontSize' 12);
ylabel{'inside diameter' 'FontName'
'times' 'FontSize' 12);
h o l d off
g r id
[C3 han3] contour(x1 x 2 ineq2 [g2val g2val] 'r--');
clabel{C3 han3);
set(h3 'LineWidth' 1)
70 GRAPHICAL OPTIMIZATION
k 3 gtext('g2');
s e(k3 'FontName' 'Times' 'FontWeight' 'bold'
FontSize' 14 'Color' 'red')
figure
contour(x1 X 2 ineq3 [g3val g3val] 'r-');
h o l d o n
contour(xl x 2 ineq3 [0.1*g3val 0.1*g3val] 'b-');
set(gca 'xtick' [0 0.2 0.4 0.6 0.8 1. 0])
set(gca 'ytick' [0 0.2 0.4 0.6 0.8 1. 0])
xlabel('outside diameter' 'FontName' 'times'
FontSize' 12) ;
ylabel('inside diameter' 'FontName' 'times'
FontSize' 12);
h o l d o f f

[C4 han4] m contour(x1 x 2 ineq3 [g3val g3val] 'b-');
clabel(C4 han4);
set(h4 'LineWidth' 1)
k 4 = gtext('g3')i
set(k4 'FontName' 'Times' 'FontWeight' 'bold'
FontSize' 14 'Color' 'blue')
figure
contour(x1 x 2 ineq4 [g4val g4val] 'r-') i
h o l d o n
contour(x1 x 2 ineq4 [0.001*g4val 0.001*g4val] 'b-')i
set(gca 'xtick' [0 0.2 0.4 0.6 0.8 1. 0])
set(gca 'ytick' [0 0.2 0.4 0.6 0.8 1. 0])
xlabel('outside diameter' 'FontName' 'times'
FontSize' 12) i
ylabel('inside diameter' 'FontName' 'times'
FontSize' 12) ;
h o l d o f f
g r i d
[C5 han5] contour(xl x 2 ineq4 [g4val g4val]
b - -' ) i
clabel(C5 han5)i
set(h5 'LineWidth' 1)
k 5 gtext('g4')i
set(k5 'FontName' 'Times' 'FontWeight' 'bold'
FontSize' 14 'Color' 'blue')
2.3 AOOITIONAL EXAMPLES 71
the equality a n d constraints a r e
not written w i t h 0 o n the right h a n d side.
I f y o u d o write t h e m that w a y y o u w o u l d h a v e t
include [0 0] i n the contour commands
1neal e x3.....m (the first constraint)
function retval ineq1_ex3(Xl X2)
global statement is u s e d t o share same information
between various m-files
global ELAS SIGALL TAUALL G A M FS G R A V
global R H O C D FLAGW SPEED L P L D E L T
A R E A 0.25* pi*(X1.^2 - X2.^2)i m a t r i x
INERTIA = pi*(X1.^4 X 2 . ^ 4 ) ! 6 4 i m a t r i x
F D 0.5*RHO*SPEED*SPEED*CD*X 1 i
M W 0.25*FD*L*L.*X1.!INERT 1 Ai
M F 0.5*FLAGW * LP*Xl.!INER T 1 Ai
S I G W G A MGRAV*Li
retval M W + M F + SIGWi S I G W is a d d e d t o al1 m a t r i x
% elements
1nea2 e x 3 . m (the second constraint)
function retval ineq2_ex3(X1 X2)
global statement is u s e d t o share s a m e information
b e t w e e n various m-files
global ELA8 8IGALL TAUALL G A M FS G R A V
global R H O C D FLAGW SPEED L P L D E L T
A R E A 0.25* pi*(X1.^2 - X 2 . ^ 2 ) i
INERTIA pi*(X1.^4 - X 2 . ^ 4 ) ! 64 i
F D 0.5*RHO*SPEED*SPEED*CD*X1;
8 FLAGW + (FD L) i
o (X1.*X1 + X1.*X2 + X2.*X2)!6.0i
retval 8.*0./NERTIA;
i n e a1t:L.m (the third constraint)
function retval ineq3_ex3(X1 X2)
global statement is u s e d t o share s a m e infonation
between various m-files
global ELAS. SG A L L TAUALL G A M F S G R A V
global R H O C D FLAGW SPEED L P L D E L T
72 GRAPHICAL opnMIZATION
A R E A 0.25* pi*(Xl.^2 - X2.^2);
INERTIA pi*(Xl.^4 X2.^4)/64;
F D = 0.5*RHO*SPEED*SPEED*CD*Xl;
d w FD*L^4./(8*E S INERTIA) ;
d f (2.0 FLAGW*L^3 - FLAGW*L*L*LP) ./(ELAS*NERTIA) ;
retval d w + df;
i n4 e x 3 . m (the fourth constraint)
function retval ineq4_ex3(Xl X2)
retval X l X2;
Figure 2.9 displays the graphical or consolidated solution to problcm.lecodc
is available lhe m.file above (by commenting the codes crealc the new figure
windows and removing the comments on the codc is currcntly commentcd). Thc
optimal solution is not very c1ear. Figu 2.10 is obtaind by zooming in near e
neighborhood of 0.6. Sec help zoom for in uction on its use. lt can be achicved by
F1llg Polcblcm
0.81-

a
o
a

u
n
u

U
E
S

-
0.2
0.2 0.4 0.6 0.8
outside diameler
Flgure 2.9 Graphlcal solulion: Example 2.3.
2.3 ADDlTIONAL EXAMPLES 73
~
0.58 0.6 0.62 0.64 0.66 0.68 0.7 0.72 0.76 0.8
outside diamClcr
Figure 2.10 Graphical solutlon (zoomed): :ample2.3.
typing zoom at the workspace prompt and using the mouse to drag a rectangle around
e region that n ds to be enlarged. From Figure 2.10 c solution is around the
outsidc diameter of 0.68 m d tllc inside diameter of 0.65 m. Typing zoom in
workspace again will toggle back to nonnaJ state. In Figure 2.10tick
marks are placed on the figure through e command window for bcttcr intetation
of the solution.
The graphics in e above exnple were created using the same statemenls
encountered previously. Color contours were used to establish the feasible region.he
zoom feature was employ to obtain a bettcr estimatc of the solution.
2.3.3 Example 2.4
T his example is u m e arca of heat transfer. The problem is to design a triangular
fin of smallest volume that will at least deliver specificd fin efficiencies. The
graphical t i tu of this code is very similar to Example 2.3. ln is cxample the
inequality constraints are computed and retumed from a single function m.file rather
an separatc files considered in the previous example. Another ncw fealure in this
example is to invoke special mathematical functions the sesseI functions that a
74 GRAPHICAL O PTlMIZATION
available in MATLAB. s example also iI1ustrs a problem where the optimization
problem can be adequately defined but the solution is easily determined from the
equality and side consaints. In other words the problem can easily accommodate
additional demanding constraints. In large complcx mathematical models with many
dcsign variablcs it is not easy to ensure that at least one of thc inequality constrnts
IS active.
D e s'n Problem: M ze amount of material used in edi ofa scrics
of identical triangular fins that cover a givcn area and operate at or abovc the specified
cfficicnci.
Mathematlcal Model: This type of problem should be able to accommodate
several dcsign variables. In this sectione problem is set up to have two design
variabl. 1e fin material is aluminum.
Design Parameters: For aluminum
h = 50 W / m
2
[convection coefficicnt]
k = 177 W / m - K [thermal conductivity)
N = 20 [number of fins]
W=O.1 m [widofthe fins)
Fin gap is thsamc as the base length of the triangular fin.
Design Variables
b: base of the triangular fin
L: height of thc triangular fin
Figurc 2.11 iJlusategeomyoffins.
Geometric Relatlons: T hfollowing are some of tharea calculations that are uscd
later in the dcvelopmcnt ofcconsaints:
A
f
= (2N - 1)*bV footprint of e fin and gap
Ac = 2*W *[L2 + (b!2)2]112 fin area for heat transfer
Ab = (N - l)*b*W gap ar
At = N*Ac + Ab total area for heat transfer
Objective Function: T he total volume of m a ial for the fin is
j(b L) = O.5*N*W *b*L (2.16)
2.3 ADDITIONAL EXAMPLES 75
b b
FJgure 2.11 Fln deslgn for heat transfer: Example 2.4.
Constraint Funct ns: Thc heat transfer equations for the fin arc available in
Reference 7. Fins are typically mounted on a specified . 1e first cons int is one
on area:
h(b L): Af = 0.015 m
2
(2.17)
Note: is constraint essentially fixes the value of b. In e graph this constraint w il
be a straight line parallel to cLaxis.lfis wcre to happen in a problcm with scveraI
design variables it would be prudent to eliminate is variablc from the mathematical
model by identifying it as a design par leter.
e efficiency of a single fin can be established
r = (l/mL) 1(2mL)lI
o
(2mL)
where
m = (2h / )112
d e l' s are Bessel equations of the first kind. The first incquality constraint is
g(b L ) : f ~ 0.95
The overall cfficiency for the heat transfer is
TJ
o
= l - N *4
1
)(
The final inequality constraint is
g2(b L): TJ
o
~ 0.94
Side Constraints: The side consaints
0.001 S b S O .5
0.01 S L S 0.03
(2.18)
(2.19)
(2.20)
76 GRAPHICAL OPTIMIZATION
M A T L A B Code: Similar to e previous example. the hash marks on the inequality
consaint can be established by plotting two vaIues of the constraint: the limit value
and a value higher. Once yhavbeen determined. then the consolidated figure can
be established. Thcode shown here will draw aIl the functions in separate windows.
For the consolidated plot this code necds to be edited. It is left to the student m
exerclse.
m T he main script file
Chapter 2: Optimization wit h MATLAB
% Dr. P.Venkataraman
% Example 2.4 Sec.2.3

graphical solution using MATLAB (two design variables)

Minimum fin volume for efficient heat transfer
material Aluminum
- - - - - - - - - - - - - - - - - - -
global statement is use d t o share same information
between various m-files
global N H K W A R E A
" - - - - - - - - -
nitialize values
N 20 number o f fins
W 0.1 width o f fins
50.0 convection coefficient W/m*m
K 177.0 thermal conductivity W/m-K
A R E A 0.015 available fin foot print area
- - - - - - -
right hand limits for the functions
h1val AREA;
glval 0.95;
g2val 0.94;
- - - -
x1=0.001:0.0001:0.005;
x2=0.01:0.001:0.03; x 1 and x 2 are vectors filled with
numbers
[X1 X2] meshgrid(x1 x2) ;
generates matrices X 1 a n d X2 corresponding to
vectors x1 a n d x2
f1 = obj_ex4(X1 X2);
the objective function is evaluated over the entire
mesh
Constraints are evaluated
!
I't'
2.3 ADDITIONAL EXAMPLES n
eq1 eq1_ex4(X1 X2); the equality constraint
[ineq1 ineq2]
in the above note that the two inequality constraints
are obtained simultaneously. Keep i n mind
that each constraint is a matrix
ready for the plots
[C1 han1] contour(x1 x2 f1 [0.00001 0.00002 ...
0.00004 0.00006 0.00008 O.OOOl] 'g') ;
clabel(Cl han1) ;
set (gca xtick' [0.001 0.0015 0.002 0.0025 0.003
0.0035 0.004 0.0045 0.005])
set(gca 'ytick' [0.010 0.015 0.02 0.025 0.03 0.125]);
xlabel('fin length' 'FontName' 'times' 'FontSize' 12);
yl el('fin width' 'FontName' 'times' 'FontSize' 12)
grid
% hold o n
figure open a n e w figure window
p10tting the equality constraint
contour(x1 x 2 eq1 [h1val h1val] 'r-');
set (gca'xtick' [0.001 0.0015 0.002 0.0025 0.003
0.0035 0.004 0.0045 0.005])
set(gca 'ytick' [0.010 0.015 0..02 0.025 0.03 0.125]);
xlabel('fin length' 'FontName' 'times' 'FontSize' 12);
ylabel('fin width' 'FontName' 'times' 'FontSize' 12)
g r i d
plotting the first inequality constraint
figure
contour (x1 x2 ineq1 [0. 968 0 . 968] r - ) ;
hold o n
contour(x1 x 2 ineq1 [0.969 0.969] 'b-');
set(gca'xtick' [0.001 0.0015 0.002 0.0025 0.003
0.0035 0.004 0.0045 0.005])
set(gca 'ytick' [0.010 0.015 0.02 0.025 0.03 0.125J);
xlabel('fin length' 'FontName' 'times' 'FontSize' 12);
ylabel (' fin width' FontNar 'times' FontSize' 12)
hold off
grid
plotting the second inequality constraint
78 GAAPHICAL OPTIMIZATION
figure
contour(x1 x 2 ineq2 [g2val g2val] 'r-');
hold o n
contour(x1 x2 ineq2 [1.01*g2val 1.01*g2val] 'b-');
set(gca 'xtick' [0.001 0.0015 0.002 0.0025 0.003
0.0035 0.004 0.0045 0.005])
set(gca 'ytick' [0.010 0.015 0.02 0.025 0.03 0.125]);
xlabel('fin length' 'FontName' 'times' 'FontSize' 12);
ylabel('fin width' 'FontName' 'times' 'FontSize' 12)
hold off
g ri d
n b . L e le objective function
function retval obj_ex4(X1 X2)
volume o f the fin
global N H K W A R E A
retval 0.5*N*W*X1.*X2
q l J . m T he equality int
function retval eql_ex4(Xl X2)
the equality constraint o n area
global N H K W A R E A
retval (2.0N - 1)*W*Xl;
The constraints
function [retl ret2]
returns both the inequality constraints
global N H K W A R E A
c 2*sqrt(2.0)*sqrt(/K) *X2./sqrt (Xl) ;
retl (besseli(1 c)../( (0.5*c) .*besseli(O c)));
Ae 2.0*W*sqrt((X2.*X2 + .25*Xl.*X1));
A b (N-1)*W*X1;
A t N*Ae + A b ;
A r A f ./At;
ret2 (1.0 - N *Ar. (1 - r et 1 ) ) ;
T he above script files should display each function in a scparate figure window.
The inequality constnlint figures should contain two contours with c blue contour
indicating e feasible region. Fig 2.12 displays a composite plot T he solution is
detcnnined by the equality constraint and the lower Iimit onefinlengdiscussed
previously.
1m
1l
E
. .
11<'
."'
2.4 AoomONAl MATl.AD GAAPHICS
0.045
iO
0.025
1.5 2 2.5 3 3.5
fin base(m)
gure2.12 Graphlcal solulion: Example 2.4.
2.4 ADDITIONAL MATLAB G R A P H I C S
79
S
x10
-3
T his section provida brief exposure to additional features of Handle Graphics
inc1uding development of a G U I in MATLAB. T he coverage is very modest and is
inc1uded hcre for thc sake of complcteness as w e have accomplishcd our goal of
graphical optimization e previous sections. This section can be avoided
postponed or emphasized for the projects in e course as it has no additional
information for graphical optimization. It does contain reasonablc infonnation in
undcrstanding MATLAB graphics and programming. The reader is encouraged to take
somc time getting familiar with e version ofMATI.AB installed on his machine. T here
are usually significant enhancements in newcr versions of MATLAB. For example
MATLABVrsion5.3 (is book is based on Version 5.2) allows you to customize your
plot and add annotations to it rough additiona1 editing t i 1ofwindow.
You do not have to customize your figure through code alone. In this section w e will
a1so be developing the GUI using MATLAB Handle Graphics so w e have more
experien wiM ATI.AB Handle Graphics. M ATI.AB doprovidc a GUI creation tool
guideat the user is recommended to explore.
In the previous section there were only two design variables. A n cqlIy spaced
vector was obtained for each variablc based on sidc constraint. A m shgrid was
80 GRAPHICAL OPTIMIZA TION
en generated. AlI functions werc evaluated on this grid. Contour plo15 were drawn
for the functions representing objective and constraints. Except for the objective
function which displayed several contours the constraints were only drawn for a
single valuc. This procedure suggests the possibility of automation. User input will be
necessary for selccting ngc for the variables as well as thc functions at will be
plotted.
In this scction w e will create plotting facility to include a G U I that will
Obtain e number of plo15
. T he range and incrcment of the design variables
Use a file selection box/utility at the user will usto identify e function to
be plotted
Prompt the user for the number of contourfor each plot
Allow selection of contour levels
Creatc the plot
le excrcise will requirc two m-files: one 10 set up e G U I and e orto assist in
the plotting. T he user will be able to tweak thc plot in the Command window after it
is drawn. The following subsections psent a brief introduction to Handlc Graphics
and G U I controls in Version 5.3.
2.4.1 Handle Graphics
M ATI..A B Handle Graphics refer to a collection of low-level graphic routines that
actually generatc or carry out graphical changes you s inthcMA. AB figure window.
For the most part these routines are transparent d typically users nced not be awarc
of them. In fact to understand optimization and to program the techniques in M ATI..AB
it is not necessary to know about Handle Graphics since using the higher-level plot
commands such as plot and contour is sufficient to get the job done."Thesc commands
actually kick in scvcral of e Handle Graphics routines 10 display e plot on the
figure. A noteworly feature of M ATI..AB graphics is it is implemented
object oriented manncr. W hat this means is that most of the es you see on the
figurc Iike axes labels tcxt and lines arc all objec15. This mcans that most of the
graphical items on the figure have properties C be changed rough program
code. In the previous sections w e used this to manipulate color font size linwid
and the Iike.
In order to change e propees of objects it is nec sary to identify or refer to
thcm. Objects in MATLAB are identified by a unique number referredas a handle.
Handles can be assigned to all objec15 when they arc crcatcd. Some objec15 partof
other objccts. Objects that contain other objec15 arc referred to as container or parent
. T here is a definitc hierarchi s cture among objects.'he root of all
MATLAB graphic 015 is e figure objt. T he figure 0ect for example can hold
sevcral G U I controls which are known as uicontrol objec15. It 31so contains the es
2.4 AOOITIONAL MATAB GRAPHICS 81
objt . objt in tum has the line text and image object. All 0 ects in
MATLAB can be associated wia set of properties at is usually based on i15 function
or usefulncss. Different object types have different sets of properties. Each of thcs
propees is describcd through a pair of related information (name value). e first
element of this pair is the namc for the property and the second is the value
corresponding to property. W h e n e oect is created it inherits all of the default
propees. For example:
plot( sin(2*)' -go'
LineWidh' 2
MarkerEdgeColor' 'k'
MarkerFaceColor' [0.49 1 0.63]
MarkerSize' 12 )
The last four 1ines abovc represent four properties of e plotted line that is being
changed. The property narn on e Icft can be easily understood. le format size
units are in points. The color value is a row vcctor ofcolor values between zero
and onc representing the red green and blue values respectively. There are sevcral
othr propees of the plotted line which will be sct at thcir default values. Thc
MATLAB online rcference should Iist all objec15 and thcir propees.
In general. object properties clln be changed by using the ser function to change
their values. Similarly object properties c be rcad by the use of the get function. T o
inquire about the property of an 0 t:
get(handle 'Property Name')
g e(h_1 'Color')
In the above h_l is e handle of the object whose color is desircd. Similarly
s e(handle 'Property Name' 'Property Value')
set(h_l 'Color' 'r') o r set(h_1 'Color' [1 0 0])
in the above value of Color for the objcct represcnt by the handlc h_l is set 10
rcd in two ways. The sccond specifithe color red through the vector valuc. It is
expectcd that the object identified through h_1 should have a Color property. M ATt.AB
will inform you of thc error in case you are assigning a property/value tis not valid
for the object.
2.4.2 Graphlcal User Interface
T he most significant featurc of software design and development today is the
object-oriented design paradigm was briefly iIIustrated in e previous section.
T he combination of the above with event-driven programming is both a natural and a
quired fcaturc in current softwarc programs. Instead of the programs controlled by
82 GRAPHICAL OPTIMIZATION
a predetermined processing suence (batch processing) today's processing is
govemed by the user through some dirl ted mouse clicks and buttons. T hese user
selections cause certain events that instruct the software to carry out some action. The
o1 fire e evenare usually referred 10 user inte elcmc. It is
important to understand that thc same evcnts can be triggered by obtaining user
responsc to sc en prompts al the command linc (unattractivc). Using buttons to
achieve the same result i5 more glamorous although it uses more compution
sources. ln fact for commercial success it is n essary to include this mechanism of
user inte. raction in the software product . Also. users expect these elements 10 behave
in a standard way. There are many elements the user can use to interact with
program5. Many applications across several disciplines havc now defined a minimum
5tandard collection of ese elements. MATLAB 50 providcs them. T he organization
ofese elemenls and their presentation to the user in a graphlcal manner is identified
as egrphlcal user interfa. A number of GUI-based t1s.nctions. available
at the MATLAB site.
This section provides only a brief introduction to e elements used in the example.
T hse elcments are primarily used through MATLAB Handlc Graphics programming.
Consider the casc of a button at the user can push to cause some action. referred to
as a push button in thc subsequent discussion. At the outset. there ru two principal
requiremcnts at are cxpectl of the push button. First the er must be able to see
it. Second. whcn e user clicks on it (single versus double clicks can be distinguished
in many software development codes) the button must spond wi some expected
action. In M A TLAB the push button is a graphical ott. It has many propethatcan
be used to describe its app rance on thc screcn. In M A'LAB. all graphical objects can
onlyappe in a figurc window. Hencc a figure window is essential to display thc
push button. is figure window a c t s a container and can contain many user
interface elcmcnts. Considering the presentation of the button on thc scrccn it is
nessary to dccide where the button should app ron efigwwindowh o w big it
should be what label it should have (typically indicates kind of action it is likely
cause)d so on. Sincc the push button is a graphical object much of these
propeesc be set if therc is a handle available for Ihe push-button objcct. Once the
graphl 1 appearance is tak
H p u s h uicontrol(Hfig_1 'Style' 'push'
'Units' 'inches'
E J

11
2.4 ADDITIONAL MATLAB GRAPHICS 83
'Position' [2.5 0.1 0.7 .2]
'BackgroundColor' 'red'
String' EXIT'
'Callback' 'close');
In the abovc Iines of codc. Hpush is the handle to the push-button object . It is created
by using the function uicontrol . This is the function that will create the user intcrface
elements. T he same function is used to place several dierent user interface elements
through the selection of the style property. In the example below. these include label
boxcs and user editable texl boxes. Several parameters identify Lhe typof lemcnt
UEat is created and the properties UEat need to beset.Each propeny is set through a
propertynamelpropertyvalue pair. Hfilin I incs of code abovc to the
handle of the figure window in which this element w iU be placed. The type of element
created is a push-button element cstablish through Style property sng name
push." The Units used to size and position the push button is inches. The Position
property is a row vector of fo values giving the Left. Bottom. Width. Height ofthe
element in the Units sclected earlier. The default location of the origin is the bottom
lcft comer of the figu window. The BackgroundColor for the button is sct to red. T hc
label that w il be on the face of the push button is EXIT. This is the TCXl property of
the push-button object . T hc CallBack property of the push button is e string close."
T his string will be thc parameter in the eval function at is called when pushlclick
event is trigge d. MATLAB command close will be executed in the Command
workspace when the push button is p ssed. Thls will cause lhe figure window
displaying the push button to c1osc.
Only a few elements are used in is scction. The Cal1Back strings are also quite
s ighorw. Before dcveloping thc code. it is useful to observe hlerarchical
order in which the graphical objects in MATLAD are organized (reproduced from e
MATLAB documentation) in Figure 2.13. T he root implies the starting resource for aIl
graphic displays in MATLAB. Its handle usually has a value of O. The root also serves
as a container for all of the figure windows. Another exprcssion for is relationship
is that a figure window is the child of the root . From the hierarchy itis essential note
e user interface elements. u i c o nrol or uimenu c only exist wiin a
Ro+ ot
Ulcontrol Axes Ulmenu
Image Llne Patch Surface Text light
F1gure 2.13 Hlerarchical slruclure lor MATlAB graphlcal objecls.
84 GRAPHICAL OPTIMlZAnON
figure window. 80 al50 the e5 object . In the following code each interface element
is first associated wiahandle 50 eproperty of the element can be fen to
if necessary. This is especially true of the text entry boxes to invokthe Call8ack
sng5.
2.4.3 GUI Code
le following code wil1 generate the G UI. The uiwait quires thal is window be
closed before subsequent processing can take place (or uiresume be uSed in E
de) .
.Gll..yl
GUI for Plotting facility
D. P.Venkataraman Applied Optimization Using MATLAB

% Ch.2 Sec. 2.4

GUI is created within a figure window

- - - - - - -
Hfig_1 ' " figure; Hfig_1 is the handle to the
figure window
set(Hfig_l 'Color' [0.3 0.6 0.5] . . . set color
NumberTitle' 'off' no window title
Name' 'Set Range for Design variables'
'Units' 'inches' inches used to layout
% other controls
'Menubar' 'none'
'Position' [4 4 4.4 1. 5]);
Position property is [left bottom width height] in
Units selected
Ht_1 uicontrol(Hfig_1
Style' text'
'Units' 'inches'
'position' [0.1 1. 1 2.4 .2]
'String' 'Number of Functions to Plo ) ;
]

J
'

n
u
-
-

1
q
d

'
l
s
L
'
'

H
'
8
6

f

h
U

L
o
-
-
C
I

r

G
n
'
t
e
-
-
'

n
'
'
n

o
'
'
o

c
'
'

1
e
s

u
l
t
i

y
i
s

z
t
n
o

s
u
p

4
'
'
'
t

H

2.4 ADOITIONAL MATLAD GRAPHICS 85
BackgroundColor' 'white'
String' '
'Callback' 'Np str2nUm(get(Ht_2 "String"))i');
HL X1L uicontrol(Hfig_l
'Style' 'text'
'Units' 'inches'
'Position' [0.1 0.8 0.5 .2]
'String' 'x1(min)')i
HT_XIL uicontrol(Hfig_l
'Style' 'edit'
'Units' 'inches'
'Position' [0.70.8 0.7 .2]
'BackgroundColor' 'white'
'String' "
'Callback' 'xlmin str2num(get(HT XIL'
String' )) i') j
HL_XIM uicontrol(Hfig_1
'Style' 'text'
'Units' 'inches'
'Position' [1. 50.8 0.6 .2]
'String' 'incremen') i
H T X1M uicontrol(Hfig_l
'Style' 'edit'
'Units' 'inches'
'Position' [2.2 0.8 0.7 .2]
BackgroundColor' 'white'
String'
'Callback' 'x1inc str2num(get(HT_X1M'
String' )) j') j
HL_X1U uicontrol(Hfig_1
Style' text'
'Units' 'inches'
'Position' (3.0 0.8 0.5 .2]
'String' 'xl(max)')j
HT_X1U uicontrol(Hfig_l
'Style' 'edit'
'Units' 'inches'
'Position' (3.6 0.8 0.7 .2]
86 GRAPHICAlOPTIMIZATION
BackgroundColor' 'white'
String'
'Callbacl 'x1max str2num(ge(HT_X1U
String' ) ) ; ) ;
E_X2L uicontrol (Hfig_l
Style' text'
'Units' 'inches'
'Position' [0.1 0.5 0.5 .2)
'string' 'x2(min)');
HT_X2L = uicontrol(Hfig_l
Style' edit'
'units' 'inches'
'Position' [0.7 0.5 0.7 .2)
'BackgroundColor' 'white'
String'
'Callback' 'x2min str2num(get(HT_X2L'
String' ) ) ; ) ;
HL_X2M uicontrol(Hfi9_1
Style' text'
'Units' 'inches'
'Position' [1. 50.5 0.6 .2)
'String' 'increment');
HT_X2M uicontrol(Hfig_1
Style' 'edit'
'Units' 'inches'
'position' [2.2 0.5 0.7 .2)
'BackgroundColor' 'white'
'String' '
'Callback' 'x2inc str2num(get(HT_X2M'
String' ) ) ;') ;
HL_X2U uicontrol(Hfig_1
'Style' 'text'
'Units' 'inches'
'Position' [3.0 0.5 0.5 .2]
'string' 'x2(max)');
HT_X2U uicontrol(Hfig_1
'Style' 'edit'
'Units' 'inches'
2.4 ADDITIONAl MA11AIi GRAPHICS 87
'Position' [3.6 0.5 0.7 .2]
'BackgroundColor' 'white'
String' '
Callbacl 'x2max str2num(get(HT_X2U'
String' ) ) ; ) ;
Hlabel uicontrol(Hfig_1
Style' text'
'Units' 'inches'
'position' [0.1 0.1 2.2 .21
'BackgroundColor' 'y'
'String' 'After entering values please press
EXT' ) ;
Hpush uicontrol(Hfig_1 'Style' 'push'
'units' 'inches'
'Position' [2.5 0.1 0.7 .21
BackgroundColor' 'red'
String' 'EXIT'
'Callback' 'close');
uiwait
In the above code estatemenlS should be cY to interpret . Much of the code c be
edit after copy and p te operation. T he last command ui wai t instruclS MATLAB
to hold off execution until the user closes e G U I window. Figu 2.14 shows the
image of the G U I itapp rs in MATLAB. letext boxes arc initializ.ed with no values
forcing e user to set values ereby causing al1 of the CallBack strings to be
evaluated. In addition thdata entered by the user is a sing. It has to be converted to
a number. The dehe is basic. It does not verify at the user did enter a number.
is verification can be done by extending e CallBack string over multiple
statemenlS. This is left an exercise in programming for the user.
The script m-file used to select d create the pl0lS is as follows.
m . m
Script file to go with Contour plotting
Dr. P.Venkataraman Applied Optimization Using MATLAB
-

El1liE

E
Flgure 2.14 Image of GUI.
88 GRAPHICA l OPTIMIZATlON
Ch. 2 Sec 2.4

script M fi1e for 2 0 Contour P10tting o f functions

% invo1ved in optimization
A11 functions are expected t o b e avai1ab1e as
MATLAB function m-fi1es
Functions wil1 b e se1ected through the input se1ection
b o x
F unction input parameters wi11 b e meshed matrices
based o n the range o f parameters avai1ab1e from G U

The Number of p10ts a n d range for the p10t

are obtained from running the
user interface script GUI2_4 prior
t o the ca11 t o P1ot20.m

G U2 4 call the GUI

X n f oYlnfo are introduced for introducing default
values
Xinfo [xlmin x1max x1inc1 ;
the va1ues x1min x1max x1inc are avai1ab1e in
the Command workspace due t o Ca11Back
if isempty(Xinfo)
Xinfo [-4 4 0.051 defau1t
e n d
fprintf ('n')
Yinfo [x2min x2max x2inc1 ;
if isempty(Yinfo)
Yinfo 4 4 0.051
e n d
fprintf ('n' )
xvar Xinfo(1) :Xinfo(3) :xinfo(2);
yvar Yinfo(1) :Yinfo(3) :Yinfo(2);
[X1 X21= meshgrid(xvaryvar);
% x l vector
% x2 vector
matrix mesh
set defau1t number o f p10ts t o 1
i f isempty(Np)
N p 1;
end
c1; c1ear figure
rl1:
2


1
1

2.4 AODITIONAl MA1 1A B GRAPHICS 89

for Mp10t l:Np
text1 ['The function which is being p10te d must
'nbe a MATLAB'
'nfunction M - Fi1e. Given a meshed matrix input'
'nit must return a Matrix'
'nP1ease se1ect function name i n the dia10g b o x .
and hit r e t u r n n ' 1 ;
% this prints the text to the screen i n the command
window a n d serves as a prompt
the n is a n e w 1ine command
fprintf(tex1)
using the uigetfi1e dia10g b o x
[fi1e pathl uigetfi1e('c:*.m' 'Fi1es of type
MATLAB m-fi1e' 300 300);
check if fi1e is string
strip the .m extension from the fi1e s o it c a n b e
ca11ed b y the program
i f isstr (file)
functname strrep(file ' .m' " ) ;
e1se
fprintf ('nn' )
text2 [ Y o u have chosen CANCEL o r the fi1e was
'nnot accepab1e'
'nThe program needs a Fi1e t o Continue'
'nP1ease ca11 P1ot20 again a n d choose a fi1e O R '
'npress the up-arrow button t o scro11 through
'nprevious commands nn'
Bye 1'];
error (text2) ;
end
c1ear text1 text2; c1ears the variab1es text1 .
a n d text2 for reuse
c1ear Fun maxva1 minva1 strcon conva1ue oneva1ue
1abcont 1abcontU
F un feva1(functname Xl X2);
maxva1 max(max(Fun));
minva1 min(min(Fun));
fprintf (' T h e contour ranges from MIN: 12.3fMAX.
12.3f ' minva1 maxva1);
fprintf ('n' ) ;
strcon input('DO y o u want t o set contour va1ues ?
90 GRAPHICAlopnMIZATION
[ no]:' ' S ' ) ;
strconU upper(strcon) ;
if strcmp(strconU'YES') I strcmp(strco 'Y')
fprintf(' Input a vector o f contour levels );
fprintf ('n' )
fprintf('between 10.2f a n d 10.2f ' minval maxval);
fprintf ('n' )
convalue input (' Input contour level as a Vector :');
labcont input('Oo y o u want labelled contours ?
[ no] : ' s' ) ;
else
labcontU upper(labcont);
if strcmp(labcontU'YES') I strcmp(labcontU'Y')
[C h] contour(xvaryvarFunconvalue);
clabel(Ch);
else
contour(xvar yvar Fun convalue);
end
ncon input('Input number o f contours [20] :');
if isempty(ncon)
ncon 20;
labcont input('Oo y o u want labelled contours ?
[ no] : ' s' )
labcontU upper(labcont);
if strcmp(labcontU'YES') I strcmp(labcontU'Y')
[Ch] contour(xvar yv arFunncon);
clabel(Ch);
else
contour(xvaryvar F u nncon);
end
elseif ncon 1
onevalue input (' Input the single contour level ' ) ;
labcont input (' 0 0 y o u want labelled contours ?
[ no] : ' s' )
labcontU upper{labcont) ;
if strcmp(labcontU 'YES') I strcmp(labcontU'Y')
[C h] = contour(xvar yvar Fun [onevalue onevalue]);
clabel (C h) ;
else
e n d
else
contour(xvaryvar Fun [onevalue onevalue]);
REFERENCES 91
labcont input ( 0 0 y o u want labelled contours ? [...
no] :' 's');
labcontU upper(labcont) ;
if strcmp(labcontU'YES') I strc (labcontU Y' )
else
e n d
e n d
[C h) contour (xvar yvar F u nncon) ;
clabel(C h);
contour(xvar yvar F lncon)
end
if N p > 1
hold on;
e n d
Hf gcf;
e n d
figure(Hf)i
grid
hold off
In the above the plotting commands have been used before. The new MATLAB
commands are isempty clf isstr strrep m a x m i n upper strcmp and
fprint. Use the help command 10 learn more about these. Run the script file d
understand the suence of actions as well e prompts regarding the contour. In
is example the functions creale the plot do not need global values for calculation
sfe'ed from e command workspace. In other words the functions are to be
plotted can be calculated independently.
R E F E R E N C E S
1. M A T L A B The Langooge of Tecll1lical Compluing: Usillg M A T L A B Version 5
M aW osInc. 1998.
2. M A T L A B The Language ofTechnical C O uling: Using M A T L A B Graphics Version 5
MathWorks Inc. 1996.
3. Bohachevsky 1. 0. 10hn80n M. E. and Slein M. L. Generalized Simulated Annealing
for FuncL ion Optimization TeclmometricVol. 28 No. 31986.
4. Arora J. S.lntroduclion 10 Oplilllum Design M cGraw-Hi I I New York 1989.
5. Ber ' F.P.andJ0s E.R. . Jr
York1992.
6. Fox R. W. and McDonald A. T.lntroduclion 10 Fluid Mechanics 4ed.Wiley New
York1992.
7. Arpaci V. S. Conductioll Heat TrallSfer Addison-Wesley Reading M A 1966.
92 GRAPHICAL OPTIMIZATION
8. Hanselman D. and Littletield B. Mastering Matlab 5 A Comprehensive Tutorial and
Reference. MA''LA B Curriculum Series Prentice-HaJl Englewood Cliffs. NY.
1996.
9. Building GUIs wiMATLAB. online reference in pdf format. Math Works Inc.. 1998.
P R O B L E M S
(Many of the graphica1 enhancements are now possible through plotedit features in
larreleases of M A1LAB. The folowing problems are just suggestions. T he suggested
problems use additional graphical elements. and a1so ruire deleting handles. Pleas
consult M ATI..AB documentation.)
2.1 Plot the ba1lisajectory o m simple two-dimensiona1 m hanics.
2.2 Producan animated display of t11jectory.
2.3 Create a two-dimensional program that will display a random firing location
and a random target location. Allow the user to choose initia1 velocity
magnitude and angle. Plot his trectory and ca1culate his error.
2.4 Create a thredimensional program at will display a random firing location
and a random target location. Allow e user to choose initia1 velocity
magnitude and direction angles. Plot his trajectory and calculate his error.
2.5 Draw the boundary layer profile of laminar l10 w over a l1at plate. Draw lines
indicating velocity profile at ten points in the boundary layer.
2.6 Create a program at will plot velocity profile for user-specific inputs.
A lIo w the user to express his inputs through a dia10g box.
3
LINEAR P R O G R A M M I N G
The major part of this book deals with mnthemntica1 models nt nre chnrncterized by
nonlinear equations. Iike most of examples used for iIIustration us far. The
presence of a single nonlinear equation in the model is sufficient to identify the
problem from the class of nonlinear programming (NLP) problems. Mathematical
programming is anoer term that is used to describe such models and eir solution
chniques. Most of enginering design falls in this category. Having represented such
problems graphica11y in the previous chapter. it app rs the gradient and th
C vare of nctions in the mathematical model had a significant impact on
identifying e solution even though no effort was made to point out is feature
sp ifically. In fact graphical solution was obtained by inspection rather than
determined by mathematical relations or entities.
There is an equally impotcIass of problems whosmathematical model is made
up excIusively of functions that are only Iinear such as Example 1.3modeled in
Equons 1.21-1.24. The same example w modified in Equation5 1.25-1.28 50 t
a graphical solution could be di5cusSed. It is appent o m graphica1 description
in Figure 1.6 at is no curvature evident in e figure. T hese problems are
m d linear programming (LP) problems. They are natural in e subject of
operations research which covers a vast variety of models used for several kinds of
decision making. E xnple 1.3 does represent a decision-making problem. Typically
L P can be a course by itself (or several courses for tmatter). A significant portion
of such a co would be to develop mathematica1 models from different areas of
applications as there is usua11y one numerical technique at is commonly used to
solve L P problems. It is cal1ed the simplex method and is based on the algorithm by
D ig [1]. The method involves mostly elementary row operations typical1y
encountered in Gauss-elimination type m ods that are part of the numerical
93
94 LlNEAR PROGRAMMING
techniques used in linear algebra. Inis chapter only a limiled but useful discussion
isp sented. First itis important to understand L P problemsmodeling issues and
their solution as ey are different from most of e other problems in this book.
S ondm Y of current numerical techniques for N L P problems obtain their
solution by linearizing the solution at the current design point. These linearized
uations n be solved by emethS of this chapter. le ader is dir ted to y
of the books on L P for a more detailed description of the models and the techniques
as the psentation in is book is simple and brief. References 1-3 are useful for this
puose.
3.1 P R O B L E M DEFINITION
Example 1.3 will be used to define the terminology associated wiLPproblem. le
modified versioD of the problem is used 10 develop e format of the L P problem so
at w e can follow through with e graphical solution. The mathematical model has
two design variables x JX 2 thal represent e number of Component Placement
Machines of type A and B. respectively. The objective is 10 maximize the number of
boards to be manufactured. Constraint gl reprts e acquisition doJ]ars available.
Constraint g 2 represen15 e floor space constraint . Constraint g 3 represen15 e
numbcr of operators availablc.
Maximize j(x): 990 XI + 900 X 2 + 5250 (3.1)
Subject to: gl(x): 0.4 XI + 0.6 X 2 S 8.5 (3.2)
g2(X): 3 XJ - X 2 S 25 (3.3)
g)(x): 3 XJ + 6 X 2 S 70 (3.4)
xJ ~ O ; X 2 ~ 0
le problem. defined in Eqtions (3.1)(3.4) is a natural representation of
mathematica1 model in engineering design or y other discipline. B y allowing
uality (=) constJ015 dgr ter an or eq 1() constrain15 the problem could
accommodate any mathematical model at is characterized by linear functions. This
representalion can be considered the inequality form of the mathematical model
although it is not exp ssed in e standardformat for the L P problem.
3.1.1 Sndard Format
The standard format of the L P problem includes only equality constrain15. It is sct up
as a minimization problem (somc authors prcfer a maximization problem). ln
addition. all of the variables in e model e expected to be semipositive ( 0) or
nonnegative [4]. Finally the consint limi15e quantity on the right-hand side of
t
}
3.1 PR08LEM DEFINITION 95
e constraint are required to be positive (> 0). In e problem defined in Equations
(3.1)-(3.4) the changes at must be carried out include converting the objective
function to opposite type and transforming the constrain15 to equality type.
Theviables XJ and X 2 representing the number of Component Placement Machines
of model A and B respectively can be expected to be semipositive P-0)iseither
some machines will be ordered or no machines will be ordered. Note the disete
m r e of the problem is being ignor as 3.75 machines ofType A is not something
that would be ordered. The solution will be rounded to a suitable integer.1tere are
extensive discussions in disc and integer programming litera to suggest
is m a y nol be e optimum value. T he justification for rounding off is it is
convenient 10 do so at this juncture.
In the objective function j(x) coefficien15 990 and 900 are called the cost
coe.cien15. Each coefficient is associated wi a design variable :presen15
increase (decase) in cost per unit change in the related variable. le simplest way to
lsform oective function to e required format is to multiply all of the terms
by -1. Thereforee new objective function is
Minimize j(x): -990 XJ - 900 X 2 - 5250 (3.5)
In Equation (3.5) constant term on the right (-5250) can be absoed into the
lefl-hand side by defining a new objective functionL(x) without affecting eoptal
va1ues for the design variables. lt can be observed at increasing the values of the
variables will make the objective more negative which is good as w e are trying to
makej(x) as low as possible. If a particular cost coefficient had a positive sign then
increasing the amount of the corresponding variable would lead to an increase of the
objective function which is nol desirable ife function needs to be minimized. This
idea is exploited in e Simplex method presented later.
The quality constraints have to be transformed to an uality coaint. The
simplest wayachieve this changistoin duce an additional semipositive variable
for each inequa1ity constraint. Thal will create E ponding equality consnt
in a straightforward way. Consider e first cons nt:
gJ(x): 0.4 x
J
+ 0.6 ~ S 8.5 (3.2)
is can be transformed to
gJ(x): 0.4 x
J
+ 0.6 ~ + x
3
= 8.5 (3.6)
The new variable X 3 is referred to as th slack variable. T he definition is quile
appropriate because it takes up the slack of the original consaint terms when itis less
8.5.1te slack w i1l be positive as e frrst two terms will be less or ual to
8.5 (for constraint satisfaction) from e original definition in Equation (3.2). In other
words it is the difference betwecn constraint limit (8.5) and first two terms of
the constraint . If tbe values of XI and X 2 cause the constraint to be at i15 limit value.
en X 3 will be zero. This is the reason it is detined to be semipositive. B y definition.
9 6 U N E A R P R O G R A M M I N G
therefore X3 is similar to the originaJ variables in the LP problem. Transforming the
remaining constraints in a similar manner the complete equations in standard format
are
Minimize j(x): -990 x -900 X2 - 5250 (3.5)
Subj tto: 8'(X): 0.4 x + 0.6 X2 + XJ = 8.5 (3.6)
82(X): 3 X -X2 +X4 = 25 (3.7)
83(X): 3 x + 6 X2 + Xs = 70 (3.8)
x ~ 0; ~ ~ 0; x
3
~ 0; x
4
~ 0; X
s
~ 0 (3.9)
Equations (3.5)(3.9) exprl the LP problem in standard format. A n inconsistency
can be observed wi resp tto e format of the problems previously defined in the
book. Namely 8(") was reserveddefine inequaJity consints while h() defined
uaJitym aints. Here 8(X) is aJso being used for equaJitynsaints. This change
is acconunodated in is chapter only. In subsequent chapters the prior meaning for
the symbols is restored. Actually symbols are not required e constraint under
disc sion n a1so be rred 10 by equation number. T he formu1ation in Equations
(3.5)-(3.9) can be exp ssed succinct1y by resorting to a matrix notation.lis
conversion is quite straightforward. Before embarking on is transformation the
representation of the objective function needs to be chang'd again. The constant term
is brought to the righl-hand side so obj tive function n be express wi
terms only involving e design variables on the right. This is shown asL(x) below.
Once again for convenience the same symbolj(x) is used in place of this new function
in subsequent discussions. These extra symbols have been thrown in to keep the
derivation distinct. This is a small price for the ability 10 develop e rnathematicaJ
model in a natuml way and construct it to conform to the standard rep sentation in
most references. Therefore:
Minimize L(x):j(x) + 5250 = -990 x -900 ~ (3.5)
Thfollowing definitions arc used to rewrittIleproblem using malrices. Let the cost
coefficients be repr ented by cC2 . c". T he coefficients in e constraints are
represented by esym 1aij (using double subscripts). le first subscript identi
constraint number w hi1e the second constraint is the same as the variable it is
multiplying. It is aJso usefu1 to identify rst subscript with the row number and
the second subscript withe column number to rrespond wi opemtions tuse
I inear algebm. For example the aint 8(x) w il1 be rep sented
g(x): a" x + a'2 x
2
+ a'3 x
3
= 8.5
3.1 P R 0 8 L E M D E A N m O N 97
where all = 0.4. a12 = 0.6. al3 = 1. Because there are five design variables in e
problem definition additional terms wi following coefficients a'4 (= 0) and
a l5(= 0). are included in eexpression for g(x) without changing its meaning:
g(x): a
ll
X + a'2 x
2
+ a'3 x
3
+ a'4 x4 + asxs = 8.5
le constraint g2(X) can be simi1arly exp ssed as
g2(X): G z x + a
u
+ a
23
x
3
+ a
24
x
4
+ sXs = 25
T he coecients in the above equation can be sily established by comparison with
Equation (3.7). Letting eright-hand side vaJues of the constraints be represented by
b" b
2
b
m
e strd format in Equations (3.5)(3.9) c be slated to
Minimize j(x): cTx (3.10)
(3.11) Subj0: g(x): A x = b
Side n aints:" ~ 0 (3.12)
x represents the column vector of design variables including e slack variables lx"
X2... x ] T ments the ansposition symbol). c presents the column vector
of cost coefficients. [c" C2. . CJT. b represents ecolumn vector of the constraint
I imits. [b" b
2
bm]T. A represents the m x n matrix of constraint coefficients. In
is book all vectors column vectors unless ot Ilerwise identified.

M
I
N
-
-
X

1

l
J

l
l
l

c
c
.
c

r

-
E
E

-
E

-
E
E
E
a
-
-

-
p
t

-
B
E
l
l
i
-
-
J

1
2
m

U
.
L
U
r
'
E
B
E
E
E
'
E
E
E
E
E
E
a
E
E
E

-
h
u

.

E
E
E
E
E
E
E
E
'
E
a
E
E
J

M
h

a
a
a

1
h

-
W
1
2
m

a
a
a
j

r
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
E
L

A

Comparing Equations (3.10) and (3.11) with Equations (3.5)(3.8) e vectors and
matrix are
c = [-990 -900 0 0 O]T
b = [8.5 25 70]T
Azh?i :!
The foUowing additionaJ definitions have been used in the standard format. n is E
number of design variables dmis e number of constraints. In tlte example n = 5
and m = 3. Nole t Ilat the slack variables are included in the count for n.
98 LINEAA P A O G R A M M I N G
Negatlve Values of Design Varlables: The L P standard mathematica1 model
allows only for nonnegative design variables. This does not limiteapplication ofLP
in any way. T here is a standard procedure to handle negative values for variables. If a
design variable say x.. is cxpected to be unreslrIcted in signen it is defined as e
differencoftwo nonnegative variablcs
11 ..1
x. =x-; - x
I - ^ J ^ J
(3.13)
where and are acceptable L P variables. If the latter is higher than the former xJ
can have negative values. Equation (3.13) is used to replace x in the mathcal
model . Note that is procedurc also increases thc number of design variablin the
model .
Type of Constraints: The less an or equal to (::::) constraint was handled
naturally in developing the L P standard mathematical model.le standard numerical
technique for obtaining the solution requires thateslack variables be posivetostart
and b also be positive throughout . If there is a constraint is required to be
maintained above a specified value for example
g(x): a" x + a
J2
xz+ a'3 x
3
b (3.14)
then it appes natural to introduce a negative slack variable - X 4 to change Equon
(3.14) to an equality constraint:
g(x): a
ll
X + a
l2
x
2
+ a'3 x
3
- x
4
b
J
W hile it is possible to ch ge e sign on X 4 by multiplying with -1 roughoutis
causes eright-hand constraint limit to be negative which should be avoided. In such
cases another variable xs is included in the exp ssion. In this case two additional
variables are now part of the model:
g(x): a" x + a'2 x2 + a
l3
x
3
- x
4
+ X
s
= b
J
'he variable Xs is called artificial variable. Artificial variables also introduced
for ch equality constraint is naturally present in the model. The use of these
variables w il1 be illustrated later through an xample.
3.1.2 Modellng Issues
The L P program is characterized by e mathematical model defind in Equations
(3.10)(3.12). Due to the linear nature of a11 of the functions the discussion
of the solution need only consider coefficients of the functions-the variabl
3.1 P A O B L E M OEFINJTION 99
themselves can be ignored. These coefficients which are present as vectors or a
m axcan be manipulated to yield solutions using concepts from linear algebra.
A look at the graphical solution to the L P problem suggests that the solution to the
problem if it exists must lie on the constraint boundary because the curvature of the
oective function is zero everywhere (it is linear). Furthermore the solution to LP
problem is primarily influenced by Equation (3.1 1). Being a I inear equation. topics
from linear algebra are necessary to underst d some of e implications of e
mathematical model.
Ifn = mat is the number of unknowns is the same as the number of variables.
there can be no scope for optimization as Equation (3.11) will determine solution.
provided one exists'. In the fol1owing. many of concepts are cxplained through
simple graphical illustration or by consideration of simple examples. They mostly
deal with examples of two variables (x" X2). For example considrtwoconsaints
gl: xl+Xz = 2
g2: -x
1
+Xz = 1
Figure 3.1 illustrates the graphical solution to the problem. The solution is at x = 0.5
and X 2 = 1.5. This is independent of the objective function (whatever it may be). ln
matrix form. A and b are

U
r
-
-
E

-
L

-
L
U

-
-
l
E
-
E
A
t
-

-
-
A
-
-

A

le existencofa solution depends on the rows of A (or columns of A - w e will deal
with rows only forconvenience). Ifthe rows of A are linearly independent then there
is a unique solution to e system of equations.
ThDefineanother problem as follows (only changing b o m the above example)
e new problem is
g : x += 2
g2: 2 x
l
+ 2 x
2
= 4
The new function g2 represents twice e value of glThis means g2 can be obtained
m 8J' T herefore 82 is said to depend on 81Technicallyetwo quations e
linear dependente ac dependence of the functions is not a concem as much
as the facteyarnotlinearly independent . Both 81 and 82 establish the same line
graphically. W hat is the solution to the problem? There are inflnite solutions as long
as the pair of (XlX2) values satisfies 81In Figure 3. 1 any point on the line
representing 81 is a solution. Not only (0.5 1.5) but (20) (02) are also solutions.
From optimization perspectiveis is quite good for unlike the previous case of
uque solution the objective function can be used to find the best solution. 1t would
be better however if the choices were finite rather than infinite.
100 LINEAR PROGRAMMING
4
n
H

R
M

u

H
M

u
-
e
e

a
u
y

l
o
u
u

R
M
R
H

U

4 2
m

-
i
g
.
f
-
n

e

F
M

3
21
)('" 1

1
-2
-3
J o
X
1
2 -2
Flgure 3.1 Unique solution-Unear independence.
While severaJ cxamples can be constructed to illustrate linear dcpendence it
appears reasonable to assume thatfora 11 ercise optimizationthe number of
constraints cannot equal the number of ulIknowns (m n). In the previous illusn
n = 2 d m = 1 (there was only one effective consaint). It is useful to recognize that
is corresponds to the case n > m.
If m > n there are more equations than the number of variables. lis implies
the system of equations represented by Equation (3.11) is inconsistent set or has a
redundant set of cquations. Consider the following illustration for m = 3 and n = 2
which uses the same 8 and g2 as in thc first illustration while adding a new 83:
g: X.+X2 = 2
g2: -x
1
+X2 = 1
g3: X + 2 x2 = 1
Figurc 3.2 illustrates the set of equations is inconsistent since a solution donot
exist. If one were to exist then the three lines must pass through e solution. Since
eyar all sght linesere can be only one unique intcrsecting point In Figurc
3.2 only two of ththree lines interect at different points.
3
3.1 PROBLEM DEFIN10N 101

F
V
v
b
/
/

K
A
A

m
d

/
/
/
/
4
Case n a 2; m a 3: No solutlon; Innslsten! 5e!
3
) ( ~ 1

1
-3
4 -2 -1 o
X
1
2 3
Flgur 3.2 No solulion Inconslstent set 01 equations.
Rcdefine g3 as
g3:XI + 2=3.5
This new 83 is e dashed line in Figure 3.3. N o w a uniqusolution to eproblem
at (0. 5 1.5) is established. lis is o the solution established by consideration of 8
d g2 alone. T his implies g3 is redundant. If 81 is multiplicd by 1.5 82 is
multiplied by 0.5 and both added. thc result is g3 defined above. That is
g3 = 1.5 gl + 0.5 g2
can be obtained by linearly combining g and g2. is. adding constant multiples
of nctions. This is another example of linear dependence. This linear
dependence and redundallCY is also iIIustrated by the set
g: X + "2 = 2
g2: -x= 1
g3: 2x
1
. .2x
2
= 4 (same as gl)
102 LlN E A R P R O G R A M M I N G
3.1 P R O B L E M DEFINITION 103
For n == m and with gl: xl + x2 == 2 and g2: X1 +X2 = 1e matrix A is
Case n a 2; m. = 3: UnlqU8 solullon Unoar dependonc9 -
E
E
E
E
E
E
J
1
1

=

A

4
h

h

-
h

-
E-

a and its detenninant is

det(A) = IAI = (1)*(1)(1)*(1)=2
- "3.5
From theorems in Iinear algebra [5] if det(A) is not zero in which case the matrix A
termed nonsingular a unique solution exists to the set of equations. T his was uc for
e example iIIus ted above as only solution w located at (0.5 1.5) shown
graphically in Figure 3.1.
Consider the example with gl: XI + X2 = 2d g2: 2x
1
+ 2x
2
=4. In is case
.r 1

det(A) = IAI = (1)*(2) (1)2) == 0

-3
-3 -2 O
X
1
2 3
lf det(A) is zero that is matrix A is singular re are either no solUlions or infinite
sollllions. For is exampleere were infinite solutions. Dctcrminants of
higherrder squ'e matrices are evaluated by setting up lower-order determinants
until eyare duced to a 2 x 2 determinant which is evaluad as above. A ny
textbook on engineering mathematics or linear a1gebra should illustrateis technique.
It is not produced here.
If n "" m discussion of the existencc of solutions requi s additional concepts Iike
rank of a matrix and augmenled matrix. Since case n > m is of interest in
optimization only that case is employed in subsuent ilIustration. A useful
discussion will need at 1east three variables and two equations. Three variables will
deny the use of graphics to develop the folIowing concepts. Using the set
Flgure 3.3 Unlque solutlon IInear dependence.
T his suggests e concept of redundancy can be associated with linear
dependence. The discussion of linear dependence and independence was established
g equations above. The same discussion can take placc by reasoning on the
coefficicnts themselvcs. In this case the coefficient m ax A should lead us to e
same concIusion rcgarding e linear independence of a set of Iinear equations. e
concept of a delerminanl is n cssary to develop the criteria for linear dependence.
gt: x
1
+ X
2
+ X
3
= 3
g2: - X+ 0.5 x
3
= 1.5
Determlnant: The determinant is associated with a squarc matrix. For a general 2 x
2 matrix A where
the matrices

E
E
E
E
B
E
E
-
J

.
a
a

-E

a
r
M
l
M
L

=

A

E
E
E

d
3
5

-

M
1

n
u

I
l

-
-

-
- A

E
E
E
E
E
E
J
3
5

--a
F
E
E
-
E
E
a
E
Z

h
u

-
-
E
E

E
.

J
I

I
l

-
-
a
-
-
a

f
l

L
=

A

(3.16)
a a
1= d d t( A)== a
a21 2
The new matrix A. is called e augmented matrix-the columns of b are added to A.
According 10 rems of linear algebra (presented here without proof):
e determinant is expres'ied and evaluated as
(3.15)
If the augmented matrix (A) and the matrix of coefficients (A) have the same
rank r < nen erM many solutions.
104 LlN E A R P R O G R A M M I N G
. Ifeaugmented m ax(A)d ematrix of coefficients (A) do not have the
same rank a solution does not exist.
If the augmented matrix (A) and the matrix of coefficients (A) have the same
rank r = 1 1 where m = n then there is a uniqusolution.
R a n k 01 a Matrix: While there are formal definitions for defining the rank of e
m axa useful way to determine rankislookedetennt. 'herankofa
matrix A is the order of e largest nonsinlar square submatrix of A that ise
largest submatrix with a determinant other anzero.
In the example above the largest square submatrix is a 2 x 2 matrix (since m = 2
and m < n). Taking the submaix which includes the first two columns of A e
determinant was previous established to have a value of 2 therefore nonsingular.
Thus the rank of A is 2 (r = 2). le same columns appear in A making its rank also
2. lereforeinfinitely m y solutions exist (r < n). One way d ermine E
solutions is to assign (n - r) variables arb values and use them to determine
values for the remaining r variables. T he value n - r is also often idntified as
deg ree of freedom (DOF) for the system of equations. ln the particular example above
e D O F can be identified with a value of 1 (i.e. 3 - 2). For instance x 3 can be
assignd a value of 1 in which case XI = 0.5 dX 2 = 1.5. O n thother hancJ. if X 3 = 2
then XI = 0.25 and X 2 = 0.75. The above cases illustrate that for a D O F of 1 once x 3 is
selected. X l d X 2 can be obtained through some further processing.
Another example for iI1ustrating e rank is e inconsistent system of equations
inoduced e lier where n = 2 d m = 3 reproduced for convenience
gl: x
1
+ x 2 = 2
g2: -x
1
+= 1
g3: x
1
+ 2x
2
= 1
The matrices for this system are
A = H l } b = [ I } A ' = H i l ]
The rank of A cannot be greater 2since n = 2 and is less th m = 3. From prior
calculations e determinant of the first two rows of A is 2. This is gr ter than zero
and threfore the rank of A is 2. For A. the determinant of the 3 x 3 matrix has to be
examined. The determinant of a general n x n square matrix A Can be calculated in
terms of the c'actors of the matrix. The cofactor involves edenninantof a minor
matrix-which is an (n - 1) x (n -1) matrix obnedby deleting an appropriate single
row and a single column from the original n X n m aix A multiplied by 1 or -1
depending on eto ofthe row and column values. References 5 and 6 c provide
morinformation. Heredeterminant for the 3 X 3 matrix is defined for iIlustration
3.1 P R O B L E M DEFINITION 105
in e book. le (+1)d (1) below ar obtained by using ( 1); wher i andj
represent erow and column values of the coefficient multiplying the value
al1 a al"l l -
A = 10;; lln iln(+l)a111273+(l)a1212J2+(+l)a13
~ 1 ~ 2 a
33
1 I ~ . ~ I 1--~ I - ~ : l l
(3.17)
lerefore
2
A

L
-
-
a
-
-
a

.
+

-
z
a
'
E
a
'
E
A

t
A

-
a
a
-
-
A

1
1

'
a

-
-
E
A
-
-
A
2
'

'

M
I
l
l

- A

= (1)(0) + (-1)(-2) + (1)(3) = - 1
A. is nonsingular and its rank is 3 w hi Ie the rank of A is 2. The system of equations
will have no solutions from the above thorcm and can be seen graphically in Figure
3.2. For the sake of completeness using the previously defined dependent set of
equations:
gl: x
1
+ x
2
= 2
g2: - x
1
+ x
2
= 1
g3: 2x
1
+ 2xa = 4
which yields the following m aces:
A =[-l l } b = [ ! } A ' = H 1 !]
The rank of A is 2. Note using the first and last row of A for the calculation of the
determinant yields a singular 2 x 2 submax. T he determinant of A is
n
u

-
-
0
0

U

+

=

'
A

4
2
A

(

+

'
A

.
E

+

'
i
a
T
a

.
z
'
Z
''z
-

a
'
A
a

'
1

L
'

A
-
-
a
-
-
E

-
A

which makes A. singular. However the rank of A is at least 2 due to presenceof
lesubmaix which determined the rank of A. From the above theoremer k
of A is equal to the rartk of A. which is 2 and this is less m = 3. Itcan be concluded
tere are infillite solons for system of equations.
106 LlN E A R P R O G R A M M I N G
A useful observation of matrix A thepceding example is two of the rows
are a constant multiple of each other. s makes e m aces singular. The same
holds for two columns at are multiples of each other.
Un/t Vectors: Real-world optimization problems frequently involve a large
number of design variables (n variables). In real L P problems. enumber of variables
can be staggering. For a successful solution to e opzation problem. it is
implicitly assumed that the set of design variables is linearly independent . T hat is.
there is no direct dependence among any two variables of the sct. In is case. the
abstract ~ p a c e spawned by the design variables is cal1ed en-dimeflsioflal Euclidean
space. T he word abstract is uscd herc to signify that it is a construct in the imagination
as only three variables can be accommodated in familiar 3 D physical space. le
design vector is a point in is space. T he Euclidean spa is a direct extrapolation of
the physical geomecal 3 D space t surrounds us. This space is termed
Cartesian space. For an i11ustration consider the point (232) in Cartcsian space or in
the rectangular coordinate system. In Figure 3. 4 the point is marked P. The adof
numbers within parentheses denotes the values along the x" X 2and X 3 es
spectively.
T he Cartesian space is identifi by a set of mutually peendicul lines or axes
markedx"x2' andx3The point wherc they intersect is the origin (0) of the coordinate
syslem. The point P can also be associated Witll the vector OP. drawn from tlle origin
o to e point P. Very often a point and a vector-two very different entities-are
used synonymously. From elementary vector addition principles the vector O P can
becons cted by consecutively joining three vectors along the thr coordinate es.
In the figure this is shown by laying out a vector along the x axis adding a vector
along e X 2 axis to it and finally adding thc third vector along thc X 3 axis. The
additional small arrows in thefigure are e Lmit vectors. ley have a magnitude of
one and a direction along each of the es. These unit vectors are labeled e" e2 and
e 3 in the figure. Mathematically point P (232) and the vector addition can be
rcpresented(column vectors)
X3
2
3
X2
Flgure 3.4 Rectangular coordlnate system.
3.2 GRAPHICAL S O L UTlO N 107

d e

+

q

-
e

4
3

+

d
L
'

E
E
-
-
E
E
E
B
E
E
S
E
E
J
n
u
n
u
'
E

F
E
E
-
-
E
E

-
B
E
E
-
L

E
B
B
-
a
'
E
B
E
E
-
-
J
n
u
'
n
u
r
l
l
l
1
1
1
L

4
3

+

1
1
1
1
1
1
1

'
z
n
u
n
u

r
'
E
E
E
a
E
a
E

-
E
L

L
=

E
E
E
E
E
E
E
E
E
E
E
E
J
r
h
H
h

(3.18)
The advantage of unit vectors is ey can cstablish y point in the Carsianspace
through an approp te triad of numbers sigfying coordinate values or 10 tion.
Acally any point in eCartian space can be detennined by any set of thl
linearly independent vectors. T hese vectors are termed the basis vectors. In this case.
the simple connection between the point and the elementary addition illustrated
above will not be availabl.Fu ermoreough the methods of I inear algebra these
three vcctors can be reduccd to the unit vcctors through elementary row/column
opcrations also called Gauss-Jordan elimination or reduction. In ndimensional
Euclidean space spanned by the n design variablese point p"(p.. P2 . . . p") will
be represented by a collection of n values enclosed in parneses. The corresponding
unit vectors are dfined as
o
o
e"=I.
(3.19)
le vector from origin (0) to P" is P sented as
O P = p e + P2e2 + . . . + p"e
ll
These new definitions will be sufficient to understand the techniques used to solve L P
problems.
3.2 G R A P H I C A L S O L u n O N
le graphical discussion is again limited 10 two design variables in original
problem description. It is used here to develop certain geometric ideas related 10 LP.
InLP thintroduction of slack variables will easily exceed this limit of two variables
for graphical illation. so the graphical solution is illus ted with spect to e
original variables in the design problem. MATLAB is once again used to a w e
graphical solution. The problem is not reduced 10 the standard format for the graphical
soluon. It is important to do 50 for a numerical solution. Since Ihe primary graphical
element is e straight li1Le a function for drawing 1ines is developed through th
function m-file drawLine.m below. is function is also used to provide more
expos to MATLAB programming.
drawLlne.m: is a function m file for drawing strght lines. le 1ine 10 be
drawn is represented as
108 LlN E A A P A O G A A M M I N G
+ b y = c
Since x y are usedsymbols for variables xl represents the lower range on x while
X 2 represents the upper rangc. Limiting Iines/constraints can bc drawn by setting either
a or b to zero. Color (red) is uscd to draw a p'allel line at 10% increase/decreasc in
the original value of c (depending on the direction of the original inequality) instead
of hash marks. Limit constraints are in magenta and thobjective function is in blue
dashed line. The new M A 11.AB command here is e line command. You are
enco aged to s k help on the command. T he code:
drawLine.m
Drawing linear constraints for L P programming problems
Dr. P.Venkataraman
Optimization Using Matlab
Chapter 3 linear Programming

Lines are represented as: a x + b y c ( c > = 0)

x1 x2 indicate the range of x for the line
typ indicates type of line being drawn 1 (<=)
9 (>=)
n (none)

The function will draw line(s) i n the figure window

the green solid line represents the actual value
% of the constraint
the red dashed line is 10 larger o r smaller
(in lieu of hash marks)
the limit constraints are identified in magenta color
the objective function is in blue dashed lines

function drawLine(x1 x2 a b c typ)

recognize the types a n d set color
i f (typ 'n')
str1 'b';
str2 'b'
cmult 1;
else
str1 '" 'g'
str2 '" 'r';
end
values for drawing hash marks
depending o n the direction of inequality
if (typ -= 'n')
if (typ = = ' l' )
cmult +1;
3.2 G A A P H I C A L S O L U T I O N 109
else cmult -1;
e n d
end
set u p a factor for drawing the hash constraint
if (abs(c) > = 10)
cfac 0.025;
elseif (abs(c) > 5) & (abs(c) < 10)
cfac 0.05;
else
cfac 0.1;
end
i f (c 0)
cdum cmult*O.l;
else
c dm (1 + cmult* cfac)*c;
end
if b 0 then determine end points of line x line
if (b -= 0)
y1 (c - a*x1)/b;
y1n (cdum - a* x1)/b;
y2 (c - a* x2)/b;
y 2 n (cdum a*x2)/b;
else
identify limit constrains b y magenta color
str1 'm'
str2 'm';
y1 x1; set y1 same length as input x1
y2 x2; set y2 same length as input x2
x1 c/a; adjust x1 to actual value
x2 c/a; adjust x2 to actual value
y1n ' " 0; set y 0;
y2n 0; set y 0
end
i f (a 0)
str1 'm' ; set color for limit line
str2 'm'; set color for limit line
end;
draw axis wit h solid black color
h h line( [x1 x2] [0 0]);
110 LlNEAR PROGRAMMING
set(hh'LineWidth' 1 'Color' 'k');
h v 1 ine ( [0 0] [xl x2] ) ;
set(hv'Linewidth' 1 'Color' 'k');
start drawing the lines
h l line ([xl x2] [yl y2] ) ;
i f (tp 'n')
set(hl 'LineWidth' 2 'LineStyle' "' 'Color' strl);
else
set(hl 'LineWidth' 1 'LineStyle' '' 'Color' strl);
e n d
if ( b - = O)&{a - = 0)
text(x2 y2 num2str(c)) ;
e n d
i f ( b == 0) I (a == 0) I (y p 'n')
grid
return e n d
grid;
h2 line ([xl x2] [yln y2n] ) ;
set(h2 'LineWidth' 0.S 'LineStyle' ':' 'Color' str2);
g r i d
h o l d o n
3.2.1 Example 3.1
The problem from page 94 is reproduced 0 0 more for cooveoieoce.
Maximize j{X): 990 XJ + 900 X2 + 5250 (3.1)
Subj tto: g.(X): 0.4x. + 0.6x2 S 8.5
(3.2)
g2(X): 3xJ - X2 S 25
(3.3)
) : 3x. + 6 x2S 7 0
(3.4)
x. 0; .i 0
Thgraphica1 solutioo is shown in Figure 3.5. The solution rcad from thc figu (you
C Zm e arcaof solutioo) isxj = 10 andxi = 7. It is c intersection of the active
constraints (3.3)d (3.4).le actual values are xj = 10.48 d xi = 6.42. Since an
integra1 number of machines have to be ordered the solution is adjusteda
neighboring integer va1uat satisfies lhe contraints.
8Or'
60
3
-40
0
3.2 GRAPHICAL SOLUTION 111
5

. .

-- - . .
.

. .

.
.

. .
.. .
h
. . .
-
. J -5250
5 10 15 20 25 30
Numbor 01 machlnas 01 Iype A
Flgure 3.5 Graphlcal sOlulion Example 3.1.
3.2.2 Characterlstlcs of the Solutlon
beglmetry evidnt in the graphical solution of Example 3.1 is used to explain some
of the concepts associated with L P and ilS numerica1 technique. Figure 3.6 is E
graphicaJ rcpsenlation of constraints involved in Example 3.1 (same as Figure
3.5 without the objectivc nction). In order to late egcometryLP conceplSthc
stanrd forrnat of L P is n S y. The standard forrnat established before is
Minimize j{X): -99Ox
J
- 900x
2
- 5250 (3.5)
Subject to: g.(X): 0.4x. + 0.6x
2
+ x) = 8.5 (3.6)
g2(X): 3x. - X2 + X4 = 25 (3.7)
g3(X): 3xI + 6x
2
+ Xs = 70 (3.8)
-x.
(3.9)
In Figure 3.6 the constraints x. 0 and X2 0 are addcd to the three functional
constraint Iines (3.6)-(3.8). The ha.'!hed area is thc feasible regionat is thdesign
112 LlN E A R PO G R A M M I N G
50
40
30
! 20

i 10
2

-30
0 5 10 15
Numbor 01 Machln08 01 Typo A
20 25
Flgure 3.6 Feasible region Example 3.1.
space in which all of the constraints are satisfied. The circles exaggerate the points of
intersection of five constraints taken two at a time. Many of them are numbered Pl
P2. . . P6. AII of these points of intertion can be associated with a certain property
conceming the values of the variables. Note there are five dcsign variables (n = 5) and
threc functional constraints ( m = 3). For enplc:
PI: (x
1
= 0 x2 = 0 X ) = 8. 5 x
4
= 25 X
s
= 70)
P2: (x
1
= 0 = 11.67 X ) = 1.5 x
4
= 36.67 X
s
= 0)
P5: (x
J
= 10.7 x2 = 7.05 X ) = 0 x
4
= 0 X
s
= -4.4)
P6: (x
1
= 0 x
2
= 14.17 x
3
= 0 x
4
= 39.17 Xs =15.42)
The values ofevariables are obtained aseintetion ofthe constraints taken two
at a time. Ine above Iist for each point exactly two of the variables are zero. T he
number 2 corresponds to the value of n - m. n represents the number of variables and
m the number of constraints. Points P5 and P6 are infeasible because one of the
variables has a negative value.
Baslc Solutlon: A basic solution is one obtained by setting cxactly n - m variables
to zero. In Figure 3.6 aIl of the points identified by the circles rcprcscnt basic
solutions. The points choscn above are all basic variables. In general for n design
variables and m consintsnumherof basic solulions is givcn by the combination
3.2 G R A P H I C A L S O L U T I O N 113
()=
T his would yield 10 points for our exnplewhich is shown in Figure 3.6.
Baslc Varlables: T he set of variables in the basic solution that have nonzero values
are called basic variables. Correspondingly the set of variables in the basic solution
have the value of zero are called nonbasic variables. For the point Pl X J dX2
are nonbasic variables while X 3X 4andxs ar basic variables.
Basic Feaslble Solutlon: T his is a basic solution that is also feasible.'hese are
e points Pl P2P3 and P4 in Figurc 3.6. In L P the solution to the problem if it is
unique must be a basic feasible solution. The basic solution can aIso be considered
geomeca Ily as a corner point or an extreme point of the feasible region.
Convex Polyhedron: T his is a bounded region of the feasible dsign space-e
rcgion defined by quadrilateral comprising the points PI P2 P3 and P4 in Figure
3.6. The term cOllvex set represents a collection of points or vectors having the
following property: For any lwo points in the sct (or within a rcgion) if all ofthe points
on the linc connecting two points also Iie in c same region the region is a convex
set. Imaginy line drl1w n in lhc region tlfinetl by e quadrilatend whose comer
points are Pl P2 P3and P4. From Figure 3.6 it is c1ear the line w ilI still be wiin
the region established by quaiJater- making it a convex set.
Optlmum Solutlon: This is a feasible solution at minimizes the objective
function-point P4 in Figure 3.6. In L P the optimum solution must be a basic
feasible solution.
Basls: 'he basis P sents c columns of the coefficient matrix A coespond
to the basic variables. They form thc basis of the m-dimensionaI space. T hey are
termed the basis vectors.
/
/
J
6
4
2
2
p
l
z

/
4
1
2
4

h
r
Y
Y

Canonlcal Form: 'he basis vectorduc to unit vectors through row/column

operations (or Gauss-Jordan eIimination). T he basic feature of e numerical
technique for L P (Simplex method) is a petitive proced e starting from an initial
basic feasible solution and determining e best neighboring basic feasible solution
that improves the objective. The procedure is carried on until the optimum sol onis
reachdor if it is determined no solution is possible. Thc canonical form is used
for rapid1y identifying the solution. Each iteration in procedure can be described
mathematicaIly as follows.
T he starting constraints ' c organized as
[A]II[ 1 = [b]
After the Gauss-Jordan elimination (3.20) is assembled as
(3.20)
114 LlNEAA PAOGAAMMING
m [X]m + [R] n-m)[X](s) = [b]
The set of m design variables in the first term the basic variabJcs. T he sct of (n -
m ) dcsign variabJes in c sccond tcrm are nonbasic valucs. In cSimpJex method
thc nonbasic variablcs are summarily set to zcro.
3.2.3 DintSol o n T y p
T hcrea at most four different resuJts at can be expccted for the soJution of the L P
probJem: (1) a unique solution (2) infiniteJy many soJutions (3) unbounded solution
and (4) the possibility at there is no soJution.
(1) Unique solution: le example used for discussion has a U sique solution
(Fi 3.5). le condition necess y for is to occur is e objective
function and the constraints have dissimilar sJopes and the feasibJe region is
boundcdlcJosed. Geometrically this can be visually expJained as the movement
of e line represcnting thc objective function pullcJ 10 itseJf in a favorabJe
direction until it remains just in contact with one of the feasibJe comers (bic
feasibJe soJution) of the feasible region. The fe ibJe region is identified
uistinctly in Fig 3.6. A simplrcisc uf imagining diffe objectiv
function Jines should convince the viewer that several different objectives n
be defined to locate the unique solution at P 1 P2P3 or P4. This 2 0 geome
and construction can be used to understand the extension to n dimensions. H e
the I ines w ilJ be represented by lIyperplanes.
(2) lnfinile solution: In order for is to occure objective function must be
parallel to one of the consints. For example in Figure 3.6 let the original
problem be redefined so the objective function is parallel to the constraint
h Any point on the constraint and lying between the line segment dcfined by
e points p 2 and P4 is an optimal solution to the problem. They will yieJd e
sarne value ofve.
(3) Unbounded sollllion: Inis case the feasible region is not bounded. In
Figure 3.6 if the constraints gl and g2 were not part of the probJem formuJation
en the feasibJe region is not bound on the top. Referring to Figurc 3.5. E
obj tive function can be shiftedunlilTUted higher values. In practice there
will be an upper bound on range of the design variabJes (not ptofstandd
format) that w ilJ be used to cJose is region. in which case solutions oftype (1)
or (2) can be recovered. T he prcsence of an unbounded solution a1so suggcsts
t the formulation of problem may be lacking. Additional meaningful
constraint(s) c be accommodated to define the soJution.
(4) N o solulion:Figure 3.6 is used to explain is possibiIity. Consider E
direction of inequality in 81 is changed to thc opposite type (). The feasiblc
region wircspect to the constmints gl and 83 is to the right of XI = J 5.0. This
3.3 NUMERICAL SOLUTION-THE SIMPLEX METHOD 115
point is not feasible with respect to econstraint 82Thereforethere is no point
is feasible. T here is no solution to e problem.
In the above discussion. e two-variable situations provided an obvious
cJassification of the solutions. In practice. L P models are large with over hundreds of
variabJcs. Modeling and transcription errors may give rise to many of the above
situations. T he genemtion of solution is based on numerical tcchniqucs of linear
algebrawhich is often sensitive to e quality of the matrix of coefficients. FiJtering
out errant data is usually a significant exercise in the search of optimal solutions.
3.3 N U M E R I C A L S O L U T I ON -T H E S I M P L E X M E T H O D
T he standard numerical procedure is based on e algorithm due to Oantzig
mentioned earlier. It is refe 10 as e Simplex method. The procedure is related
to solution of a system of Iinear equations. T he actual application of
procedure can be associated with the Gauss-Jordan method from linear algebra.
where the coefficint rows are transformed through e1ementary multiplication and
addition. Most mainframe computcr installations usually carry software that will
help solve L P problems. MATLAs also provides procedures 10 solve L P problems
in its Optimization TooJbox. In this scction the Simplx method is applied to
simple problm primarily tu undcrstunu thc pl'Ogrumming and geomctric features.
T his will be used when w e discuss direcI techniques for nonlinear problems. In the
next section the Simplex method is introduced in detaiJ with explanations. In
subsequent sections MATLAB. or spreadsheet programs I ike Excel can also be used
to implement the SimpJex method.
3.3.1 Features of the Slmplex Method
In this section. the machine seJection exnpJe introduced earJier is set up for applying
the Simplex m cod. It is instructive to notc the method is iterative. Given a starting
point it will march forward through improving designs un it has found the sohon
or cannot proceed further. For completeness the original problem is rewritten here:
Maximizc j(X): 990x
l
+ 90Ox
2
+ 5250 (3.1)
Subject to: 81(X): O.4xl + 0.6X2 :s; 8.5 (3.2)
(3.3)
(3.4)
4
5
F
i
r
-
2

T
I
z
-
-

g2(X): 3xI - X2 :s; 25

83(X): 3xI + 6x
2
:S; 70
x
1
~ O ; x 2 ~ 0
The problem was aJso transformed 10 the standard format as follows.
116 LlNEAR PROGRAMMING
Example3.1
Minimize J(X): -99Ox
I
- 90Ox
2
- 5250
Subject to: g(X): 0.4xl + 0.6x
2
+ x3 = 8.5
(3.5)
(3.6)
g2(X): 3x -x2 +X4 = 25 (3.7)
g3(X): 3xI + 6x2 + Xs = 70 (3.8)
x
1

0; x
2
~ O ; x
3
0; x
4
0; X
s
o (3.9)
X3 X4 and Xs are slack variables.
The Simplex method is used on the problem being expressed in the standard
format.Thefollowing information is useall in organizing the calculation as well as
cognizing the motivation for subsequent iterations. 1ese refcr to a problem for
which a unique solution exists. W hile m y of the items bclow wcrc in duced
earlier they are referenced here for completeness.
Thc number of variables in thc problem is n. includes the slack and surplus
variables.
1e numbcr of constraints is m ( m < n).
1e problem is a1ways to minimize the objectivc functionf
The number of basic variables is m (same as number of constraints).
The number of nonbasic variablcs is n - m.
The set of unit vectors expected is e" e 2 e
m

The column of the right-hand side (b) is positive and greater an or equal to
zero.
T he ca1culations are organized in a table which is called a tableau or Simplcx
tableau. It will be referred to the table in this book.
Only the va1ues of thc cicients ar necessy for the calculations. The table
crefore contains only coefficicnt va1ues the matrix [A] in previous
discussions. These the coefficients in the constraint equations.
1eobjective function is elastrowin etable. 1econsintcoefficients are
written t.
R o w operations consist of ndding (subtracting) a definite multiple of the pivot
row to othEr rows ofthetable.The pivotrow identifies the row in which theunit
vcctor (in a.ccrtain column) will havc the value of 1.
R o w operations are needcd to obtain the canonical fonn. Practically this plics
you should bEable to spot thEunit vectors in the table.If this happens
then the current iteration is complte and considertions for the next onc
must begin.
3.3 NUMERICAL SOLUTlON -THE SIMPLEX METHOD 117
With this in mind the Simplex method is applied to the machine selection problem.
Thc example will use and cJarify the items in the above Iist .
3.3.2 Appllcatlon of Slmplex Method
Simplex Table 3.1: In Table 3.1 (Simplex Tnble 1)c first row indicatcs the
variable names. The last row is e objective function. Spreadsheets are an efficient
way to pro ss the infonnation. it is being cmployeden symbolf can be
removed frome last column and last row to allow numerical ca1culations (shown).
ThelnstcoluD1nis rightnndsidvalues. T he rest of thc cntries the coefficients
ofthe consaint equations.
The current itcration is over if the table displays the canonical fonn. In practice e
canonical fonn comprisspotting the m unit vcctors in the table as well making
sure the enes under the b column except for the rows representing e objective
function(s) are nonnegative P-Q). A glance at Simplex Table 3.1 indicates tE
canonical fonn is present.
Thc unit vectors in c table also idcntify those m variablcs at will belong to the
b is. Those variables will have a nonzero value for is iteration. The remaining
n-m nonbasic variables are sct to zero. This solution is directly intcrpretedomTable
3.1. The solution therefore is
x = 0.0 xz= 0.0= 8.5 x
4
= 25 X
s
= 70df=-5250
Has the Optimal Solution Been O bined? The optimnl solution requires that f
be reduced much as possible. For the current solution value of XI is O. From
Equation (3.5) if x were increased from zeroen the objective functionf decreases
further in value. Similar reasoning cnn be used for X2Thereforethe solution can be
further inproved. Solution hns not converged. The second/next iteration will quire
nssembling another table Table 3.2.
The only way for x and/or X2 to have a positive value is if y b ame a basic
variable. Since the number of bnsic variables is prescribed for the problem some of
e current bnsic variables must become nonbasic. For each iteratione Simplex
method a1lows only one pair of variablcs to conduct is basis-nonbasis exchange.
Each iteration takes place through a new table. T o affect the exchangit is necessary
Table 3.1 Simplex Table 3.1
XI X
2 XJ X4 X ~ b
0.4 0.6

8.5
3 -1

25
3 6

70
990 -9

5280
119
Simplex Table 3.2 (Table 3.2): The pivot row is obtained from origin row 20f
Table 3.1. T he following is original row
25
The value of the first element in the new row w i1l be 1. The row is modified by
dividing through by 3. T he new second row in Table 3.2 will be
8.3333
In Table 3.2e first element of the first row (Table 3.1) must be reduced to O. This
is always achieved by adding/subtractingfrom the row being modified an appropriate
multiple ofthe pivot row. The modification to e first row is
row 1 - [0.4 " (pi vot row 2)} to yield
o 0.73331 -0.1333 0 5.1667
Thefourrow w i1l be obtained by
row 4 + [(990 " (pivot row 2) I
Table 3.2 is obtained after these calculations.
The negative value of the coe:cient in Table 3.2 in row 4 indicates w e need
at least one more iterationltable. The E B V is X 2Byexamining positive ratios of
e va1ues in b column to the coefficient in X 2 column the L B V is id tified as
XsThe pivot row is row 3. Table 3.3 w i1l be constructed by transforming the rows of
Table3.2.
Simplex Table 3.3 (Table 3.3)
pivot row 3 = (row 3)
new row 1 =r ow 1 - {0.73333 * (pivotrow 3)}
new row 2 = row 2 + {0.3333 * (pivot row 3)}
new row 4 = r o w 4 + {1230 (pivot row 3)}
Table 3.3 is obtained based on the above operations.
Simplex Table 3.3 Table3.3
b
n
u
-
-
-
n
u
n
u

3.3 N U M E R I C A L S O L U T I ON -T H E SIMPLEX M E T H O D

3
O 0.3333

0.3333
The third row will be obtained by
row 3 - {3 "(pivot row 3)}
0.4524
10.4762
6.4285
21437.14
Xs
-0.047
0.0476
0.1428
175.71
X4
-0.0285
0.2857
-0.1428
154.28
X )
'

n
u
n
U

U
X 2
n
u
n
u
-
-
A
U

XI
LlN E A R P R O G R A M M I N G
to determine which is the variable will enter the basis (XI or X2) Entering Basic
Variable (EBV). Also to be established is which of e variabl (X3' X4' or xs) will
leave basisLeaving Basic Variable (LB V). In terms of the numerical calculations
in the new table the unit vector under the L B V column w ilI be transferred to the E B V
column through elementary row/column operations.
118
Entering 8asic Variable: The E B V is chosen by its ability to provide for the largest
decrlin thobjective function in e current iterationltable. lis is detercnined by
the largest negati ve coeicientin the row associated with the objective function. From
Tabl3.1this value is -990 and copondsto e variablc XIE B V is X!. A tie can
be brokcn arbitrariJy.
Leaving 8asic Variable: The L B V is also chosen by its ability lprove e
otive. T he L B V is only determined after the E B V has been chosen. is
important because only the column undere E B V isexaIned to estabIish LBV.
The minimum l'atio of the values of e right-hand side (b column values) to the
corrcsponding coefficients in the E B V column provided c coefficicnt valuc is
positivc d cidcs e LBV. This rninimum valuc idcntifies the row which is aIso the
pivot row. In this row the column which contains the unity value of some unit
vector (or a current basic variable) is thc column of c Ieaving basic variable. lis
is theLBV.
InTable3.1 eE B V has already been identified as Xl' T he ratios under this column
are 21.25 8.3333 and 23.3333. T he least value is 8.3333. lIs corr ponds to e
Scond row of Table 3.1. The unit vector under the X 4 column is in is row (row 2).
Thcrefore X 4 is e L B V d e second row is the pivot row for Table 3.2. This
impIies seond row will be utilized to chang current XI column (EB V
column) iilto a unit vector al is currently under X 4This mechanical procedure is
based on the following reasoning. le value o f E B V must be such that itwill not cause
constraint violation with the current values of e other basic feasible variablcs. The
application of the Gauss-J ordan process should shift the column under the L B V to e
E B V leaving the column of the other unit vectors (that are part of the basis) unaltred.
Table 3.2 is then constructed using thc pivot row 2.
Simplex Table 3.2 Table3.2
b
n
u
-
-
n
u
n

5.1667
8.3333
45
13530
Xs
n
u
n
U
E
n
u
X4
-0.1333
0.3333
330
.t)
-

n
u
n
u
n
u
Xl
0.7333
-0.3333
7
-1230
XI
121
next
A(: 6)/A(: 1)
ans
o 0 0 -17/1980
o 0 0 -51198
o 0 0 -7/99
0 0 0 16/3
'something not right - The above division should have';
'been a n elemenb y element one' i
3.3 NUMERICAL SOLUTION-THE SIMPLEX METHOD
h e create t o row pivo the identifies 'The row
table' ;
A ( : 6) ./A(: l)
ans =
21. 2500
8.3333
23.3333
-5.3333
'Second row is the pivot row and x4
'Constructing Table 2';
'note the scaling faco r for the matrix' i
A
A
1.0e+003 *
0.0004 0.0006
0.0030 -0.0010
0.0030 0.0060
-0.9900 -0.9000
'The element at A(2 1)
A(2 :) = A(2 :)/A(2 1)
A
1.0e+003
0.0004
0.0010
0.0030
-0.9900
A(: 6) ./A(: l)
ans
85/4
25/3
70/3
-16/3
format short
is LBV' ;
0.0085
0.0250
0.0700
5.2800
o
o
0.0010
o
0.0010 0
o 0.0010
o 0
o 0
must b e 1';
0.0085
0.003
0.0700
5.2800

0.0010
0
o
0.0003
o
o
0.0010
0
0
0
*
0.0006
-0.0003
0.0060
-0.9000
1 l
l
-

4
u
r
H
1
U
4
y
LlNEAR PROGRAMMING
Table 3.3 does not have y negative value e objective function row. A
solution has been achieved. T he basic variables have the unit vector in their column.
The nonbasic variables are set to zero. From Table 3.3 the opn u m values can b
observed :
x ;= 10.4762 xi = 6.4285. X; = 0.4524I = -21437.14
Note the valueofj" +21437.14=Ofrom which thsolutionforthe objective function
(f.) can be obtained.
120
Solutlon Uslng MATLAB
The tables in the previous computation can be generated quite effortlessly using a
spresheet . lis would be accomplished by setting up .e mathematical
manipulation in one cell and copying and pg to e other cells. Since this book
also sists in MATLAB progrnmingthe illusation here will allow opportunity to
practice the use of MATLAB for matrix manipulations. The code presented here is
ptured using MATLAB'S diary command. The command requires the name of a
file which records all communication wilh MATLAB as well its response during
interactive session (see help diary). The text file is commented liberally (for your
understanding) and using the semicolon at e end of the str.gs prevents eirecho.
h is file the comments are quodsng. The code represents details of an actual
session. T hese statements are typed aer e MATLAB prompt the Command
window. T his is therefore an interactive session. Values toeright of ans = is the echo
of result of e ex ution of e MATLAB command. Note: comments need not be
typed.
3.3.3
Example 3_l.txt
format compact
format rational
'Set u p the coefficient matrix and b as matrix A';
-1 0 1 0 25;3 6 0 0 1 70;-990 1 0 0 8.5;3
5280]
0.6
o 0
A=[O.4
-900 0
A
17/2
25
70
5280
canonical form';
o 0
1 0
o 1
o 0
note the
column' ;
2/5 3/5 1
3 1 0
3 6 0
990 -900 0
'This is Table 1 -
' E B V is x1 - first
'To find L B V divide last column b y coeff. i n E B V ' i
'column' ;
'Take the minimum of the positive values';
122 LINEAR PROGRAMMING
'The element a t A ( l l) must b e a 0 ';
A ( l :) = A(l : ) 0.4*A(2 :)
A
1.0e+003

0.0007 0.0010 -0.0001 o 0.0052

0.0010 0.0003 o 0.0003

0.0083
0.0030 0.0060

o 0.0010 0.0700
0.9900 -0.9000 o o

5.2800
'Element at A ( 3 1) must b e a zero';
A(3 :) = A(3 :) - A(3 1)*A(2 :)
A
1. Oe+003 *

0.0007 0.0010 0.0001

0.0052
0.0010 -0.0003

0.0003 o 0.0083
o 0.0070

-0.0010 0.0010 0.0450

-0.9900 -0.9000 o o o 5.2800
'Element a t A(4 1) must b e 0';
A(4 :) A(4 :) - A(4 1)*A(2 :)
A
1.0e+004 *

0.0001 0.0001 0.0000 o 0.0005

0.0001 -0.0000

0.0000 O 0.0008
o 0.0007 o -0.0001 0.0001 0.0045

0.1230

0.0330 o 1. 3530
'Table 2 complete - and canonical form is present' ;
'Solution is not converged because o f A(4 2)';
E B V is x2' ;
'Calculation o f LBV' ;
A(: 6) ./A(: 2)
ans
7.0455
25.0000
6.4286
-11.0000
'Pivot r o w is third row a n d L B V is x5'
ans
Pivot r o w is third r o w a n d L B V is x 5
'Oops forgot the semicolon';
'Calcu1ation o f LBV' ;
'Construction o f Table 3.3 - n o echo o f calculations';
'A(3 2) must have value 1';
A(3 :) = A(3 :)/A(3 2);
'A(1 2) must have a value o f 0';
A ( l :) A(l :) - A(1 2)*A(3 :);
3.3 NUMERICAL SOLUTlON -THE SIMPLEX METHOD 123
'A(2 2) must have a value o f 0';
A(2 :) = A(2 :) - A(2 2)*A(3 :);
'A(4 2) must have a va1ue o f 0';
A(4 :) A(4 :) - A(4 2)*A(3 :);
A
A '"
1.0e+004 *
o 0
0.0001 0
o 0.0001
o 0
format rational
A
n
u
n
U
F
O

n
u

n
U
2
d
n
u
n
u
n
u
n
H
E

n
u
n
u
n
U

-
-
-
-
n
u
n
u
n
u

4
n
u
n
u
n
u
p

n
u
n
u
n
u

4
n
u
n
u
n
u
n
u

n
u
n
u
n
u
n
u

n
u
n
u
n
U

n
u
n
u
n
u
E
J

n
u
n
u
n
U

n
u
n
u
n
u
n
u

-
-
-
-
n
u
n
u
n
u
n
u

u
n
u
n
u
n
u

n
u

n
u

n
u

A

R
d
r
f

o
'
'
'

'
'
'
a
u

-
a
n
u
-
-

4
n
u
n
u
n
u
n
u
n
U

4
n
u
n
u

4
n
u
n
u
-11/105 19/42
1/21 220/21
1/7 45/7
1230/7 150060/7
'No further iterations necessary';
diary off
T he last few commands suggsl Ihc possibility of developing a for loop for the row
manipulations. It is suggested as a problem at the end of e chapter.
3.3.4 Solutlon Using MATLA8'S OptimizaonToolbox
This section is useful for thosc that have access 10 the Optimization Toolbox . from
M A'1.AB (Ver 5.2).
Start M ATI..AB. In the Command window type help lp. This will provide information
on the use of the linear programming routine 10 solve the problem. Here. the standard
use is employed. It involves specifying va1ues for the cosl coefficients (vector). the
coefficientlconstrainl matrix (A-max). and the right-hand side vector (bvector).
be following sequence of steps illusates e use of the program.
Using the Linear Programming Routine
>> f 990; 900] ;
> > A [0.4 0.6 3 -1; 3 6)
> > b [8.5 25 70)'
>> x '" 1p(f A b)
The solution as posled in the Command window:
x
10.4762
6.4286
>> sol ' " f'*x
124 LlNEAR PROGRAMMING
8 0 1
1.6157e+004
T o this solution must be added constant -5280 which was omitted in problem
definition for MATLAB. There armany different ways to use linear programming
function in MATLAB.
3.4 A D D I T I O N A L E X A M P L E S
h is section additional examples e p sented. T hese examples illustrate e
extension/modification of the Simplex method to handle greater or equal
constraints negative values for the design variables equalily constraints and so on.
In alI e cases the problem is transformed appropriately and e same Simplex
melhod is then applied.
3.4.1 Example 3 . 2 -Transportation Problem
The Fresh M iIk cperative supp1 ies miIk in gallon jugs from its two warehouses
located in Buffalo (Ne w York) and W ilIiamsport (Pennsylvania). 1t h a capacity of
2000 gallons per day at Buffalo and 1600 gaIlons per day at W i11iamsport. It delivers
800 gallons/dayRochesr (New York). Syracuse (New York) ruires 1440
gallonslday and e remainder (1360 ga1lons) are trucked to N e w York City. le cost
to ship lhe milk to each of lhe destinations is different and is given in Table 3.4.
Estab1 ish e shipping stegy for minimum st.
Problem Formulat/on: Let x be e number of gaIlons shipped from Buffalo to
Rochester. Let y be e number of gallons shipped m Buffalo to Syracuse.
The warehousing cons int at BuffaIo is 2o gallons/day. Therefore
x + y $ 2000
Amount shipped o m Williamsport to R hes = 8 o o - x
A mount shipped o m Williamsport to Syracuse = 1440 - y
Amount shipped from Williamsport to N e w York City is
1600 - (800 - x) (1O-y) o
Table 3.4 Shipping Cost (Cen perGallon)
Buffa10
W iIlinsport
Rochester
4.2
4.7
Syracuse
4.5
4.6
(3.21)
New York City
6.0
5.1
3.4 AODITIONAL EXAMPLES 125
x + y ~ 6 4 O (3.22)
The side constraints on x and y are threspective warehouse I imits. The shipping cosl
IS
Cost=4.2 x + 4.5 * y + 6.0 * (2000 - x - y) +
4.7 (8oo-x) +4.( 1 0 -y) + 5.1 (x + y - 640)
Cost= 1.4'" x -y + 19120
Assemb1ing the problem (Example 3.2)
M bze f(xy): 1.4 x - y + 19120
Subject81(X y): x + y :5 2000
82(X y): x + y 640
0 $ x :5 8; 0 :5y :5 1440
(3.23)
(3.24)
(3.25)
(3.26)
(3.27)
Figure 3.7 iIIustrates the graphical solution to Example 3.2. The line representing the
constant objective function values lerm does not include the constant term (19120).
The solution is at the interseclion of constrint gl and the upper 1 imit on e value of
xas e objective is to decreasefas much as possible. From the figure the solution is
x=800;y= 1200f = 168ooor$168.oo (3.28)
Two-Phase S/mplex Method: The main difference belween Examples 3.1 and
3.2 is e consaint in the Iatter. Since the standard L P problem only requires the
design variables be semipositive 0)e right-hand constraints on x and y have to
be accommodated through additionaI inequaIity constraints. Applying the regular
Simplex method without additionaI processing would cr te a problem in recognizing
e c oni 1 form. Example 3.2 expressed in e standard format of linear
programmmg IS
Minimize ft.xy): 1.4x - y + 19120
Subject to: gl(X y): x + y + SI = 20
g2(X y): x + y - S2 = 640
g3(X y): x + S ) = 800
84(x y): Y + S4 = 1440
(3.24)
(3.29)
(3.30)
(3.31)
(3.32)
126 LlNEAR PROGRAMMING
25

g
o
=

H
h
h

2
2
~ 500
E

D
-

-
-

h

-

-
h

-5
-
-1000
800
-1500
0 200 300 400 500 600
x:g10 110mBfoloRh 101
100 700
Flgure 3.7 Graphlcal 80lullon Exampla 3.2.
x ~ 0; y ~ 0; sl' s2' s3' s4 ~ 0 (3.33)
HereslS3' s4 are the slack variables and S2 is surpl . (similarto a slack-used for
~ constraints) variable. In Example 3. 1 when the f1rst Simplex table was set up
(Simplex Table 3.1) the slack variables were helpful in identifying the canonical
form. Here in Example 3.2 it do not workout tway because the coefflcient of S2
is -1. Multiplying Equation (3.30) by -1 migrates the negative sign to the right-hand
side. which is also disaJlowed as f : as e recognition of e canonical form is
concerned. This means additional problem-specific preprocessing must take place to
identify the initial canonical form. Since the Simplex m eod is used to handle large
problems with several variables it is more convenient to apply the Simplex procedu
in a consistent way. This is accomplished by applying the same Simplex technique in
a two-part sequence.
le f1rst p is recogtzed as Phase 1. Here a new kind of variable. an artificial
variable is defined for each surplus variable in the same equation. Also a new
objective function called an artificial cost functioll or artificial objective function
is introduced. The artiflcial objective is defined a sum of aJl the artificial variables
in the problem. In Phase 1eartiflcial objecti ve is duced to zero using the standard
Simplcx procedure. W h e n this is accomplished Phase 1 is complete. Ife artificiaJ
i40
3.4 ADDITIONAL EXAMPLES 127
objective depends only on the artificiaJ variables. d if its vaJue is zerois implies
that artificiaJ variables are lIonbasic variables. T his also suggests that these
variables werc basic variables at the start of the procedurc. W h e n Phase 1 is completed
en both the artificiaJ objctive function and the artiflcial variables are discarded m
table and Phase 1/ begins.
Phase II is the standard Simplex technique applied to the table from the end of
Phase 1 neglecting all artificial elements. T he table should be in canonical form.
Additional tables are obtained as necessary until the solution is reached.
In Example 3.2 set up earlier there will be one artificiaJ variable a 1 and an artificiaJ
cost function A" In the example only Equation (3.30) will be affected as
g2(Xy): x + y-s2 + a
l
=640 (3.34)
The artificiaJ cost function will be
A f = a
l
(3.35)
Table 3.5 rep sents f1fSt table in Phase I. T he f1rst four rows rep sent E
constraints. The f1fth row psentseoriginal obje function. T he last row is e
artificiaJ objective function. This is row uSed to iv iterations in Phase 1.
Table 3.5 is not in canonicaJ form. but replacing the sixth row by the result of
subtracting e second row from the six row will provide a canonical form. Table
3.6 illustrates the result of such a row manipulation.
SlmplexMeo dPhBse 1: In Table 3.6 there are two choices (x y) available for
e E B V (enteIg basic variable) as both of them have a coefficient of -1 in e last
row. While the choice can be arbitrary x is a good choice because it has a larger
negative coefflcient in the originaJ objective function. le L B V (leaving basic
variable) is identified through the minimum positive vaJue of the ratio of the values in
the b column to e values under xcolumn. lesc ratios are 2000/164011 and
80011. The selection identifies e second row as pivot row and a 1 e L B V .
Using the second r o w e pivot row the unit vector [0 1 0 0 0 O]T h t o be
Table 3.5 Example 3.2: Initial Table Phase 1
x y SI S2 01 S3 S4 b
o o o

2000

640
l

o 800

1440
1.4

o

f- 19120

o

A
f
128 LlNEAR PROGRAMMING
Table 3.6 Example 3.2: Slmplex Table 1 Phnse (Canonl IForm)
x $1 $2 b al $ ) $4 y
O
-1.4
1
l
u
l
l
-
-

-
s
h
u
n
u
n
u
h
u
n
u

o f - 19120
o A
f
-640
n
u
h
u
n
u
'
2000
640
800
1440
A
U
'
a
n
u

u
n
u
n
u
h
u
n
u
-
a
n
u
n
u
h
u

lsfeed from the al column to the x column. Construction ofTable 3.7 stts with
the second row in Table 3.6 which is also the second row ofTable 3.7 because ofthe
coeicient of 1 in the second row under the x column (no scaling is necessary). T his
is the pivot row. To obtain a 0 in the first row in the x column the pivot row is
subtrcted from rst row in Table 3.6. The new first row is

-1 0 0 1360
To obtain a 0 in the x column in the third row. the pivot row is subtracted from the
third row ofTable 3.6. The fourth row has a 0 in place and hencc is copied 'OmTable
3.6. T hc fifth row is obtaincd by adding a 1.4 multiple of the pivot row to the fit1row
in Table 3.6. The last row is the addition of the pivot row to the last row from Table
3.6. Table 3.7 is compiled as follows:
Table 3.7 Example 3.2: Slmple Table2Ph 1(Canonlcol Form)
x $2 b $ ) $1 al $4 y
n
u
'
A
U
n
u
n
u
n
u

u
n
u
'
a
n
u
n
u
n
u
o f-18224
o A
f
n
u
n
u
h
u
'
a

1360
640
160
1440
a
a

0
1

1
0
0
'
a
n
u
n
u
n
u
n
u
n
u

O
-1.4

'
a
'
E
'
a
n
u
-
a

-
-
From Table 3.7. e value of Af is O. and al is not a basic variableThis identifies
the end of Phase 1. The a 1 column and last row are discarded and Phase 11 is started
with Table 3.8.
b
V
L
F
i
j
i
E

-
E

3.4 ADDITIONAL EXAMPLES 129

Table 3.8 Exumple 3.2: mplex Table 1 Phase s (CanoniI Form)
x
$4
b y $2 $) $

u
'
a

U
n
u
n
U
n
u
n
u
n
u
'
a

1360
0
160
1440
0
0.4
-
E
A
U
A
u
n
u
n
u

n
u
n

'
a
n
u
n
u

o
-1.4 o f-18224
Simplex Method Phase 11: The column with the gr test negative coefficient
(1.4) is s2lis is eEBV. The L B V is s3le ird row is therefore the pivot row.
The unit vector [0 0 1 0 O]T must be ansferred from the S3 column to S2 column
through row manipulations using the pivot row. sexercise results in Table 3.9.
The negative coefficient under the y column suggests we are not fmished yet. For
the next iteration E B V is y. L B V is SI' The first row is epivot row. T he unit vector
[1 0 0 0 0 ]T must be constructed under the y column using row manipulation using
the pivot row. T his results in Table 3.10.
There are no negative coefficients in the last row. The solution has been rellched.
From inspection of Table 3.10 the solution is
x = 800 Y = 1200. S2 = 1360. S4 = 240f= 16800 (3.36)
In this exnplethe same Simplex method was repeatedly applied although the
problem was solved using o phases. T he first phase was a preprocessing phe
move the surplus variables away from initial set of basic variables. lis was
achieved by introducing additionnl variables for each sulus variable and an
additional cost function that drove e iterations of the frrtphase.
To summarize for ~ and = constraints artificial variables and artificial cost
function e indu d into the problem for convenience. le Simplex method is
applied in two phases. le first phase is terminated when the artificial variables clln
be eliminated from problem.
Table 3.9 Example 3.2: Slmplex Table 2 Phasn (Canonlcal Form)
x $ ) $4 b y $
$2

A
U
A
U
n
u
0
1.4
o 12
o 8
o 160
1440
o f-180

u
n
u
'
a
n
u
n
u
'
a
n
u
s
a
E
E
a
-
-
'
E
n
u
n
u
n
u
n
u

3 A
3
130 UNEAR PROGRAMMING
Table 3.10 Example 3.2: Slmplex Table 3 (Fioal) Phasc D
x y SI S 2 S ) S4 b

12

1

800

o

1360

240

0.4

f-16800
3.4.2 Example 3-Equallty Const ints a n d Unrestricted Varlables
Example 3.3 w il1 inc1ude an equality constraint and iJlustrates the technique for
handling variables are unresicted in sign (can have negative values). Variables
like profit temperature and net income can be negative quite ofl 1. The uality
cons int is handled by in ducing ificial variable and applying etwo-phase
Simplex t liqu illus ted in Example 3.2. W hen this correspondicial
variable is zero ( nd of Phase 1)constraint has been m et. The negative value for
a variable is simulated through e difflrence between two positive variables. Once
again these changspreserve the Simplex method in its original form. T he following
is an imaginary problem illustrate additional features so some L iberty is taken in the
formulation.
The Problem: Today a full-tirne student on campus is always driven to maximize
grades. A strong influence for a favorable outcome is e amount of investmade
in hittingbooks versus e time spent playing pinball. In a typical day at least one
hour is definitely extended to the pursuit of learning or pressing the sides of the
coin-operated machine. Not more than five hours is availabl for such dispersion.
Over e ye sa fatigue threshold of four units. based on a combination of two
activities has been established as a norm for acceptable perform ce. This
combination is the sum of the hours spent at the pinball machine and twicthehours
spent on homework an acknowledgment that hitting ebooks is stressful. A negative
hour of pinball playing w i1l go to increase the time spent studying which will
contribute positively to the grades. The overall grade is determined as a linear
combination of twice the time spent academically and subtracting e time spent
playing. The problem erefore is to distribute the time to maximize the grades
obtained.
Problem Formulation: There are two original design variables in the problem. XI
is the number of hours spent studying and X 2 is the time spent enjoying the game of
pinball when thsame time can be advantageously spent maintaining high grades.'he
obj tive function can be expressed as
3.4 ADOITIONAl EXAMPLES 131
Maximize j{x
l
X
2
): 2x
1
The constraints can be recognized from the statement ofe problem directly as
g.:
g2: 2x. + x
2
= 4
g3: x
1
1
x
1
0 x
2
is unrestricted in sign
Figure 3.8 using the drawLine.m m-file is a graphical description of the problem and
the solution. 1e solution is atxl = 3 and X 2 = -2 in figurewhere conveniently e
objective function is also passing through.
Standard Format: A s the standard format expe only nonnegative variables. the
variable x 2 is replaced by a pair of variables in the formulation:
6
1
0

0
o
a
Z ; !
4 5 6
Flgure 3.8 Graphlcal solution Example 3.3.
132 LlNEAR PROGRAMMING
x
2
= x
21
- X
22
(3.37)
The standard fomlat after converting to a mininuzation problem introducing slack
surplusd artificial variables and including Equation (3.37) results in
3.4 ADDITIONAL EXAMPLES 133
Phase " Table 2: Table 3.13 is 1he second table under Phase 1 . The vaJue of the
artificial function is 2 so Phase 1 is not yet over. The E B V is X22TheLBVisalThe
second row is 1he pivot row. This row w iII be used for row manipulations yield the
next table.
Minrize f -2x1 + x 2 1 - X 2 2
(3.38)
Table 3.13 Example 3.3: Tablc 2 Phasc I

Sujectto: gl:X1 + X 2 1 - X 2 2 +81 = 5

X22 $) U) 8
2 a2 b X) X2)
gl: 2x
1
+ X 2 1 - X
+ a t = 4

4

g3: X1 + X21 - X22 - 82 + a 2 = 1

2 -2 2

l
The artificiaJ function to be driven to zero is

3 -3

-2 2 /+2
(3.39)

-2 3 Ar- 2
A f = a l + ~
Simplex Table 0: Table 3.11 p1 hes dard fOm 1 8t in a table. It is caIIed Table
o to aIIo w for preprocessing wi1h respectthe artificia1 variables so that a canonica1 fOlID
nbeob rved. Addingesecond d irdrow dsubtracgeresulto m elast
row and replacing the last row wi1h this computation w il1 yield the next table.
Phase " Table 3: Table 3.14 is the new table after the row operations. The value of
artificiaJ funcon is 0 as bo1h a l and a 2 e nonbasic variables with a vaJue of 0
for is iteration. lIssigfies eend of Phase 1. Phase n will start with the last row
( artificia1nction) and the two columns that represent the articial variables removed
from the current table.
Tablc 3.11 Examplc 3.3: Table 0 Phasc I
Tblc3.14 Exmplc 3.3: Table 3 Phase I10111)
8) a) S2 a2 b
8
2 oz
b
X) X2) X22
X) X2) X22 $) 1
1

5
l o 4

4

2 I

2 2 2 o

-1 1

3
1

o
-2 I

f

3 4 4 /+8

Ar

l

1 Ar
Phase 1 Table 1: Table 3.12 provides e start of e Simplex method. In Phase 1
thmotivation is to drive the artificial function to 0 which has a value of 5. T he E B V
iSXlTheL B V is a2' T he third row is the pivot row. R o w manipulations with the pivot
row lead to Table 3.13.
Tllblc 3.12 Examplc 3.3: Table 1 Phscl
X X2' X22
$ a) $2 a2 b
1

4
o

-1

f
2

Ar- 5
'
A
q

1
2
1
2
3

-
-
Phase 11 Table 1: Table 3.15 represents etablto start the itemtions for Phase 11.
However there are no negative coefficients in last row (row corresponding to the
objective function). Scanning the le fuher1he canonical form can be observed.
lereforethe solution has been obtained. le solution is
Tablc 3.15 Examplc 3.3: Table 1 Phasc 11
x X z
$
82
b

4

2 2
o o

3

4 /+8
134 LJNEAR PROGRAMMING
x
l
= 3;x
21
=0;x
22
= 2; sl =4; s2 =0;/=-8
The value of X 2 is
X
2
= (X
21
- x
22
) = (0- 2) =-2
which was identified earlier graphically.
le above example illustrated the Simplex method for handling negative variables
and equality constraints. Not application of the Simplex technique itself did not
change. An interesting feature of e probJem was that Phase II was immediatly
obtained aftel' Phase 1 without any further iteration which suggests the search for
the optimum is taking place in Phas1 even though e focus is on the artificial
variables.
3.4.3 Example 3.4 -A Four-Varlable Problem
Example 3.4 presents a problem with four variables. T he procedure is identical to the
one in Example 3.1 except at ere is no graphical solution to the problem.
The Problem: The RIT student-run microelectronic fabrication facility is taking
orders for four indigenously developed ASIC chips at can be us in (1) touch
sensors (2) L C D (3) pressure sensors and (4) controllers. T here are several
constraints on the production based on space equipment availability student hours
and the fact at e primary mission of e facility is student training. Firste
handling time cons int outside of processing for all chips is 600 hours. Touch
sensors require 4 ho sL C D 9 hours press 'e sensors 7 hours and conollers 10
hours. Second the time available on the lithographic machines is about 420 hours.
Touch sensors and L C D ruire 1 hour pressure sensors 3 hours and controllers 8
hours. Packaging considerations place the maximum at 800 volume units. Touch
sensors require 30 voJurne units L C D 40 volume units press' c sensors 20 units and
conoller 10 units because of eir compact size. All e consaints above are
indi ted per week of operation of the facility. enet revenue is $6 $10 $9d$20
for the touch sensor L C D p ssure sensor and controller respectively. le facility
is interested in maximizing revenue per week and would ike to determine the right
mixof four devices.
Problem Formulation: The formulation is straightforward based on the
statements above. Let x ( rep sent the number of touch sensor chips per w kX 2 E
number of L C D x 3 the number of pressure sensors and X 4 enumber of conollers.
objective function is
Maximize: f. 6x. + lOx
2
+ 9x
3
+ 20x
4
The handling time constraint can be expressed
3.4 ADDITIONAL EXAMPLES 135
gl: 4x
I
+ 9x
2
+ 7x
3
+ IOx
4
S 600
The availabi1ity of the lithographic machine can be developed as
g2: x
l
+ x
2
+ 3x
3
+ 8x
4
s; 420
Je packaging constraint is
g3: 30x
I
+ 40.+ 20x
3
+ IOx
4
S 800
All design variables arp ted to be g ater zero. As formulated abovee
problem suggests a degree of incompleteness.nere arefour design variables and only
three constraints. le number of variables in the basis can only be e. Hence at
least one of e variables must have a value of zero. Sevral useful additional
constraints can stiU be included to define a valid optimization problem with a nonzero
solution. lis is now a modeling issue. For problems with a limited number of design
variables paying attention 10 the problem development allows anticipation of the
solution as well the opportunity to oubleshoot decisions from practical
considerations. It is left to estudent to explo this problem further.
Standard Format: The objectiv function requires a minimum formulation.
Henc
Minimize f. -6x
1
- 1 O ~ - 9x
3
- 20x
4
The symbol / has been retained for convenience even though the dIIction of
optimization has changed. le cons ints are set up using slack variables SIS2ands
3
g.: 4x
I
+ 9x
2
+ 7x
3
+ 10x
4
+ S. = 600
g2: x. + ~ + 3x
3
+ 8x
4
+ s2 = 420
g3: 30x. + 40x
2
+ 20x
3
+ lOx
4
+ S3 = 800
All variables are ~ O. le formulation above suggests the standard Simplex method.
S 'P1ex Table 1: Table3.16is eini tablefor Example 3.4. T he canonicaI form
is observable from the table and the initial basic feasible solution can be determined.
Table 3.16 Examplc 3.4: Simplex Table 1
XI X 2 XJ X
4 81 J"2 SJ
b
4 9 7 10

6
3 8

420
30 40 20 10

800
-6 -10 -9 -20

f
136 LlNEAR PROGRAMMING
Table 3.17 Example 3.4: Simplex Table 2
x X2 X3 X4 s 82 8J b
2.75 7.75 3.25

l -1.25

75
0.125 0.125 0.375

0.125

52.5
28.75 38.75 16.25

1.25 275
-3.5 -7.5 -1.5

2.5

f + 1050
SIS2. and $3 are the basic variables. The E B V is X 4 and e L B V is $2 which is E
minlum of {60. 52.5 80}. The pivot row is the second row used for the row
manipulations to lead to Table 3.17.
Simplex Table 2: Using the pivot row identified in e last tablee unil vector [0
o 1 O]T under the $2 column needs to be transferred to e X 4 column through
elementary row operations. Table 3.17 shows e canonical form after completion of
all the operations. The basic variables are X4. $ " d S
3T he E B V is X 2 and e L B V
is S3le pivot row is the ird row. The objective of the row manipulations is to
transfer the unit vector [0 0 1 O]T from the $3 column to exzcolumn.
Simplex Table 3: Table 3.18 denotes the reduced table withe canonicaJ form after
the required row operations are completed. The basis variable X 2 . x 4and SIThere
m no negative coefficients in the last row suggesting the solution has been obtained.
From e table solution is
X
J
= 0 x
2
= 7.096 x 3 = 0 x
4
= 51.61/= 1103.22
The solution above is not satisfactory because the actual decision w iIl involve integer
values for the design variables. At this time integer programming is not an option. The
adjusted solution is
X
1
= Ox
2
= 7x
3
= OX 4 = 52/= 1110
While is choice satisfies e constraints g3. g2 however needs a Iittle elasticity (limit
is 423) to be satisfied. Notat sJ is a basic variable so constraint gJ should not be a
problem.
Table 3.18 Example 3.4: Simplcx Tnble 3
x X2 XJ X4
$
82
$) b
-3

-0.2 20
0.032258 0 0.322581 o 0.1290322 -0.00323 51.6129
0.741935 1 0.419355

-0.032258 0.025806 7.096774

2.064516 0 1.645161

2.25806451 0.193548 f + 1103.22

3.4 ADDITIONAL EXAMPLES 137
Using MATLAB
This is a standard L P problem. This section uses MATLAB in an interactive session to
soIve e problem. le following is a diary file for the MATLAB session for is
exnple. Note the waming from M A TLAB. as w eI l e subsequent use of the L P
program using bounds on e design variables: le bold italicized statements are
commands used to obtain the solution. T he MATLAB prompt is edited out .
help lp
L P Linear programming.
X=LP(f A b) solves the linear programming problem:
m i n f 'x subject to: A x <= b
X=LP(f A b V L B V U B ) defines a set o f lower a n d upper
bounds o n the design variables X s o that the solution
is always i n the range V L B <= X <= V U B .
X=LP(f A b V L BV U B XO) sets the initial starting point
t o XO.
X=LP(f A b V L BV U B X O N ) indicates that the first N
constraints defined b y A a n d b are equality constraints.
X=LP(f A b V L BV U B X O N DISPLAY) controls the level of
warning messages displayed. Warning messages c a n b e
turned off with DISPLAY -1.
[xLAMBDAJ= LP(f A b) returns the s eof Lagrangian
multipliers
LAMBDA at the solution.
[XLAMBDAHOW] LP(f A b) also returns a string h o w
that indicates error conditions at the final iteration.
L P produces warning messages when the solution is either
unbounded o r infeasible.
c = [-6 -10 - 9 -20];
A = [4 9 7 10; 1 1 3 8; 30 40 20 10];
b = [600 420 800];
x = [c A b]
x = lp(c A b)
Warning: T h e solution is unbounded a n d a t infinity;
the constraints are n o t restricive enough.
x
1.0e+015 *
-0.0000
3.7067
8.8391
2.8513
138 LlNEAR PROGRAMMING
v l b = [0 0
v u b = [100
x = lp(c A
x
0.0000
7.0968
o
51.6129
0] ;
100 100];
b v l b vub)
This is thc solution obtained by working through the Simplex method.
3.5 A D D I T I O N A L T O P I C S IN LlN E A R P R O G R A M M I N G
T his section looks briefly at additionaJ ideas associated wi the L P problem. First
ere is a discussion of the dual problem associated with an L P problem. Duality is
impoantin discussion of sensitivity analysis-varion in optimization solution
due to variation in tfte origil 1 parameters of tlte problem. Following is is a
discussion of several pieces of informon that can be ex cted from e finaJ table
of the Simplex method.
3.5.1 Prlmal a n d Dual Problem
Associated wi every L P problem which will be referred t o a primal problem
there is a corresponding di 1 problem. In m y wayse duaJ pl'Oblem is a
sposition of the primaJ probJem. lIsis meant to impJy that if the primal probJem
is a minimization probJem e duaJ problem will be a maximization one. lf the primaJ
problem has 11 variables the dual problem will have n constraints. lf the primal
problem has m constraints the duaJ problem will have m variables. For the discussion
of dualitye standard primCll problem is generally defined as a maximization
problem with inualities (though e standard problem in the L P programming
discussion was a minimization problem [sE quations (3.10)-(3.12)] d e
ns Ints were ualities). The srd primaJ problem in this section is defined as [3]
Maximize z: CIXI + CIX 2 + . . . + cx (3.40)
Subject 10: allxl + al2x2 + . . . + al.xn b (3.41)
a21xI + + . . . + az"xn b2
a" IXI + anaX2 + + an"xn
Oj = 12.....n (3.42)

: ;

3.5 ADDITIONAL TOPICS IN LlNEAR PROGRAMMING 139

This is also referred 10 a normal maximum problem.
le dual of the above problem is defined as
Minimize w: blYI + blY2 + . . . + b"J'rn (3.43)
Subject 10: allYI + a21Y2 +. . . + arnlYm cJ (3.44)
al2Y1 + a22Y2 + . . . + am2Ym c2
alJ'1 ... a2"Y2 + . . . + atuJ'"C"
i = 1 2... m (3.45)
Relations (3.43)(3.45) describe a normal minimum problem. There is also an inverse
relationship between definition of the two problems above as far as
identification of the primal and dual problem. If latter is considered a primal
problemen the former is e corresponding dual problem.
le following observations c be made wispect 10 the pair of problems
established above:
1. The number of dual variables is same e number of primal constraints.
2. The number of dual constraints is thc same as the number of primal variables.
3. The coefficient matrix A of the primal problem is transposed to provide E
coefficient matrix of the dual problem.
4. Theinualities ' C reversed in direction.
5. The maximization probJem of the primaJ problem becomes a minimization
probJem in thdual problem.
6. T he cost coefficients of thc primal problem become the right-hand side vaJues
of the duaJ pl'Oblem. The right-hand side vaJues of the primal become the cost
coefficients of the dual problem.
7. T he primal and duaJ variables both safy the nonnegativity condition.
Example 3. 5 le following example adapted and modified from Reference 3 is
used to iIIustrate the primal/dual versions ofthe same problem. The problem is solved
using techniques developed in this chapter.
e Problem: The local college's School of A merican Craftsman has decided 10
participate seriously in a 10caJ charity event by manufacturing special commemorative
desks tables and chairs. Each 1ype of fumiture requi s lumber as w eIl as two types
of skilled labor: fmishing d c'Pentry. Table 3.19 documents the resources needed.
The 10caJ lumber cooperative has donated 48 board feet of lumbr. Faculty and staff
have voluntd 20 finishing hours and 8 carpentry hours. The desk will be sold for
140 llNEAR PROGAAMM1NG
Tnblc 3.19 Example 3.5: Rources
Resources D esk Tabl Chair
Lumber(bo d feet} 8 6
Finishing hours (hours) 4 2 1.5
Carpentry (hours) 2 1.5 0.5
$60e table for $30 and chair for $20. The school does not expect to seJ1 more
ve tables. The school would like to maximizrevnue coJ1ected.
8tandard Forma Primal Problem: Defining Xl as the number of desks
produced X 2 as the number of tables manufactured and X 3 as the number of chairs
manufactured the primal problem can be expressed in standard format
Maximize z: 6 0 x + 3 0 X 2 + 2 0 X 3
Suect to: g : 8x + 6x
2
+ X 3 :::; 48
g2: 4 x + 2x
2
+ 1.5 X 3 :::; 20
g3: 2x + 1.5 x 2 + 0 . 5 X 3 ::; 8
1!4: X2::; 5
X l> X 2 x 3 ~ O
(3.46)
(3.47a)
(3.47b)
(3.47c)
(3.47d)
The solution to the primaJ problem is avai1able in Tables 3.20-3.22. Table 3.20 is th
initiaJ table. The E B V is x and the L B V is $3The pivot row is the ird row. The
subsequent row operations determine Table 3.21. In this tableeEBVisx3' theLBV
is szzmd the pivot row is the SEcond Eow.Table 322is the anal table establishing E
optimal values for the variables. xte optimaJ number of desks equals 2. ~ e
Table 3.20 Exnmple 3.5: Primal Problcm Simplex Table 1
x X2 X) $ $2 S ) $4 b
B 6

o 48
4 2 1.5

o 20
2 1.5 0.5

8

5
-60 -30 -20

f + O
l l i l 1

d d d f
S
l 1
E
3.5 ADD1T10NAl TOP1CS IN UNEAA PAOGAAMM1NG 141
Tablc 3.21 Examplc 3.5: Prbnal Problcm Simplcx Tablc 2
X X 2 X 3
$ $2 $3 $4 b

1

4 o 16

0.5

-2

4
0.75 0.25

0.5

4

5
o 15 -5

30

f+240
optimal number of chairs to be made is 8. N o tables will be made. le totaJ revenue
is$280.00.
81 . ndard Format Dual Problem: While the duaJ problem can bxpressed d
setup m h ica1lyit is cult to associate e design variables in a direct m ner
aswasdone the primaJ problem. Using y" Y 2 Y 3 and Y 4 as the design variables the
problem can be formulated as in Equations (3.48) and (3.49) below using the
definition at e beginning of this section. Some associations are possible in light of
transposition ofe cost coefficients and the constraints. For example the variable y
can be associated with the lumber constraint because in the dual problem its cost
coefficient is the lumber constraint of e primal problem. In a similar manner Y 2 is
associated with the finishing hours Y 3 with carpentry hours and Y 4 wilh the speciaJ
table constraint.objective function of the dual problem is expressed in Equation
(3.48). In view of e form of the expression of the objective function wE is a
possibility that e desi V iabl in the duaJ problem can have econornic
implication as fol1ows:
y unit price for a board foot of lumber
Y2 unit price for a finishing hour
Y3 unit price for an hour of carpntry
y4 iscannotbe e unit price as e desk has a price defmed in the primaJ
problem-maybe a speciaJ price for the desk resource
Table 3.22 Example 3.5: Primal Problem Simplex Table 3
X X 2 X3
$ $2 .1'3 $4 b

-2

2 -8 o 24
o -2

2 4

8
1.25

-0.5 1.5

2
o o o

1 5

5 o o 10 10 o f+280
142 LlNEAR PROGRAMMING
The dual can be expressed as
Minimize w: 48YI + 20Y2 + 8Y3 + 5Y4 (3.48)
Subject to: h
l
: 8YI + 4Y2 + 2Y3 + OY4 60 (3.49a)
h
2
: 6YI + 2Y2 + 1.5Y3 + lY4 ~ 30 (3.49b)
h
3
: lYI + 1.5Y2 + 0.5Y3 + OY4 ~ 20 (3.49c)
YI> Y2 Y3 ~ 0 (3.50)
lntroducing surplus variables (SI> S2 S3) and artificial variabls (al 2' a3) e dual
problem is expressed as a standard LP problem:
Minirnize w: 48YI + 20Y2 + 8Y3 + 5Y4 (3.48)
Subjectto: h
l
: 8Yl + 4Y2+ 2Y3 - SI + al = 60 (3.51a)
h2: 6)'1 + 2Y2 + 1.5Y3 + lY4 - S2 + a2 = 30 (3.5Ib)
h
3
: lYl + 1.5Y2 + 0.5Y3 + OY4 - S3 + a3 = 20 (3.51c)
[n Phase 1 of the two-phase approache artificial variables are removed from E
basis using an arcial cost function A defined as
MinimizeA: a
l
+ a
2
+ a
3
(3.52)
The Phase [ computations are displayed in Tables 3.23-3.26. Table 3.23 is identified
as thsimplex Table 1 and it inc1udes some preproccssing by which a" a2 and a3 e
made e basis variables. As a reminder the last row presents e arti ficial st
function. From inspection ofTable 3.23 YI is the EBV. The L B V is a2 and pivot
row is the second row. Following the standard row operations Table 3.24 is obtained.
[n this table the E B V is Y2 (e largest negative coefficient).le L B V isa3 (E ird
Table 3.23 Example 3.5: Dual Problem Phose 1 Table 1
Y1 Y2 Y3 Y4 51 52 S )
l
Q 2 a ) b
8 4 2

60
6 2 1.5

30
1.5 0.5

20
48 20 8 5

w
-15 7.5 4

o A - 110
3.5 AOOITIONAL TOPICS IN LlNEAR PROGRAMMING 143
Table 3.24 Example 3.5: Dual Problem Pbase 1 Table 2
Y1 Y2 Y) Y4 51 52 S) a1 a2 a) b

1.3333 0 -1.3331 1.3333

-1.3333

20
0.3333 0.25 0.1666 o -0.1666 o

0.1666

5
o 1.1666 0.25 -0.1667

0.1666 -1 o -0.1666 15

4 4 -3

o -8 o w - 2
o -2.5 -0.25 1.5 1.5

2.5

A - 3 5
row contains the rninimum of ([20/1.333] [510.3333] [15/1.6667]}). le pivot row is
row 3. Table 3.25 rep sents thnext table. leEBViss
2
Notein this cthere are two
candidates for E B V (S2S3)' S2 is chosen arbitrarily. le L B V is al' The pivot row is E
trow. After the [uired row manipulations Table 3.26 is generated. Scanning the last
row the are only positive coefficients. Phase 1 is now complete. The artificial variables
and tificial cost function can be removed from the LP problem.
Elirninatingeartificial variables and the artificial cost function from the problem
Phase II is started with Table 3.27. Notthat Table 3.27 contains no information that
is not already available in Table 3.26. It only provides an unc1uttered table for
continuing th application of the Simplex lechnique and therefore is not really
necessary. In Table 3.27 the E B V is S3The L B V is Yl after a toss between Yl and sl'
The pivot row is the second row. This row is used to obtain Table 3.28. In this table
note that the value of the basic variable SI is O. This is called a degenerate basic
variable. T he E B V for the next table is Y3The LBViss
3
The pivot row is es ond
row (note: the pivot row can be e row with the degenerate basic solution if the
column of the E B V has a positive coefficient in at row). Table 3.29 is efinal table
and contains the optimal solution. The final solution of the dual problem is
Y; = 0= l O y; = 10= 0s;=5 w=280
To summarize the solution of the primal problem is
x; = 2 xi = 0 x; = 8 s;p = 24p=5z*=280
fhe subscript p" is added to associate the slack variable with e primal problem.
The solution of the dual problem is
Y; = 0= 10 Y ;= 10 Y4 = 0 si = 5 W =280
S o m e Formal Observations
These observations are justified morc formall y in many books on linear programming
some of which have been inc1uded in the references at the end of this chapter. Here
the formal results are elaboraled using the solution 10 the example discussed in this
-

Table 3.25 Example 3.5: Dual Problem Phase 1 Table 3

y
Y2 Y3 Y4
$
$ 2 S3 a a
2 a3 b

o -0.2857 -1.1429
1.1429 1.1429
-1.1429 -l.429
2.8571

0.1786 0.2143

-0.2143 02857

0.2143
-0.2857
0.7143
0.2143 -0.1429

0.1429 -0.8571

-0.1429
0.8571
12.857

o -4.8571 2.4286

7.4286
3.4286

7.4286 -3.4286
w - 291.43

0.2857
1.1429
-1.1429 -1.1429

2.1429 2.1429
A - 2.857
Table 3.26 Example 3.5: Dual Problem Phase 1 Table 4
y
Y2 Y3 Y4
$
$ 2 S3
a
~ a3 b

o -0.25 -0.875 0.875 2.4999

0.125 -6E-O -0.1875 6E-06 0.5 0.1875 -6E -0.5 1.25

0.25 4 E-05 0.125 -4E-05 -0.125 4E-05 12.5

-3.0001 4.9998 6.4998 0.0002 -3.9998 -6.4998 -0.02 3.9998 w - 3 I O

o

A

146 LlNEAR PROGRAMMING

Tab1e 3.27 Exomp1e 3.5: Duo1 Prob1em Phasc D Tab1c 1
YI Y2 Y3 Y4 SI S2 S3 b

-0.25 -0.875 2.5

0.125

0.1875

0.5 1.25

0.25

0.125

12.5
O

-3 5 6.5

4
w - 31O
section. N o proofs 'e given. A l of the observations may not be relevant to Example
3.5 discussed above but are incIuded here for completeness.
1. The primal and dual problems have the samvalue for the optimal objective
function 7." = w " = 280. T his is true however only ifthe problems have optimal
solutions.
2. (f x is any feasible solution to the primal problcm. and y is y feasible solulion
to thc dual problcm. w(y) ~ 7.(x). This feature provides an estimatc of the
bounds of the dua1 optimal value if a fcasib1e primal solution is known. This
rcsult also holds for the reverse case when a feasibldual is known.
3. Ife prima1 problem is unbounded. the dual problem is infeasible. The
unbounded problem implies the objective function va1ue can be pushed to
infinite limits. lis happens if the feasible domain is not c1osed. O f coursc
practical considerations willlimit thc objective function value to corrcsponding
Iimits on the design variables. The inverse relationship holds too. If the dua1
problem is unbounded the primal is infeasible.
4. If the ith primal constraint is an equa1ity constraint the ith dual variable is
unreslricted in sign. The rcverse holds too. If the primal variable is unrcstricted
in sign. en thc dual constraint is an equality.
5. Obtaining primal solution from dual solution:
(i) If the ith dual constraint is a strict inequality then e iprimal variable is
nonbasic (for optimum solutions only). From the dual solution the second
Toble 3.28 Example 3.5: Dual Problem Phose 11 Tnble 2
YI Y2 Y.l Y4 SI S2 5J b
-2

-0.5 -0.5

0.25

0.375

2.5
2 0.5

-0.25

15
8

-2 5 5

I V - 300
:11
~ i
3.5 ADDITIONAl TOPICS IN LlNEAR PROGRAMMING 147
Tab1e 3.29 Examp1e 3.5: D ua1 Prob1em Phose 11 Tob1e 3
2
8
2
24
YI Y2 Y.l Y4 .fl S2 SJ b

-1.25 2 5

-1.5

4 10

o 0.5 o -2 10

5 2

8 .f-280
constraint (3.51b) is satisfied as an inequality. Therefo the second primal
variable x 2 is nonbasic d is is evident in Table 3.22.
(ii) If the ith dua1 variable is basic then the ith primal constraint is a strict
equality. From Table 3.29 Y 2 and Y 3 are basic variables therefore (3.47b)
d (3.47c) must be equalities. This is indecd truc.
Thesc relations can also bc cstablishcd by valucs of slack variables.
6. Rccovering primal solution from final dual table: W h e n lhe primal and dual
problerns are in the standard form the value of thc ith primnl variable equals
the reduced coefficient (that is the coefficient in the last row of the final tablc)
of the slacklsurplus variable associated with the ith dual constraint. In Table
3.29 the values of the reduced cost coefficient corresponding to the surplus
variables SIS2' and S3 are 2 0 and 8 respectively. T h e are precisey the
optimal values of the primal variables.
7. lf e ith dual variable is nonbasic the value of its reduced cost coefficient is
the value of slacklsurplus variable of the corresponding prima1 constraint .
In Table 3.29 YI and Y4 are nonbasic variables wireduced cost coefficients of
24 and 5 resptively. These arc the values of SI (Slp) and S4 (S4p) in Table 3.22.
8. Obtaining duaI solution from primal solution: W h e n the primal and dual
problerns are in the standard form the value of the ith dual variable equals the
reduccd coefficient ( isthe coefficient in the last row of the final table) of
the slacklsurplus variable associated with e ith primal constraint. ln Table
3.22 the values of the reduced cost coefficient corresponding to the surplus
variables SI S2' S3 and S4 010 0and 0 respectively. These are precisely
the optimal values of the dual variables in Table 3.29.
9. If the ith primal variable is nonbasic the value of its reduced cost coefficint is
the value of the slacklsurplus variable of the corresponding dua1 constraint . In
Table 3.22 X 2 is a nonbasic variable with reduccd cost coefficients of 5. This is
the value of S 2 in Table 3.29. Since XI and X 3 are basic with the rcduced cost
coefficient ofOecor sponding slacklsurplus SI and S3 variables will bzero
as observed in Tle 3.29.
T he above list referred to the primal and dual problems in standard form. A similar
list can bc obtained for nonstandard form. There will be appropriate modifications for
148 LlNEAR PROGRAMMING
negative values of variables well as equality constraints. T he listed references can
be consulted for further informalion.
3.5.2 Sensltlvity Analysls
The solution 10 the L P problem is dependenl on the values for e coefficients c b
and A (also termedparameters) involved in the problem. In many practical situations
these parnelers are only known approximately and may change from its current
value for any number of reasons particularly afler a solulion has been established.
Instead of recomputing a new solution it is possible to obtain a fresh one based on the
existing one and its final Simplex table. This issue is significant in practical problems
with thousands of variables and constraints. The adaptation of a nw solution without
re-solving the problem is called sensitivi.analysis. In L P problems sensitivity
analysis refers to determining the range of parameters for which the optimal solution
still has the same variables in thbasiseven though values al Ihe solulion may change.
Example 3.1 is used for is discussion. Figures are ed for i1Iustration rather than
a formal proof. Figures are not useful for more than two variables. Generally mosl
compuler codes that solve Iinear programming problems also perform sensitivily
analysis. Revisiting Example 3.1 it w necessary 10 identify the number of
component plncemcnt machines of lype A and B. The objective is to maximize e
number of boar to be manufactured. Constraint g I represents the acquisition dollars
available. Cons'aintg2 represents the floor space constraint. Constrainl g3 represents
the number of operators available. T he problem statement and the results are
reproduced below (note X3' X4 and Xs are thc slack variables)
The solution is
Minimize j(X): -99Ox
I
- 91X2 - 5250
subjecI to: gl(X): O.4xl - + 0.6X2 + X3 = 8.5
(3.5)
(3.6)
g2(X): 3xI - X2 + X4 = 25 (3.7)
g3(X): 3xI + 6x
2
+ X 5 = 70 (3.8)
x
1
~ O ; x
2
o x
3
~ 0 x
4
~ 0 X
s
~ 0 (3.9)
x ;= 10.4762 X;= 6.4285x ;= 0.4524] = -21437.14
Changing Cost Coefficlent Values (c): First consider the eect of changing
the cost coefficienls of the dcsign variablcs. If CIO the coefficicnt of design variable XI
is changed from -990 will the set of basis variables rcmain the same?
20
5
30
g s
1

-15
-20
-30
0
35 AODlTIONAL TOPICS IN LlNEAR PROGRAMMING 149
Sensltlvlty 10 Cost Coefflclent
25
?
. - -
- - - - -
- 16150
-
.
-- 16150
-16150
c1 = 200 -optlmum changed 10 A
5 10 15
Machlnes of Type A
guro 3.9 Sensitivity analysis changing cosl cicient .
Changing thc cost coefficient changes the slope of the line representing e
objective function at may lead to a change in solution. For example Equation
(3.5) can be written
x
2
= {990/900) X I([5250 + j]/900)
If CI were to be made -1900en the slope of e line will be (1900/900). T he
sensitivity due to CI can be experienced by running Sensltlvlty_cost.m.
1
T he
change wilh respect 10 the original objective function is shown in animation. T he
original solution is marked on the figure by a circle. T he original solulion is unchanged
when themaitude of the slope is raised for instance CI = -1900. O n the other hand
if CI were 10 be made -200 Ihe dotted line indicates the new cost line and in this case
the solution w i1l change to point A. T his is shown in Figure 3.9 is E sult of
running the m-fiIe. T he floor constraint is no longer binding. It cun be established by
simple caculations infinitely many solutions are possible if CI has a value of
-450. In Ihis casc the cost function is parallel to the binding cons int 83' From this
discussion it is apparent 10 keep the location of the original solution unchanged
CI must be grealer than -450. Such anulysis is possible for all Ihe other cosl
coefficienls involved in is problem. In particular the analysis should indicate for
lFiles 10 bc downlonded from tl 1e web silc nrc indicnled by boldface ftype
150 L1NEAR PROGRAMMING
what range of cost coefficients the solution would still remain where it is. This is
referred to as determining sensitivity to cost coefficients.
Change In the Resource Llmlts (b Vector): When the valuSon thc right-hand
side changcs the constraint I ineslplan/ bounries are moved pamllcl to themselves.
lis changes the feasible region. Threforethe optimal vaIues may aIso ch ge due to
the changes in efl sibl:gion. Once again the changes can be illus ted by running
S ensvis.m. Figure 3.10 iIIus tes the change in the right-hand side vaIue of
e frrst nstrnt. b.. m its vaIue of 8.5 to 6.5. T his changcaus to become an
active consaint . In this case the optimaI solution has moved from A to B. Note that in
the originaI problem g 1 was not active. Inenew formulation itis. However in sensitivity
analysise problcm is to discover thc range of b. so the solution still has esame
variables in e basis. lf b
l
we to remain above 8.0475 then the solution would still be
atA d gl will not be an active consaint
Change In the Coefflclent Matrlx A: A change in the coefficient matrix is
similar in effect to that due to the change in the cost coefficient while directly
20
15
10
5

o -5

E
10

-15
-30
0
S ensltlvlty to resource IImlts
b1:6.5-optimum changed- active constraint
5 ro 15
Machlnes 01 Type A
F1gure 3.10 Sensillvity analysis changlng righthand side.
REFERENCES 151
20r
Sensitivity to constraint coelficients (a11)
l I i l ~ O
-
- .
- ....
. . .
":----

5
@
~

-15
-20
m y
a1':0.6 - optlmum changed
s 10 15
Machines of Type A
Flgure 3.11 Sensillvily analysis changing cosl coefflclenl .
impacting problcm by changing the feasible region. These changes also depend on
whether the variable in the column is a bnsic varinblc 01' a nonbasic variublc.
Sensltlvlty_coeff.m illustrates the result of changing the coefficient all' A n
increase in the vaIue moves the optimaI solution to B while a decrease leaves the
solution unchanged. In both situations XI and x 2 are still part ofe basis. In the first
case g. becomes a binding constraint. This is illustrated in Figure 3.1 1. Thc
coefficient is in the flftcolumn which corresponds to a basic variable. Changing
coefficient values in the first column in other constraints w i1 yield similar
information. T he reader is encouraged to try other changes in the coefficient using c
m-files available in the code section of this chapter.
R E F E R E N C E S
1. Dantzig G. B. Linear Programming and Extensio Princeton University Press
Princeton. NJ. 1963.
2. Luenberger D. G. Linear and Nonlinear ProgrammilJgAddison-Wesley. Reading.
MA.1984.
152 LlNEAR PROGRAMMING
3. Winston W. L. lntroduction to Mathematical Programming. Applications and
AIgorithms Duxbury PssBclmont C A 1995.
4. Arora J. S.lntroduction Optimal Desigll. McGraw-Hi I I New York. 1989.
5. Wil1iams G.. Linear AIgebra with Applications. W m . C. Brown Publishers Dubuque.
IA 1991.
6. Nob1c.B.. d Daniel J. W. Applied Linear AIgebra. enti-HallEng1ewood Cliffs.
NJ.1977.
P R O B L E M S
(For all two-variable problems provide a graphical definitionlsolution of thc
problem.)
3.1 Solve the following I inear programming problem:
Min ft..xlX :z}: XI + X 2
Sub: 3xI - X2 3
XI
X I + X 2 S 4
XI
3.2 Solvc the following linear programming problem:
Max f(x )o xz): XI +
Sub: 3xI 3
XI +2x
2
S 5
X I + X 2 S 4
XI 2! 0; X 2 2! 0
3.3 Solvethfollowinglincarprogrammingproblcm:
Min f(x " X2): XI - X2
Sub: 3xI - X 2 3
XI +2x2 S 5
XI + X 2 S 4
XI 2! 0 ; X 2 2! O
3.4 Solve the following I inear programming problem:
M f(x " X2): XI + X2
Sub: 3xI - X 2 2! 3
XI +2x
2
S 5
X I + X 2 S 4
XI 2! 0 ; X 2 2! 0
3.5 Solve e following I inear programming problcm:
' A':_
1'1.. u ..
Sub: 3xI - X2 S 3
XI
XI
XI 2! 0 ; X 2 u cd in sign
PROBLEMS 153
3.6 The local bookstore must determine h o w many of each of the four new books
on photonics it must order to satisfy the ncw interest generated in the discipline.
Book 1 costs $75 w iII provide a profit of $13 and quires 2 inches of shelf
spacc. Book 2 costs $85 w i1l provide a profit of $10 and requires 3 inches of
shelf space. Book 3 co $65will provide a profit of $8 and requir 1 inchof
shelf space. Book 4 costs $100 will provide a profit of $15 and requires 4 inches
of shelf space. Find the number of each type that must be ordered to m ize
profit . Total shelf space is 100 inches. Total amount available for ordcring is
$5000. It has been decided to order at 1east a to of 0 of Book 2 and Book 4.
3.7 T he local community college is p1anning to grow biotechno10gy offering
through new fcderal and state grants. A n ambitious program is bcing p1anncd
for rccruiting at least 2stude from in and out of state. They are to recruit
at least 40 out-of-statc students. T hey w i11 attempt to ruit at least 30 students
w h o are in the top 2 0 % of their graduating high school c1ass. Current figuS
indicate about 8 % of the applicants from in state and 6 % of the applic ts
from out of state belong tois pooL T hey also plan to recruit at least 40 students
w h o have APcourses in bio10gy. T he data suggest that 10% and 1 5 % ofin-state
and out-of-stale applicants respective1y bclong to is pool . T hey anticipalc
that the additional cost per student is $800 for cach in-statc sludent and $1200
for each out-of-statc student Find thcir actual enrollment necded to minimize
cost and their actual cost.
3.8 Figure3.12repsents an optimization problem to delermine the m ax .imum tOla1
number of smaI1er rectang1es at w il fit within thc larger one. The
dimensions are indicated on e fig" C.ne number of the 1arger rctang1e
should be at 1east 5 morc than the sma1er oncs. Thc rectangles cannot be
rolated. N o fractional rclangles are al1owed.
(Optional: confirm aIl solutions using the Optimizalion Toolbox . from M ATI.AB.)
100
t
1.5
2.5 70
Rguro 3.12 Problm3.8.
4
N O N L I N E A R
P R O G R A M M I N G
Optimization problems whose mathematical models are characterized by nonlinear
equations are called Nonlinear Programming LP) problems. In C haptr 3 it w
noted at these problems also fell into the category of mathematical programming
problems. Engineering design problems are mostly nonlinear. In Chapr 2 several
problems were examined graphically nnd it was evidentthat curvature and the gradient
of the functions involved had a significant influence on the solution. In subsequent
chapters w e will continue to center e discussion of optimality conditions and
numerical tcchniques around two-variable problems because e ideas can also be
expressed graphically. Extension to more than two variables is quite straightforward
and is most simple when the p sentation is made through the use of vector a1gebra.
M ATI.AB will be utilized for all graphic needs.
Traditionally there is a bottom-up presentation of material for nonlinear
optimization. Unconstrained problems are discussed first followd by constrained
problems. For constrained problems e equality constrained problem is discussed
first. A silar progression is observed with gard to the number of variables. A
single-variable problem is introduced followed by two variables which is en
extended to a genernl problem involving " varinbles. This order allows incremental
introduction of new concepts but primarily allows the creative use of existing rules
to establish solutions to the extended problerns.
A n analytical foundation is essential to understand and establish the conditions at
the optimal solution will have to satisfy. lis is not for the sake of mathematical
curiosity but is essentinl component of the numerical technique: notably the
stopping criterin. T he necessary mnthemntical definitions and illustrations are
introduced in this chapter. References arnvailnble for refreshing the calculus and the
154
4.1 PROBLEM DEFINITION 155
numerical techniques essential 10 the development o f N L P [12). The books familiar
to the render should do admirably. T his chnpter also introduces the symbolic
computioll (computer algebra) resource availnble in MATLAB namely Symbolic
Math Toolbox [3).
4.1 P R O B L E M D E F I N m O N
In N L P it is not essential at all the functions involved be nonlincar. It is sufficient if
just onc of them is nonlinear. Ther are many examples in engineering in which only
e objective function is nonlinear w hie the constraints are linear. lf in this case the
objective function is a quadratic function these problems are termed lillear quadralic
problems (LQP). Optimization problems for most p rely on experience to
identify the mathematical model comprising of the design variables objective and the
constrainls. A knowledge of engineering or the appropriate discipline is also
essentialto cstablish a matllematical model . Primarily this involves determining the
functional relationships among thc design variables. Thc remaining task then is 10
establish the solution.
H o w does one establish the solution to the nonlinear optimization problem?
In mathematics (nfter nll at this stagc Lhere is a mathematical model for the
problem) the solutioll is obtained by satisfying the necessaa n d SI{cientconditions
related to the class of The necessary conditions ar those relations that a
C didate for the optimum solution mllst sati..Ifit doesenand this is important
it may be an optimal solution. T o qualify a design vector XP (X represents the design
vector) as an optimum it must sntisfy additionnl relations cnlled the sujJicient
conditions. Therefore optimum solution must satisfy bo necessary and sufficient
conditions. This chapter establishes these conditions for the optimization problem.
Example 4.1 is established next and is used in several ways in the remainder of E
chapter 10 devclop the conditions mentioned above. Once 1he condilions are
available the numerical techniques in optimization w iU incorporate them to establish
the solution.
4.1.1 Problem Formulatlon -Example 4.1
T he problem is stricted to two variables to draw graphical support for some of E
discussions. There are two constraints which during the development of this chapter
may switch betwccn equality and inequality constraints to ilustrate lated features.
Problem: Find thc rectangle of thc largest ar (in positive quadrant) atcan be
anscribed w in a given ellipse and satisfy a prescribed linear constraint.
From c problem specification the ellipse will provide an incquality constraint
while e linear relation among thc variablcs will provide cquality constraint for
tbis cxample.
156 N O NLlN E A R P R O G R A M M I N G
3
2.5
x
z
0.5
/
/

/
/

/
f

J
f
f

/
/

0.5 1.5 2 2.5

X
1
Flgure 4.1 Conslrainls of Example 4.1.
Mathematlcal Model: Figure 4.1 capturcs the esof the p lcm. Code filcs
E x 4 L 1 . m
l
and create_elllpse.m are filcs n cssary for M ATI.AB to create the
figure.'here are two variables X I and X 2lere are standard mathematical expressions
for e ellipse and the straight line is no problcm after Chapter 3.
Standard Format: The standard format of the N L P is reproduced here for
convemence:
Minirnize f(xtt X2' X
n
) (4.1)
Subject hk(x. h . x) = 0 k = 12. L (4.2)
(XIX 2. . X n) $; 0 j = 1 2 . . m (4.3)
xl$; X j $; xl' ;= 12... n (4.4)
In vector notation
IFiI 10 be downloaded from Ihe web silc c indicllled by boldfnce riflype.
3

4.1 P R O B L E M DEFINITION 157

Minirnize f{X)ln (4.5)
(4.6) Subject to: [h(X)] = 0
[g(X)]m $; 0 (4.7)
X
10w
$; X $; Xup (4.8)
For the specific problem being discussed and refening to Figure 4.1 the design
variables are the cool'dinate values xl d X 2 al allow thc computation of the
rectl Iar area. T he optirnization problem is
Minirnize -XI X 2 (4.9)
(4.10) Subjectto: hl(x. xz}: 20xI + 15 X 2 - 30 = 0
81(X. X2): (xT/4) + (4.11)
O$;xl$;3; O$;x2$;3 (4.12)
The side constraints in (4.12) can also be postulated as one-sided and can be written
as
X
1

(4.13)
4.1 .2 Dlscussion of Constraints
Using the relations (4.9)(4.12) several additional c1asscs of problems can be
described by including only a subset of thc re1ations. T hey are examined below.
Unconstralned Problem: Thcre are no functional constraints although the side
constraints arc necessary to keep the solution finite. For this cxample
Minirnize (4.9) j{X
I
-x
1

In this problem if design variablcs are unbounded at c upper limit then e
solution would be at the largest positive valuofXI and X 2A two-sided limit for the
dcsign variables is usually a good idea. The designer does have the ponsibility of
defining an acceptable design space.
Equality Constralned Problem 1: The functional constraints in this problem
only equalities. With reference 10 Example 4.1 c following problem can be scl up
(after changing c inequality to an cquality):
158 NONL1NEAA PAOGAAMMING
Minimize ft..x. X2): -x. x2
Subject to: 11)(x.. X2): 20 x. + 15 X2 - 30 = 0
h
2
(x.X2): + (4) 1=0
::;;3;
Intuitively such a problem m a y not b optimized since the two constraints by
themselves should establish the values for the Iwo design variables. T he argumenls
used in L P for acceplable problem definilion are also valid h. There is always the
possibility of multiple solutions-which is a strong fealure of nonlinar problems. In
such an event the set of variables that yield elowsl value for the objective will be
the optimal solution. NOle such a solution is obtained by scanning the acceptable
solutions rather an through application of any rigorous conditions.
Equallty Constralned Problem 2: Ifthe problem w e 10 incIude only one of lhe
constraints for example
Minimize ft..xlX2): -XI X2
Subjecllo: h
2
(x"X2): +
0::;; XI ::;; 3; 0::;; X2::;; 3
This is a valid optimization problem. A similar problem can be defined wilh the first
constraint by itself.
Inequallty Constrained Problem: In this case the constraints arc all inequalities.
A variation on Exarnple 4.1 would be (the equality consaint is transformed to an
equality constraint)
Minimize ftx.. x2l: -XI Xz
Subjcct to: g.(x" X2): 20 XI + 15 X2 - 30::;; 0
g2(X"XZ): (xt/4) + (4) 1::;;0
O::;;XI ::;;3; 0::;;x2::;;3
Like its countet in linear programmingis is a valid oplimization problem.
Equally valid would be a problem that incIuded just one of lhe constraints or any
numberofinuality constraints.
It is essenlial to understand both the nature and the number of constraints as
well as h o w they affect the problem. In general equality constraints are easy 10
handlemathematICallydifficulttosatisfy numerically.andmorerestrictiveonthe
search for the solution. Inequality constraints are difficult to resolve
mathematically and are more flexible with respect to the search for the optimal
solution as they define a larger feasible region. A well-posed problem that
4.2 MATHEMATICALCONCEPTS 159
someconstraintsareactive (equality) at the optimal solution as otherwise it would
be an unconstrained solution.
4.2 M A T H E M A T I C A L C O N C E P T S
Like L P some mathematical definitions are nsary before the nsary and
sufficient conditions for the N L P can be established. Definitions are needed for both
e analytical discussion as well as numerical techniques. MATLAB provides a
Symbolic Math Toolbox which permits symbolic computation integraled in c
numerical environment of MATLAB. This allows Ihe user 10 explo problems in
calculus linear algebra solutions of system of equations and other areas. In fact
using symbolic computation students can easily recover e prquisite information
for the course. A short hands-on exercise to symbolic computation is provided.
1
'he
primary definitions w e need are derivatives partial derivatives matrices derivatives
of matrices and solutions of nonlinear equations.
4.2.1 Symbollc Computatlon Uslng MATLA
One of the besl ways to gel fami1iar with symbolic computation is to take the quick
online introduction 10 the Symbolic Math Toolbox available in the MATLAB Demos
dialog box [4]. The following sequence locates the lutorial:
> > d e m o s
Symbolic M a t h - > Introduction
The computalional engine execuling symbolic operations in M A T1.AB is thc
kemel of Maple markeled and suppo d by Waterloo Maple Inc. lf E ader is
already familiar with Mapleen M AT1.AB provides a hook through which Maple
commands can be ex uted in MATLAB. T he symbolic computation in MATLAB is
performed using a symbolic object or sym. This is another data type I ike the number
and string data types used in rlier exercises. The Symbolic Math Toolbox uses sym
objects to present symbolic variables expressions and matrices.
In the exercise that follows a function of one variable and two functions of Iwo
variables (constraints from Example 4.1) at1 used for iIIustralion. Drawing on the
auor's c1assroom exnce is preliminary discussion is in terms of variables X
and y for improved comprehension. In later sections subscripts on x are used to define
multiple variables so that e transition to the general problem can be facilitated using
vector description. The functions in thcsc exeaises are
ftx)= 12+(x-I)2(x-2)(x-3)
g.(xy): 2 0 x + 15 y - 30
g2(Xy): x2/4 + l- 1
ITOO rclcasc of 100 new Swdent Edilloll includcs Ihc symbolic loolbOl ulong wilh Ihe basic MATLAD
sortwt
161 4.2 MATHEMATICAl C O N C E P T S

-
g

'
E
=

S
E
.

f
S
5

1
n
l
d
a

9 column vector based o n gl g 2 9 '" [gl;g2)
9
[ 20*x+15*y-30]
1/4*x+y-1]
9 can b e the constraint vector in optimization
problems
the partial derivatives o f 9 w i t h respect to design
variables is called the Jacobian matrix
the properties o f this matrix are important for
numerical techniques
x y .. [x y ] ; r o w vector o f variables
J '" jacobian(gx y ) calculating the Jacobian
15]
1]
a plot o f f for -2 p i
d f diff(f);
h01d 0 0
ezp10t(df [0 41) plotting function and derivative
combine with M A T L A B graphics - draw a line
1ioe ([0 4] [0 0) ' C010r' r' )
g
9
[ 20*x+15*y30]
1/4*x+y-1]
(default) 2 p i
< = 4
J
20
[ 1/4
ezp10t(f) x
plot between 0 < = x ezp10t(f [0 4))
"J'ljl
N O NLlN E A R P R O G R A M M I N G
le following M A T L A B session was capred as a diary file and edited in a text
editor. T he foe M A T L A B prompt does not appe. le boldface words are
commands at e reader will type at the command line.
160
defining x as a single symbolic object x " sym('x')
x
x
definition of multiple objects
variables in the workspace
Bytes Class
126 sym object
126 sym object
128 s y m object
128 s y m obje'ct
126 s y m object
126 sn object
g
of
g 1 g2
types
Size
1 X 1
1x1
1 X 1
1x1
1X1
1X1
syms y f
whos %
Name
f
g
g l
9 2
x
y
Grand total is 14 elements using 760 bytes
constructing f (x1)*(x-1)*(x2)*(x3) f.. 12 +
f
12+(x-1)"2*(x2)*(x-3)
diff(}
a o s
2*(x-1)*(x-2)*(x-3)+(x-1)^2*(x-3)+(x-1)"2*(x-2)
first derivative
Y 2.5
Additional symbo1 ic computations will be introduced through code as appropriate.
Note that e result of both numeric and symbolic computations can be easily
combined a10ng with graphics to provide a powerful computing environment .
t o evaluate 9 a t x 1
subs(g{ xy} {12.5})
ans
27.5000
1.7500
3
5

. Z O : ~

3
2
note the chain rule for derivatives
% note the independent variable is assumed t o b e x
diff( x 2)
ans
2*(x-2)*(x3)+4*(x-1)*(x-3)+4*(x-1)*(x2)+2*(x-1)"'2
the second derivative wrt x
the third derivative wrt x }

X
2

'
n
H
E

a
L
=

(
x

f
g
*

f
n
4

1
a
2

d

define 9 n
u

q
d

y

-
w
n
u
w

E
d

-
-
A
T
Y

*

X
5

*

4
0
+

2
x

=
*

E
n
u

t
-
1
4

a
g
g

define g2
g l g 2 c a n only have partial derivatives
independent variables have t o b e identified
}

x

t
-
g

'
t
=

8

-
a

E
M
n
u

i
n
2

d
a

partial derivative
g2 '" 0.2S*x + Y 1;

162 NONLlNEAA PAOGAAMMING

4.2.2 Basic Mathematical Concepts
The basic mathematical e1ements in the discussion of N L P are derivatives parti
derivatives vectors matrices Jacobian and Hessian. W e havc used the Symbolic
Math Toolbox in the previous scction to calculate some of these quantities without
dcfining them. T hese topics will have been extensively covered in the foundation
courses on mathematics in most disciplines. A brief review is offered in is section
using MATLAB. lis opportunity will also be utilized to increase familiarity withe
Symbolic Math Toolbox and incorporat it with e MATLAB commands that were
used in earlier chapter.
Functlon o'One Variable: ft..x) identifies a function of one variable:
f(x) = 12 + (x- l i (x- 2) (x-3)
is used as a specific example of such a function. The derivalive ofe function at e
location x is written as
- 1kx + ax) -.f(x)
dx-ilm Ax =llfI1-L
a xo a x ax->oax
(4.14)
x is the point about which derivative is compud. r is thc distame tu a
neighboring point whoslocation therefore will be x + ax. The value of the deri vati ve
is obtained as a limit of thc ratio of e differen in the value of the function at e
two points ( e dis separating the two points () this sep tion is
reducedzero. T he computation of thc derivative for the specific ex nple is usually
obtaind using the product rule. Results from the exe ise on symbolic computation
provide thc resuJt
= 3) + (x - 1) ) + 1) ^ 2)
Table 4.1 illustrates e limiting process for thc derivative of the function chosen as
e cxampJe. The derivative is being evaluated at point 3. As the dispJacements are
reduced value of the ratio approachcs value of 4 which is the aClual value of
the derivative. W hen a x is large (with a value of 1)cratio is 18. This is significantly
different from the value of 4. With a x of 0.1 the value is around 4.851. With furer
reduction to O.1 the ratio has a value of 4.008. From these computalions it Can bc
expected as a x approaches 0 the ratio will reach the exact value of 4.0.
Numerical Derivative C o m p u'ion: Many numcrical techniqucs in N L P (1ui
e computation of derivatives. Most software implementing these techniques do not
use symbolic computation. Automation and se ofuse quire at ese derivatives
be computed numerically. le results in Table 4.1 justify e numerical computation
of a derivative through a technique called thefirslforward d rence [5]. Using a very
small peurbation a x lhderivative at a poinl is numerically calculated as the ratio
4.2 MATHEMATICAL CONCEPTS 163
Table 4.1 Calculation or D erivaUve
x dx x + d x M
dfldx Derivative
3 4 18 18 4
3 0.1 3.1 0.4851 4.851 4
3 0.01 3.01 0.0408 4.0805 4
3 0.001 3.001 O.4 4.008 4
of the change in the function value 4f10 e change in displacemenlax. For E
exampJe the derivalive at x = 3 with a x = 0.001 is obtained as
ElI _f.:(3 + 0.001) 1(32-.11 nnSl
dxl.rc3 0.001
(4.15)
The derivativc for lhe singJe-variable function al any valuc x is aJso caJled lhe slope
or the gradietlt of lhc function al point . If a line is drawn tangenl 10 the function
al the value x the tangent of E gle that is Jine makes withe x-axis will have
the sne valuc as the derivative. Ifis angle is 9en
= tan9
Figure 4.2 illustrates e tangency property of the derivative.
M A T L A B Code: Figure 4.2 was crealed by the code tangent.m. In the same directory
e code derlvative.m provides the illustration of tangency through a fi re
animation in MATLAB. In e figure window both the function (drawn as the curve) as
well as the line representing 4fand a x are drawn. As the neighboring points arc closer
it is evideo m the figure that the curve can be approximated by a straight line.
When a x = 0.001e derivative is tangent to the curve as ey are indistinguishable.
H her Derivatives: The derivative of the dcrivative is the called e second
derivalive. Formally it is dcfined
-!!..(-li"L!;rn 1. . -4yalx
tJil- dx ldx)--10 a x : a x (4.16)
Similarly the lhird derivalive is defined as
164 N O NLlN E A R P R O G R A M M I N G
2 4
22
201-"
18
12
10
a
o 0.5
tangent -slope -derlvative at x=3
lo12+(xt)(x-l)(x-21(x-3)
1.5 2
x
2.5
Flgure 4.2 lIIuslrallon 01 the langent .
=)
3 3.5 4
(4.17)
This can go on provided e function has sufficient higher-order dependence on E
independent variable x.
Function of Two VBriBbles: The lwo-variable function
j{x:x214+y2-1 (4.18)
is chosen to iIlustrate the required mathcmatical concepts. The first important fcature
of two or more variables is the derivatives defined for a single variable (ordinary
derivatives) do notapply. eequivalent concept here is the partial derivative. Partial
derivatives are dcfmed forcach independcnt variable.le partial derivative is denoted
by the symbol d. These dcrivatives are obtained in the same way as the ordinary
derivative except thc other variables are held at a constant value. In the cxnplewhen
compuling e ptial derivative of x thc value of y is not allowed to change.
4.2 M A T H E M A T I C A L C O N C E P T S 165
dfl + ax y) -f(x y) (4.19)
dxl ^ /:u
1(.<. ) 1 )
Thc above relation expresses the partial derivative of the functionfwith respect to E
variable x. Graphically this suggcsts that thc two points are displaced horizontally
(assuming the x-axis is borizonta1). In the above expression (and prior ones too) the
subscript after the verticalline establisbes tbe point autwhichepartial derivative with
pectto x is being evaluatcd. (xy) se any/all points. A similar expression c be
written for the paral derivative offwith spect to y. For this example:
For the pointx = 2 y = 1:

-:"=2y
dx -2' dy
df df
f(2I) = 1; = 1 = 2
dx-" dy
Until this point w c have made use of symbols reprcsenting changes in values
(functions variables) without a forrnal definition. In is book
s(): presents finitelsignificant changes in the quantity ( )
d( ) o( ) : represents diercntial/infinitesimal changes in ( )
Changes in functions occur due to changes in the variables. From calculus [1] thc
di rential change inj{xy) (due to the differential change in variables x (dx)
and y (dy) is expressed as
L" dj . df
df= -:J d x + -:J dv
"J dx _. . dy-J
(4.20)
For convcnience and simplicity the subscript representing e point whcre the
exprcssion is evaluated is not indicated. The definition of the partial derivative is
apparent in the above expression as holding y at a constant value implies at =0.
Another interpretation for the partial derivative can be observed from thc above
definition: cltange of tlte function per unit citange in tlte variables.
Gradient 01 the Function: In the function of a single variable the derivalivc was
associated wi the slop. In two or more variables the slope is equivalent 10 e
gradient . T he gradient is a vector and at any point rep sents the direction in which
the function will increase most rapid Exarnining the conventional objective of N L P
minimization of objective mctions the gradient has a natural part to play in the
development of methods to solve the problem. The gradient is composed of the partial
dcrivatives organized as a vcclor. Vectors in is book are column vectors unless othcr
noted. The gradient has a standard mathematical symbol . lt is dcfined as
166 NONLlNEAR PROGRAMMING
Vf =
(4.21)
At this stage it is appropriate to consolidate is information using graphics. T he
graphicaI dscription of e exnple defined in Equation (4.18) has to be three
dimensional as w e need one axis for x anoer for y and the third forj{xy). Chapter
2 introduced three-dimensional plotting using MATLAB. A more useful description of
the problm is to use contour plots. Contour plots are drawn for specific values of the
function. In the iIIustration the values for the contours offare 0 12d 3. In Figure
4.3 two kinds of contour plots are shown. In the top half a three-dimensional (3D)
contour plot is shown while on lowerhalf same information is presena
two-dimensional (2D) contour plot (this is conside d thstandard contour plot). Thc
2 D contour plot is more useful for graphicaI support of some of the ideas in NLP. T he
3 D contour will be dispensd with in e remainder of thc book.
M A T L A B Code: Flg4_3.m. The annotation in Figure 4.3 (including the tangent linc
and the gradient ow)wasdonerough the plotiting commands available on e
menu bsr in the figure window (MATLAD version 5.2 and later). A m 0 1" pOl"tion uf
c plots a generated through the statcments in the m -fiIe indicated above. The codc
mixes numeric and symbolic computation. It allows multiple plots and targets
graphic commands to specific plots.
Discussion o( Figure 4.3: In Figure 4.3 point P is on contour f = O. Point Q is on
the contourf= 2. Point S is on the contourf= 1. Point R has e same y valuc as point
P d esame x vaJuc as point Q. Fore values in the figure the contour value is 0.75
(this value should be displayed in MATLAB window whcn the codFlg4_3.m is run).
line P Q is a measure 4f
line P R represents changes in dfwhcn d.y is 0
lineRQ presents changes in dfwhen d.x is 0
Mathematically fcan only be estimated by adding the changcs along the lines P R
andRQsince c definition of the partial derivatives only pcrmits calculating changcs
along the coordinate directions.
At point S the dotted line is the tangent. The gradient at thc samc point is normal
(pcrpcndicular) to c tI.ngent d is dir ted toward increasing the vaJue of e
function (indicated by e arrow). B y definition if dfrepresents a differentiaJ move
aJong the gradient at any pointen (dxP measured along the gradient vector)
df= +dy
(4.22)
3
2
0
3
4.2 MATHEMATICALCONCEPTS 167
3D conlour for f ( 1= x2/4+I-l
.
2
2.5
3

x
2Dconlour
3
2.5
gllldlenl 01 S
- longenl 81 5
2 3
x
Flgur 4.3 Gradienl and langen! IIn a! a point .
From figure the value offwiIl change along gradient direction. If dfrepresents a
differentiaJ move aJong gent lineen(dXare measured aJong gent line)
df=
e Jy
(4.23)
Equation (4.23) should bc zero because moving tangcntially (a small amount) the
vaJuoffis not changed.
T he Jacobian [J] defines a useful way to organize the gradients of sevcraJ
functions. Using threc variables and two func!ionsj{xyz) and g(xyz) the definition
of thc Jacobian is
I eJf
[JJ =1 1

(4.24)
168 N O NLlN E A R P R O G R A M M I N G
In Equation (4.24) the gradients ofthe function app in e same row. The first row
is e gradient of f while the second row is the gradient of g. lf the two functions are
coUected into a column vector. the differential changes [df dg]T in e functions due
to e differential change in thc variables [dx dy dz] can be expressed as a matrix
multipucation using the Jacobian

d
r
i
l
l
-
-
L

]

4 [

=

l
E
E

-
J
4

r
B
E
E
-
E
E
E
L

(4.25)
which is similar to Equation (4.20).
Hessian: T he Hessian matrix [H] is the same as the matrix of second derivatives of
a function of several variables. For ft...xy)
I da
[=1
I iJL[ iJL[ I
I dydx
(4.26)
T he Hessi matrix is s y m m ic. For example defined by Equion (4.18).
n
v

&

h
f
A
U

Functlon of n Varlables: For ft...X). where ] =[XIX2. . . . x.rr. e gradient is

T he Hessian is
d'i d
~ dx
1
iJx
2
iJx1ax
n
~ a'i
=1 dX2ax1 ~
~
a
iJxndx d..c
(4.27)
(4.28)
4.2 M A T H E M A T I CA l C O N C E P T S 169
Equations (4.27) and (4.28) will app r quite often in succeeding chapters. A few
minutes of familiarization will provide sustained comprehnsion later.
4.2.3 Taylor's Theorern/Serles
Slngle Varlable: T he Taylor series is a useful mechanism to approximate e value
of the functionft...x) at point (x
p
+ ) ife function is completely known at point
x
p
l e exp sion is (for finitc n)
I ~ ( 2+
+ ) ) + I( )
I ~
The series is widely used in most disciplines to establish continuous models. It is E
mainstay of many numerical techniques incIuding those in optimization. Equation
(4.29) is usually truncated the first two or terms wie understanding
approximation will suffer some error whose order depends on term tis being
truncated:
+ I{ A Y .1.1. fll
[(Xp+ At) f ( X J & J h ) + 2 1 d l J )2+ O ( )3
(4.30)
lf the first term is brought to e lefte equation. discarding the error term can be
written as
M= f(x
p
+ ) f(x :X/) = ))= ()+
dxL 21 dX- 1
- A b - x h
In Equation (4.31) the first term on e right is called thefirst-order/linear variation
while the second term is e second-order/quadratic variation.
Figure 4.4 demonstrates the approximations using Taylor' s series of various orders
at point 2.5 with respect to original function (red). T he principal idea is to deal wi
the approximating curve which has known propees instead of the original curve.
lC constant Taylor series is woefully inadequate. le linear expansion is only
marginally better. The quadratic expansion approximates the right side of the function
comp to the left side which is in significant eor. le fifthorder expansion is
definitely acceptable in the range shown. T o see how the Taylor series is used
consider the quadratic curve about c point x = 2.5. At this point value of the
functionft...2.5) = 11.4375 the value of thfirstderivativef'(2.5) = -0.75dthcvaluc
of the second dcrivative isf"(2.5) = 4. Using Equation (4.31)
(4.3 1)
ft...2.5 + = 1 5 + (- 0 ) +
(4.32)
For different values of At. bopositivc and ncgative the value ofj(x) can be obtained.
le plot of Equation (4.32) should be the same as one labeled quadratic in Figure 4.4.
171
(4.34)
4.3 GRAPHICAL SOLUTIONS
f(xp+dx+)=f(xp') + +


(4.35)
Fornv ableswith the current point and d.X the displacement vector
fp + IlX) = f(X
p
) + + j m T m
G R A P H I C A L SOLUTIONS
legraphica1 solutions are presented for three kinds of problems: unconsained equality
constrained and inequality cons inedbased on the functions in Example 4.1.
4.3.1
o 3; 0 3 (4.37)
The sidconstraints srve to limit the design space. Considering the objective
function in (4.36) it is clear that the minimum value of f w i1l be ralized if the
variables are at the maximum (Xi = 3= 3). The value of the function is -9.
Figure 4.5 il1ustrates the problem. The solution is at the boundary of the design
space. Fig4_5.m is the MATLAB code at will produce Figure 4.5. The maximum
value of the function is 0 (based on the side constraints). In Figure 4.5 the tangent
and the gradient to the objective function at the solution are drawn in using
MATLAB plotedit commands. lD is particular example the side constraints are
necessary to determine the solution. If the design space were increased the solution
would correspondingly change.
The xample chosen to illustrate e unconstrained problem defined by (4.36) and
(4.37) is not usually employed to develop the optimality conditions for general
unconstrained problems. T w o requirements are expected to be satistied by such
problems: (1) the solution must be in the interior of the design space and (2) there must
b a unique solution or e problem is unimodal . Nevertheless in practical
applications these conditions may not exist. Most software are developed to apply the
optimality conditions and they rarely verify ese requirements are being m et. It is
usua1ly the designer's responsibility to ensure these requirements are m et. To develop
eoptimality conditions an aItemate unconstrained problem is presented:
(4.36) JtXI x2): -XI X 2 Minimize
(4.38) JtX Ix2): (XI- 1)2 + (X2 - 1)2 - X I X 2 Minimize
1t 1
I
fifth-order orlg!nal
N O NLlN E A R P R O G R A M M I N G
Approxlmat!on us!rtg Taylor serles
170
22
-
0

a
Unconstralned Problem
4.3
20

18
i
E
12
4
M A T L A B Code: Figure 4.4 is completely created using the code Flg4_ 4.m. 1t uses the
MATLAB-provided symbolic tayloc" function to generate the various expansions
about point 2.5.
3.5 3 2.5
Taylor series approximalion.
2
x
1.5
Figure4.4
0.5 D
T w o or More Variables: The series only expanded to qu 'atic terms. The
truncation error is ignored.'he two-variable function expansion is shown in detai1 and
also organized in terms of vectors and matrices. The first -order expansion is expressed
in terms of the gradient . The second-order expansion will be expressed in terms of tbe
Hessian m aix
IMPI+
f(x'J + Y/J + Ily) = Jtx'J' y) + 1 +
Q 1 (.I"p}
IttpY/
e)2f I
(w-Y+
1 r 2 q
21 (4.33)
Ifdisplacements are organized as a column vector [ ]Tthe expansion in
(4.33) can be expressed in a condensed manner as
72 N O NLlN E A R P R O G R A M M I N G
4
3.5
3
2.5
.N 2
1.5
0.5

0.5 1.5 2
x
'
2.5
Flgure 4.5 Unconslrained solution.
o $ X I $ 3; O $ X 2 $ 3
3 3.5 4
(4.37)
Figure 4.6 displays the problem and its solution. Flg4_.6.m provides the code.that w iII
generale most of Figure 4.6. It is cJear at the solution appears to be ot
xi = 2 and xi = 2. The optimal value of e objective function is -2. In the contour
plote optimum is a point making it difficuJt to draw the gradient. Ifis is the case
en in three dimcnsions at the optimumc gradient w il Iie in a plane tangent to the
function surface. Moving in is plane should not change the value of the objectivc
function. T his observation is uscd to develop the n essary conditions later.
4.3.2 Equallty Constrained Problem
For a two-variablc problem w e can only utilize one constraint for a meaningful
optimization problem:
MinImize j{XIx2): 'XI X2
Subject h l(XI x2): X!/4 + ~ - 1
(4.36)
(4.39)
4.3 G R A P H I C A L S O L U n O N S 173
x
t
gure 4.6 Unconstrained problem: interior 1lIllon.
O S X
1
$3; O S x
2
S 3
(4.37)
Fiu "e 4.7 (rough FIg47.m) i1Iustrates the problem. The dashed line is e
nsaint . Since it is an equaJity constraint the solution must be a point on the dashed
line. The contour of f =1 appears to just gre constraint and therefore is the
minimum possible vaJue of the function without violating the constraint . In Figure 4.7
e gradient of the objective function as w eIl as e gradient of e constraints at the
solution arilIustrated. It appears at at is point these gradienls are parallel cvcn
though ey are directed opposite to cach other. B y definition the gradient is in the
direction of thc most rapid increase of the function at the selected point . This is not a
coincidenc. T his fact is used to establish the necessary conditions for the problem.
4.3.3 Inequallty Const Ined Problem
The number of inquality constraints is not dependent on the number of variables. For
i1Iusnbo consint functions of Example 4.1 ar formulated as inequality
consamts.
Minimize j{x) X2): - X I X2 (4.36)
174 N O NLlN E A R P R O G R A M M I N G
4
3.5
3
2
1.5
0.5
0.5 1.5 2
X
1
2.5
Flgure 4.7 Equalitynstralned problem.
Subjectto: gl(XIx2): 20xI + 15x2- 3 0 S 0
g2(Xl> X2): xI/4 +I S O
OSXI S3; O S X 2 S 3
3 3.5 4
(4.40)
(4.41)
(4.37)
Figure4.8 ilIustrates thegraphicalsolution. Thesolution isat the intersection ofthe
dotted lines. The code is a vailable in Flg4_8.m. le gradients are drawn using the
plotedit functions on the menubar in the window. Also font style and font size have
becn adjusted using the plotedit commands. The optimal solution must lie on or to E
left of the dashed line. Similarly it must I ie below or on the dashed curve.
Simultaneously it should a1so decrease the objective function value as much as
possible.
4.3.4 Equality a n d lnequallty Constralnts
Example 4.1 is developed with both types on constraints present . Il is produced here:
4
3.5
3
2.5
1.5
0.5
0.5
4.4 ANAlYTICAlCONDITIONS 175
1.5 2
x
t
2.5
Flgure 4.8 Inequality constralned problem.
Minirnize j{Xx2): -xl x 2
Subject 10: h
l
(XIx2): 20 XI + 15 X2 - 30 = 0
gl(XIx: (xI/4) + ( ~ ) I S 0
o S XI S 3; 0 S X2 S 3
3
(4.9)
(4.10)
(4.11)
(4.12)
4
T he graphical solution of Section 4.3.3 suggests that the inequality constraint will be
active at the solution. Ifis the case then Figure 4.8 will once again represent e
graphical solution to this section so. T his will be vcrified in thc ncxt scclIon.
4.4 A N A L YTICAL C O N D I T I O N S
A nalytical conditions rcfer to n essary and sufficient conditions twill permit
the recognition of the solution to the optimal design problcm. T hc conditions
developed hcre empower the numerical techniques to follow later. They are
176 N O NLlN E A R P R O G R A M M I N G
introduced in same sequence as in e previous section. lnstead of fonnal
mathematical details the conditions are established less fonnally from the
geometrical description of problem andlor rough initive reasoning. Fonnal
dcvelopment of thc analytical conditions can be found in Refercnces 6-8. For
establishing the conditions it is expected that the solution is in the interior of e
feasiblregionand there is only one minimum.
4.4.1 Unconstrained Problem
The problem used for illustration is
Minimize J(x..X2): (XI - 1)2 + (X2 - 1)2 - XI X2 (4.38)
(4.37) O S x
I
S3; O S X 2 S 3
Figurc 4.6 provided a contour plot of the problem. Figure 4.9 provides a
three-dimensional plot of the same problem. It is a mesh plot. The commands to create
Figure 4.9 are available in Flg4_9.m. The figure needs to be rotated intemctivcly to
appear as illus ted. A tangent plane is d m w n minimum for emphasis. Figure
4.9 w il1 be usd to identify the propees ofthe functionJ(xlxz) at the minimum.
The minimum is identitied by a supecript asterisk ( Xor lxi X2J
T
). Studying
Figure 4.9 the function has a minimum value of -2 while xj = 2 and x2 = 2. If e
values of X1 andlor X2 w e to change even by a slight amount from optimal value
in any direction the value ofthe function w il1 certainly increase since X' is the lowest
point of the concave surface rep senting e function f Reprcsenting E
displacemcnt from the optimum values of the variables as dXand thc changc in the
function valuc from the optimum as llf from direct observation it is clear that the
optimal solution must be a point at satisfies
4f> 0 for al1 d X
Flrst-Order Condltlons: The same idca can bc applied in the limit that is for
infinitesimal displacement dx
1
and dx
2
about X . The function itself can be
approximated by a plane tangent to the function at the solution (shown in Fire4.9)
(considcr is a first-ordr Taylor series expansion). Moving to any point in the
plane from the optimum (see Figure 4.9) will not change the valuc of the function
E fore df = O. Moving away from optimum implies dx
1
and dx
2
are not zero.
Rewriting Equation (4.20) in tcnns of X1 and X2
df=1+= 0
df=][:J=O
(4.42)
4.4 A N A l YTICAL CONDIT10NS 1 n
10
4
B
6

F
H
X
2

-2
4
X
2

F1gure 4.9 Three-dimensional plot 01 Figure 4.6.

Since this should hold for al1 points in e plane dx
1
1:- 0 and dx
2
1:- O. T herefore
df - f l - f l
d X
1
- v dX2 - v
or the gradient of f at the optimum must be zero. at is
V f ( x ~ x ; ) = 0
(4.43)
Equation (4.43) expresses the necessary condition or firstorder conditions (FOC)
for unconstrained optimization. It is termed the frrstordercondition because Equation
(4.43) uscs e gradient or the first derivative. Equation (4.43) is used to identify thc
possible solutions to the optimization problem. Iffunction were to flip over so at
the same function w c to maximize the value of e function the solution for the
variabls will be at the same value of the design variablc. It is clear that Equation
(4.43) appliesthe maximum problem also. Equation (4.43) by itself will not
determine the minimum value of the function. Additional considerations a necessary
to ensure at the solution established by first-order conditions is optimal in is
casc a minimum. For a general unconstrained problcm the necessary conditions of
Equation (4.43) can be stated as
178 NONLlNEAR PROGRAMMING
Vf(X')=0
(4.44)
which is primarily a vector expression of the relation in Equation (4.43). Equation
(4.44) is used to establish the value of the design variables X' both analytically and
numerically.
Second-Drder Conditions: The second-order conditions (SOC) are usual1y
regarded as sufficient conditions. It can be infe'ed at these conditions will involve
second derivatives of th function. The S O C is often obtained through the Taylor
expansion of the function to sccond order. If X is solutionand d.X represents e
changeof variablcs from optimal value which will yield a change d i f then
+ 6 X ) ) = (X+ t a h r
(4.45)
This is sirnilar to Equation (4.35) except e expansion is about c solution. 6f
must be grcater than zero. Employing the necessary conditions (4.44) the first term
on the right-hand side of Equation 4.45 is zero. This Icavcs e following inequality
a f = j a T H
(4.46)
whereH(X) is the Hessian matrix (the m aix of second derivatives) of the function
f at thc possibloptimum value X For e relations in Equation (4.46) to hold the
matrix H ( Xmust be positive definite. There are three ways to stablish the H is
positive definite.
(i) For all possible .6Xd.X
T
H(X').6X > O.
(ii) The eigenvalues of H(X') are all positivc.
(iu) The determinants of ] lower orders (submatrices) of H (X') that include th
main diagonal are aIl positive.
Ofthe reeonly (ii) and (iii) can be practically applied which is illustrated below.
Example
Minirnize j(XIx2): (XI - 1)2 + (X2 - 1)2 - XI X2 (4.38)
(4.37)
F D C
=2(x
l
- 2 ) - x
2
= O
(4.47a)
4.4 ANAL YTICAL CONDITIONS 179
= 2- 1 ) - x
1
=
(4.47b)
EQuations (4.47a) and (4.47b) represent a pair oflinearequations which can be solved
as-X1=2I2=2.The value of the t-unction f is-2.So far only the necessary
conditions have been satisfied. The values above can also refer to a point where the
nction is a maximum or where there is a saddle point .
SDC: For is problem [see Equation (4.26)]

b
l
r

-
H

Is it positive definite?
(i) Not possible 10 test all d.X
(ii) T o eigenvaues of H
= -3)-- ))
-1 2-l
'he eigenvalues ar= = 3 and the matrix is positive definite.
(ui) T o calcuate deterrninants of all orderal incJude the main diagonal and
incJude the element in e frrst row and first column
121>0
n
u

q
J

-
-
2

'a

The matrix is positive definite.

Sec_ 4 _ 4-1.m provides econfirmation of the numerical values for this example
using Symbolic Toolbox and basic MATLAB commands.
4.4.2 Equality Constrained Problem
T hc problem:
Minirnize f(XIx2): -XI X2
(4.36)

(4.39)

(4.37)
180 NONLlNEAR PROGRAMMING
Figure 4.7 illustrated e graphicaJ solution. It w noticed at at the solution the
gradient of objective function d thc gradient of the constraint w e parallel and
oppositely directed.Examining other feasible points in FistIm4.7(on the dashed
curve) it can be ascertned that the special geometrical relationship is only possiblc
at the solution. At thc solution a proportional relationship cxists between the gradients
at the solution. Using thc constant of proportionality AI (a positive value) thc
relationship between the gradicnts can be expressed as
=IVhl or V f + = 0
(4.48)
Equation (4.48) is usually obtained in a more formal way using thc method of
Lagrange multipliers.
Method of Lagrange: In lhis method the problem is transformed by introducing
an augmented function called the Lagrangian. as the objective function subject to the
same uality consnts. The Lagrangian is defined as the sum of the original
objectivc function and a linear combination of the constraints. le cocfficicnts of lhis
lincar combination known Lagrange multliers. With rcfcrence 10 thc
examplc in is scction
Minimize F(x2AI) =f(xlxu + AI h
l
(XIx2) (4.49)
be complete problem is developed as
Minimize F(XIx2AI) = -XIX2 + 1(./ 4 + 4 - 1 ) (4.50)
Subjecl to: h
l
(XIx2): 4 / 4 + 4 - 1
(4.39)
O::;;XI ::;;3; 0::;;x2::;;3
(4.37)
Is e solution to the lsformed problcm [Equalions (4.50) (4.39) (4.37)] the same
as the solution to the original problem [Equations (436)(4.39)(437)]?Ifthedesign
is f i sibleen most definitely yes. For feasible designs hl(x" X2) = 0d
otive functions in Equations (4.36) and (4.50) are the same. If design is nol
feasible thcn by definition there is no solution anyway.
be F O C are obtained by considering F(x" X21) as unconstrainedmctioll in
e variablcs XIX2l' T his provides ree relations to solve for xj xi
dF i)f ah
=
+ . -!.=o
ax

i)x
l
' .'. ax
a F af." ahl
=+= 0
aA2 ax
2
" '. aX
2
a F
= h . = O
(4.51)
4.4 ANAL YTICAL CONOITIONS 181
Equations (4.51) express e F O C o r nSy conditions for quality constrained
problem in two variables. Thc last equation in thc above set is the COns int equation.
is ensures the solution is feasible. The first two equations can be assembled in
vcctor form to yicld the same information expressed by Equation (4.48) which was
obned graphically. le Ieft-hand exp ssions in e first lwo equations abovc are
the gradient of the Lagrangian function:
V F = +IVh. = 0
Applying Equations (4.51) to thc example ofis section:
a F I X1
?=-x.+= 0
o X
1
L
a F
? =-x. + 2x.=o
".:__ - -""2 ' "'""ol
A
2
VA2
h 1 4 4 - l = o
(4.52)
Equations (4.52) present equations in variables which should determine
the values for xj. xij. Note Equations (4.52) only define the necessary
conditions which means the solution could be a maximum also.
T hcre is one problem with respect to the set of Equations (4.52). T hequations are
a nonlincar set . Most prerequisite courses on numcrical methods do not attempt to
solva nonlinear system of equations. Usually. lhcy only handle a single one through
Newton-Raphson or the biscction method. In fact N L P or design optimization is
primarily about tcchniques for solving a system of nonlinear uationsalbeit of
spific forms. T his means Equations (4.52) cannot be solvcd until w e advanccrther
in is course. Fortunately having a tool Iike M A AB obviates this difficulty and
provides a sng justification of using a numerical/symbolic tool for supporting
development of the course.
2
M AAB Code: In M A T L A B . there are two ways of solving Equations (4.52) using
symbolic support functions or using thc numerical support functions. Both procedurcs
havc limitations when applicd to highly nonlinear functions. The symbolic function is
solveand numerical function isfsolve. The numerical tcchnique is an iterative one
and requires you to choose an initial gucss to start the procedure. Quite often sevcral
different guesses may bc quired to find c solution. Solutions of systcms of
equations are fundamental to the rest of e book. The following code is included
a hands-on exercise. It is avaiIable as Sec4_ 4_2.m. It requires eqns4_ 4_2.m to
executeolve command. Thc coofe filcs are listcd bclow
111C nrc olher 10018 besldcs MATLAD such MUlhcnd Id ccl.
182 NONllNEAR PROGRAMMING
S 4_4_2.m
Necessary/Sufficient conditions for
Equality constrained problem

Optimization with MATLAB Section 4.4.2

% Dr. P.Venkataraman

Minimize f(X1 x2) -x1

- - - - - - -
symbolic procedure
- - " - - '
define symbolic variables
format compact .
syms x 1 x 2 lam1 h1 F
define F
F -x1*x2 + lam1*(X1*X1/4 + x2*x2 1) ;
h1 x1*x1/4 +x2*x2 -1;
h e gradient of F
grad1 ~ diff(FX1);
grad2 = diff(FX2);
optimal values
s aisfaction of necessary conditions
[lams1 xs1 xs2] sOlve(grad1 grad2 h1 'X1 x 2 lam1');
the solution is returned as a vector of
the three unknowns in case o f multiple solutions
lams1 is the solution vector for lam1 etc.
; m O R m T : t h e results a r e sorted habeti l y
fprintf is us ed to print a string i n the
command window
disp is u s ed to print values of matrix
f -xs1.*xs2;
fprintf('The solution (x1* x 2 l a m* f * ) n ' )
disp(double([xs1 xs2 lams1 f]))
- - - - - - - - - -
% Numerical procedure
- - - -
solution t o non-linear system using fsolve
see help fsolve

the unknowns have to b e defined as a vector

the functions have to b e set u p in a n m-file
4.4 ANAL YTICAL CONDITIONS 183
define initial values
xinit=[l 1 0.5]'; initial guess for x 1 x2 1am1
the equations t o b e solved are available in
eqns4_4_2.m
xfina1 ~ fso1ve('eqns4_4_2' xinit);
fprintf('The numerical solution (x1* x2* lam1*):n')
disp(xfina1) ;
eqns4_ 4_2.m
function ret eqns4_4_2(x)
x is a vector
x(l) x 1 x(2) x 2 x(3) lam1
ret=[(-x(2) + 0.5*x(1)*x(3))
(-x(l) + 2*x(2) x(3))
(0.25*x(1)*x(1) + x(2)*x(2) -1)];
Output In MATLAB C o m m a n d W l n d o w
The solution (x1* x2* lam1* f*):
1.4142 0.7071 1.0000 -1.0000
-1.4142 -0.7071 1.0000 -1.0000
-1.4142 0.7071 -1. 0000 1. 0000
1. 4142 -0.7071 -1. 0000 1. 0000
Optimization terminated successful1y:
Relative function va1ue changing b y 1ess than
OPTIONS.TolF un
T h e numerica1 solution (x1* x2* l a m*) :
1. 4141
0.7071
0.9997
The symbolic computation generates four solutions. Only the first one is valid for this
problem. This is decided by eside constraints expressed by Equation (4.37). This is
iIIustration of the a priori manner by which the side constraints affect the problem.
O n the other hand. enumerical techniques provide only one solution 10 the problem.
lIs is a function of the initial guess. Generally. numerical t hniques will deliver
solutions closest to lhe point they started from. The solution is
x;= 1.4141; x;=0.7071; = 1.0
'he sohons for XI dX2 C be verified from the graphical solution in Figure 4.7.
T he solutions can also be verified through the hand-held calculator. The value and sign
for are usually immaterial for establishing the optimum. For a well-posed problem
it should be positive sincc inc asing the constraint valuc is useful only if wc are trying
to identify a lower value for e minimum. T his forces h and f to movc in opposite
184 NONUNEAR PROGRAMMING
directions (gradients). Increasing constraint value can be associated with enlarging the
feasible domain which m a y yield a better design.
Lagrange Mult llers: T he Lagrange multiplier method is an legant formulation
to obtain the solution to a constrained problem. In overview it seems strange atwe
have to introduce an additional unknown (1) to solve the constrained problem. This
violates e conventional rule for N L P the fewer the variables the better the
chances of obtaining e solution. As indicated in the discussion earlierE
Lagrangian allows the ansformation of a constrained problem into an unconstrained
problem. The Lagrange multipliralso has a physical significance. At the solution it
expresses e ratio of change in the objective function to the change in
constraint value. T o illustrate is consider:
F = f + Ihl
d F = d f + Idl1
d F . d F . d F
:: aA.
dX
1
_.. . dX2 - ' 2 d-..
At the solution the F O C deems that d F = 0 (which can also be seen in the above
detailed expansion). Hence
L
1 - dh 611
The above dependence dos not affect the establishment of the optimal design. It
does hav an important role in the discussion of design sensitivity in N L P
problems.
General Equallty Constralnecl Problem: Remembering n - 1> 0
Minimize f(X)J'I (4.5)
(4.6) S u b j e c t [lI(X)] = 0
X
10w
S X S X u p (4.8)
The augmented problem with the Lagrangian:
Minim F(X')=f(X)+ L (X)
k=l
=f(X)+A' h;
=f(+ l T h (4.53)
4.4 ANALYTICAl CONDITIONS 185
Subject to: [h(X)] = 0
(4.6)
X
10w
S X S X u p
(4.8)
10 Equation (4.53) three equivalent representations for the Lagrangian are shown. The
FOCare
d F h" (4.54)
= + ) A." =0; i= 1 2
aJEi a x i z k h '
Subject: (X)]= o
X
10w

(4)
(4.8)
Equations (4.54) and (4) provide the 11 + 1 relations to determine n + 1 unknowns
x.Equation (4.8)is used after the solution isobtained-if necessaly.Equation
(4.54) is also expressed as
n
u

-
-

-
h
H

V

H

V

'
n
n
v

r

L
+

J

V

E
E

V

(4.55)
Second-Order Conditions: At solution determined by F O Cthe function
should increase for changes in 6X. Changes in X are not arbitrary. They have to
satisfy e linearized equality consaint at the solution. 1t is taken for grd atthe
sufficient conditions M e usually applied in a small neighborhood of the optimum.
Alsochanges are contemplatedonly with reSPECt to X and not with respect to thE
Lagrangemultiplier-TKLagrangemethodis o b n calledthe methodqfundetermined
coefficient.indicating it is oot a variable. 1n the aly cderi va6o noftheFOCfor
:pp emm l Lagrangia I was c onu nc o ns i B0r r n g f m mn t he
previoussc0 n( c on strained m in1iza t ion )S O C c beexpecd to satisfy the
following relations:
= F (X * + )-F '(X.) )=VF( '(X') dT [ v z 2F(X' ( X') )] O
V h T A X = O
In the above [V2F(Xlis the Hessian of the Lagrangianwith respect to the deSIgn
variables only evaluatdat the solution.Alsothe F 0 C mquimthatVF(X)=0. With
reference to two variables and one constraint:
186 NONLlNEAR PROGRAMMING
t:..F =+ 2)+
M = i+ 2 t L)+
(4.57)
t:..x
l
dh/ox
2
(4.58)
t:..x
2
dh1/dx
1
Substitute Equation (4.58) in Equation (4.57) and the S O C requires that the expression
in b r a m u s t be positive. derivatives in Equations (4.57) d (4.58)
evaluated at the m imum.
Applying the second-order condition to the example of is section is 1ft
exercise for the ader. Compard to e S O C for the unconstrained minimization
problem Equations (4.57) and (4.58) not syapplyespecially the substitution
of Equation (4.58). From a practical perspectivS O C is not imposed for uality
constrained problems. It is left to the dcsigncr to ensure by other means e
solution is a minimum solution.
4.4.3 Inequallty Constrained Optlmization
Theproblem
Minimize f(XIXz}:XI X2 (4.36)
Subject to: gl(XIx2): 20 XI + 15 X2 - 30 S 0 (4.40)
S O (4.41)
OSXI S3; O S X 2 S 3 (4.37)
The number of variables (n = 2) d the number of inequality constrints ( m = 2) do
not depend on each other. In fact. an inequality consained problem in a single
variable can be usefully designed and solved. Section 4.4.2 solved the uality
constrained problem. If the above problem can be nsformed to an equivalent
cquality constrained problem. en w e have found the solution. 1e standard
1sformation reqres a slack variable zj for each inequality cons ntg
j
UnlikeLP
problems the slack variable for N L P is not restricted in sign.berefore the square of
the new variable is added to the left-hand side of the corr ponding consaint . Thls
adds a positive value to the left-h d side to bring the cons int up to zero. O f course
a zero value will be added if the constrllint is alrelldy zero.
1; 1
4.4 ANAL YTICAL CONDITIONS 87
Transformatlon to s n Equality Constrained Problem
Minimize j(XIX2): -XI X2
Subjectlo: gl(XI'Xz} + zi: 20xI + - 30 + zi= 0
g2(X. X2)
OSXI S3; O S X 2 S 3
(4.36)
(4.59)
(4.60)
(4.37)
There are four variables (XI.X2. ZI. Z2) and twoequality cOllstraints. It is a valid equa1ity
cons ined problem. T he Lagrange multiplier m thod can be applied to this
m a u - - T o d nguish e multipociated wi iity cons
e symbol is used. T his is sct1y for cIty.
Method of Lsgrsnge: The augmentcd function or the Lagrangian is:
Minimize

=f(x
l
.X
2
) + s
1
[gI(X
1
x
2
) + zil + s2[C2(XI') (4.61)
If the Lagrangian is considered an unconstrained objective function. the F O C
(necessary conditions) are
d F d f . /l ogl . /l dg2
= - 4 1 + s
= 0
axl a x l a X 1 2 a x l
d F o f . /l ogl . /l
= + + s2 = 0
dX
2
dX2' I"' .dx
2
.
d F _ n
= 2 sz= 0
a Z
1
d F . n
? =2s.z.=0
a Z
2
d F ?
=g. + z f = O
dPI u l
iJF ?

ds
2
U "
(4.62a)
(4.62b)
(4.62c)
(4.62d)
(4.62e)
(4.62t)
188 NONLlNEAR PROGRAMMING
E quations (4.62e) and (4.62t) areuality constraints. Equation set (4.62) provides six
uationstosolveforx1xEz;Z5.91. . B y simple recombination Equons(4.62c)
to (4.620 can be collapsed to two equations while the slack variables ZI and Z2 can be
eliminatedfrom tlze problem.
First multiply Equation (4.62c) by ZIReplace z t by -81 from Equation (4.62e).
Drop the negative sign as wellthe coefficient 2 to obtain
18.=0
13282 = 0
(4.63a)
(4.63b)
Equation (4.63b) is obtained by carrying out similar manipulations with Equations
(4.62d) and (4.620. The F O C can be restated as
dF df. n d8. . n d82
= ;J + 132 = 0
dx. dx. r1dx.'
dF dr d8 I d8
=::L+ 132= 0
dX
2
dX
2
r1dx2
1318. = 0
13282 = 0
(4.62a)
(4.62b)
(4.63a)
(4.63b)
These four equations have to be solved for xj xijs 2 Note tzi Z2 are oot being
determincd-which suggests ey can be discarded from problcm a1togcer.
It would be useful to pretcnd z's ncvcr existcd in the first place.
Equations (4.63) lay out a dcfinite feature for a nontrivial solution: cither sj is zero
(and 8/ 0) or 8/ is zero ( ) . Since simultaneous cquations being solvede
conditions on e multipliers and constraints must be satisfied simultaneously. For
Equations (4.63)is slas into e following four cases. Thc information on 8 in
brackets is to cmphasize an accompanying consequence.
Case a: s. = 0 [8. < 0]; 132 = 0 [82 < OJ
Case b: =0[8.<0]; * 0 [82 = 0]
Casec: 13
1
* 0 [8. =0]; 13
2
= 0 [82<0)
C d: I0[8. =0); * 0 [82 = 0]
(4.64)
In Equation (4.64) if sj * 0 (or corresponding 8/ = 0) then the corresponding
consaInt is an equality. In the prcvious scction a simple reasoning was usedshow
c sign of e mtiplier must be positive (> 0) for a well-formulated problem.
W h ile the sign of the multiplier was ignod for e equality constrained problem it
is included as part of the F O C forinequality constrained problem. Before restating
l
4.4 ANAL YT'CAL CONOtnONS 189
Equation (4.64) the Lagrangian is reformulated without the slack variables (weare
going to pretend Z never existed)
Minimize F(x. x
2
.s .13
2
) = .f{x.x
2
)+ s.[g.(x..x2)] + s
2
[8T .X
2
)] (4.65)
which is the same formulation in Equation (4.53). The slack variable was introduced
to provide transformation to an equity cons int . It is also evident e
cons ction of L agrangian function is insensitive to type of constraint Since
e multipliers tied to the inequality constraint are required to bc positive while those
corresponding to the equality constraints are not this book will continue to distinguish
between multipliers. T his will serve to enforcc clarity of presentation. The F O C
for the problem:
e)F dr d8 I d8
= L 4 1 4 4 H = o
dx dx ' I"" 'OXt . I""Zdx.
dF df. n dg. . n dg
2
= + + 132 = 0
dXz dx
2
' 1""' d.Xz . I""Z d.Xz
Case a: = 0 [gl <0]; 2= 0 [g2 < 0]
Case b: 13 = 0 [8. < 0]; > 0 [g2 = 0]
Case c: s. > 0 [g. = 0]; s2 = 0 [g2 < 0]
Case d: s. > 0 [8. = 0]; > 0 [82 = 0]
(4.62a)
(4.62b)
(4.64)
Equation set (4.62) and any single case in Equation (4.64) provides four equations to
solvfor thc four unknowns of the problem. All four scts must be solved for the
solution. The best design is decidcd by scanning the several solutions.
Thc sign of the multiplier in the solution is not a sujJicient condition for the
uality constrained problem Thc value is un portant for optimization but m a y be
relevant for scnsitivity analysis. Generally a positive value ofmultiplier indicatcs
the solution is not a local maximum. Formally verifying a minimum solution
reqs consideration of the sccond dcrivative of thc Lagrangian. In practical
situations. if thc problem is well defined thc positive value ofc multiplier usually
suggcsts a minimum solution. This is used cxtensivcly in thc book to identify the
optimum solution.
Solutlon of the E xample: Thc Lagrangian for the problem is defincd as
F(x.x
2
.s.s
2
) =-x..Xz + s(2Ck. + 15Xz - 30)
TheFOCare
190 NONLINEAR PROGRAMMING
dF
Z ; = 4 2 + 2 O R I + 0 5 h = 0
(4.66a)
dF
5 5 = 4 1 + 1 5 P l + 2 h = 0
(4.66b)
s1
(2Ox
I
+ 15
Xz
- 30) = 0
s2
(0+ 4 - 1 ) = 0
(4.66c)
(4.66d)
T he side constraints
O S X
1
S3; O S
x.z
S 3 (4.37)
Case a:
sl = 0;
s2 = 0: The solution is ivial and by inspection of Equations (4.66n)
and (4.66b).
X1=0; x
2
=0; 1 = 0 ; g J = 30;
g2 = (4.67)
The inequa1ity conts are satisfied. The side con aints are isfied. The values
in Equation (4.67) rep sent a possible solution.
The solutions for the other cases ' c obtained using M A11.ABle code is available
in Sec4_4_3

m. For Cases b and c the appropriate multiplier is set to zero and the
resulting thrEe Equations in three unbEowns are solved.For casE d the complete set
of four equations in four unknowns is solved.cased is also SOlved numerically. le
M ATI.AB code will a1so run code Flg4_8.m which contains thcommands to draw e
figure. The output from the Command window is patched be10w.
The solution *** Case a ***(x1* x2* f*
91
9
2 )
o 0 0 30 1
The solution

**Case (b) **
(x1* x2
b2* f* 9 1 92):
1.4142 0.7071 1.0000 -1.0000 8.8909 0
-1.4142 -0.7071 1.0000 -1.0000 6.8909 0
-1.4142 0.7071 -1.0000 1.0000 -47.6777 0
1.4142 -0.7071 -1.0000 1.0000 -12.3223
0
The solution
*
Case (c)*** (x1* X2* b1* f*
9 1 92):
o . 7 5 0 0 1 . o o oo o . 0 5 0 0 - o . 7 5 0 0 o
o .14o 6
The solution
***Case(d}***{xl* x2* b l * b 2 * f * g lg2):
1.8150 0.4200
0.0426 1.4007 0.7624 0 0
0.8151 0.9132
0.0439 0.0856 -0.7443 0 -0.0
Maximum n u

of function evaluations
exceeded;
lncrease options.MaxFu
n Evals
Optimizer is stuck a t a minimum that is not a root
T r y a9ain with a n e w
startin9 guess
The numerical solution (x1
X 2
b1* b2*) :
0.8187 0.9085 0.0435
0.0913
4.4 ANALYTICALCONDITlONS 191
The solution for Case a was discussed earlier and is conrtrmed above.
Case b hns four solutions. The first one is unacceptable because constrnint gl is not
satisfied. The second solution is feasible ns far as the functiona1 constraints g and g2
are concemed but they do not satisfy the side constraints. T he third and fourth
solutions are unacceptable for the same reason. Thus. Case b is not nn optimnl
solution.
Cnse c is unacceptable because constraint g2 is in violation.
Cnse d hns two solutions the first of which is not acceptable for several rensons. The
second solution sntisfies nll of the reqrements
It sntisfies the constraints. 80th constmints nctive.

The multipliers e positive (maybe a sufficient condition).
It satisfies the side constraints.
le solution is
x
1
=0.8151; X2 = 0.9132;

=0.0439;

=0.0856;
1 = 0.7443;
gl =
0;
g2 =
0
This is nlmost confirmed by numericnl solution which nppears to hnve a problem
wi convergen. It is almost at the solution. 1t must be noled here that the MATLAD
function used to solve problem is part of the standard package. T he functions in
the Optimization Toolbox will certain1y do better. Nevertheless. the incomplete
solution is reported below but will be ignored due to e symbolic solution available
in Case d.
x=0.8187; x
2
=0.9085;
s
=O. 35;

=0.0913
?
?
?

E
n
s

There are now two candidates for e solution: e trivinl solution in Case a and the
solution in Cnse d. The solution in Cnsd is favored ns ithns a lower objective function
va1ue. All of the cnses above can be explored wispect to Figure 4.8. It is an
excellent facilitator for comprehending the cases and solution.
4.4.4 General Optimlzatlon Problem
'he genera1 optimization problem is described in the set of Equations (4.1)(4.8). le
specific problem in this chapter dfined in Section 4.3 is
Minimize j(Xl):'XX2
(4.9)
Subject to: h1(Xl
x
V: 20 x + 15 X2 - 30 = 0 (4.10)
gl(XxV: <4/4) + ( ~ ) I S 0 (4.1 1 )
OSXl S3; OSX2 S 3 (4.12)

i
1
1
1
1
1
4
I
1
1
J
j
q
193
(4.3)
4.4 A N A L YTICAL C O N O I T I O N S
;(XIoX2. . . xJ S O. j = 1.2. . . . m
N O N U N E A R P R O G R A M M I N G
ZAiil;;355
192
(4.4) XI j = 1.2. . . . n
The Lagrangian:
;Mu/iod: The problcm sfiy minimizing thc
Minimize
F(x
1
XnA.... /. sI' . . . . srn) = F(XJ
X
2.A...sU = - X .X2 + 1(20x. + 15x2 - 30) + s.(0.25x? +xi -1)
Minimize
Subject 10:
(4.70)
There are n + 1 + m unk.nowns. T he sarne number of cquations are r .uired to solve
e prob1em. ese "e provided by the F O C or e Kuhn-Tucker conditions:
f(x..X
n
)+h + ++s.g+ + srnCm
"1(XJ
X
2): 20xI + 15x2 - 3 0 = 0
OSX2S 3
OSXI S3;
(4.7 1)
i=1.2.....n
11 cquations are obtained as
dF df. dh" . dhl . n dg. . n dg
= + A.-:;++ P I - - + + sm-:
m
=0;
dx dx" - ' dx ---'dx -r ax -r m ax
(4.68a)
(4.68b)
a F
Z ; = - X 2 + 2I + O.5s = 0
a F
=-x. + x
2
= o
T h e F O C
luations arobtained dircctly ruugh ecquity constraints
a F
- = 2 1 +
(4.72)
m cquations are app1 ied through 2
m
cases. This implies ere are 2
m
possib1e
solutions. Thesc solutions must include Equations (4.71) and (4.72). Each cassets the
multiplier or corresponding inequality c 0nstmain tto ze Ifthe ml t
set to zero. en e corresponding consint must be feasib1e for an acccptable
solution. If e constraint is set to zero (active constraint). then the corresponding
multiplier must be positive for a minimum. With is in mind c m equations can b
expressed as
k = 1.2.. ...1

. X
n
) =0; (4.68c)
(4.69a)
(4.69b)
32;:; ; z z u y f a uirthesoluuo system
i23222!1223;
gl < 0
g. = 0
s =0;
s>O;
Case a:
Case b:
(4.73)
If con ons in Equations (4.73) are not met. e dign is not acceptab1e. In
implementing Equation (4.73). for each case a simu1taneous to of m va1ues and
cqualities must be assigned. Once these F O C conditions determine a possible solution.
thc side constrainlS havc to be checked. A s cvidenced in c examp1es earier. this is
not built into FOC. lt is on1y conftrmed after a possib1e solution has been
identified. Equations (4.71)(4.73) are referred to as the KuhnTuckerconditions [see
Exarnple 4.3 (Section 4.5.2) for additional discussion of the Kuhn-Tucker conditions].
then gj < 0
> 0
n

p
u

U
n
u
-
-
=

n
u

-
-
.

J

e
o

d
d

;:
(4.1)
(4.2)
f(X.x2.. .xJ
"k(XIX2... ..xn)=O. k = 1.2... ..1
Sulto:
Minimize
194 NONlINEAR PROGRAMMING
4.5 E X A M P L E S
T w o exarnples are presented in this section. The first is an unconstrained problem
has significant usin data reduclion. Specifically it illustrates the problcm of curve
fitting or regression. The second is thc bearn design problem explo d in Chapters 1
and 2.
4.5.1 Example 4.2
Problem: A set of y Z data is given. It is neccssary to find the best straight line
rough the data. This exercise is termed c ve fitting or dataduction or regression.
T o k P this exarnple simple the data points are generated using the parabolic equation
Z = O.5y2. The data are generated in arbitrary order and nonunorm intervals.
Least Squared Error: The objective function drives a large number of curve
fittinglregression methods is the minimization of the least squared error. le
construction of the objective ires two entiti-the data and type of
mathemalical relationship betw n the data. In this xarnple it is a straight line.
Generally it can be any polynomial or any function ror at manner. This is usually
the way corrclation equations are detcrmined in ex.perimcntal fluid dynamics and heat
transfer. For this exnple e data are the collection of points (Y;Z/)i = 12 n
T h 1ee. xp ecd straight Iinc y
Z=X1Y+Xz
where xl is the slope and X2 is e intercept. Using optimizing terminology there are
two design variable. For a cubic polynomia I there w il be four design variables.
Zpj is fitted value for e independcnt variable Y;e objective function which is
the minimum of thc squ e of the error ovcr aIl of the data points. can bc xp ssedas
n 11
Minimize f(x
1
)= L [Z/- zpi = L [Z/ (X+ X2)]2 (4.74)
;=1 1=>1
It is possible to complete formulation wi side cons nts aIough is not
necessy. le F O C ar
=2[Zj-(x
(4.75a)
=;2IZi- i
(4.75b)
4.5 EXAMPLES 195
Exp ding the expressions in the brackets and reorganizing as a line m aixequon
11
"
11
yf 2 Y; LZ;Y;
LY; L 11 Iiz;
/=1 I=J ;=1
Notthat the matrices can be set up easily and solved using MATLAB. The Hessian
matrix is the sque matrix on the left T he S O C requires at it be positive definite.
Code: Sec4_5_1.m: Exarnple 4.2 is solved using MATLAB d the results from e
Command window are shown below. Compare this code with at in Chapter 1. The
results are displayed on a plot (not included in book). N e w features in code are
the use of random numbers using the sum command and writing to the Command
window. From the plot and tabular vaIues it is clear that a linear fit is not acceptable
for is example. T he quaIity of the fit is not undcr discussion in is xarnple as it is
apparent a nonlinear fit should have bcen chosen. This is lefl as an exercisc. Thc
following will appear in e Command window whcn e code is run.
Results from Linear fit
objective function: 16.5827
design variables x 1 x 2 :
2.3458
-1. 5305
y i Zl. Z p diff
4.7427 11. 2465 9.5951 1. 6514
3.7230 6.9304 7.2031 -0.2727
4.3420 9.4266 8.6552 0.7714
0.3766 0.0709 0.6470 0.7179
2.4040 2.8895 4.1089 -1. 2193
4.1054 8.4272 8.1002 0.3270
4.1329 8.5404 8.1646 0.3757
3.3369 5.5674 6.2973 0.7299
2.7835 3.8739 4.9991 -1.1253
4.2032 8.8334 8.3296 0.5039
3.1131 4.8456 5.7723 -0.9267
3.1164 4.8561 5.7802 -0.9241
3.2415 5.2538 6.0737 0.8199
0.1954 0.0191 -1. 0720
1.0911
0.7284 0.2653 0.1781 0.0871
4.5082 10.1619 9.0450 1.1168
1.9070 1. 8183 2.9430 -1.1247
0.4974 0.1237 -0.3638 0.4874
196 NONlINEAA PAOGAAMMING
1. 9025
0.1749
1.8098 2.9325 -1.1227
0.0153 -1.1203 1.1356
A

X

-
- r

a

M

F

o

s

e

u

4
F
E
q
4
a
7
2

V
0
4

n
4
5

e

9
5
4

1
0
e
2

4.5.2 Example 4.3
This exmnplc is the same as Example 23.the flagpole pmMem.It is quite difficult to
3315
itziF c z
Minimize f(xl) = 6.0559 E -
(4.76)
Subject to:
gl(X ):7.4969E05.x7+ 40000x
1
- s 0
(4.77a)
g2(XI'):(5 + 1.4994E 05x
r
)(.x7+ X ) 17083E07(Xi S 0 (4.77b)
g3(Xl'x2):1.9 0 9 I E - 03xl +6.1116E-- 0.05(x1- xi) S 0
- x
1
+ 0.001 S 0
(4. 77c)
(4.77d)
(4.78)
0.02 S x
1
S 1.0; 0.02 S x
2
S 1.0
K'uh n-7 i 'ucke. r 'ondii Nons : F ou rLagrang c m L t pli ersa r innIr r
uty c o treUnts dthe Lagra ngian n is
F(Xs) = 6.0559 E 05(xf +
s p .4969E 05xf + 40 OX
r
- 9.7418E06(x1-xi)) +
13
2
((50+ 1.4994E 05x
l
)(xf + +
(1.9091E-03x. +6. 1l 1 6 E - 0.05(x1-4) +
s4( X
2
- x .+ 0.001)
(4.79)
4.5 EXAMPLES 197
where X = [Xo X2 ] and s = [131 132s4]T. le Kuhn-Tucker conditions require two
uations using the gradients of Ihe Lagrangian wispectlo e design variables
4
2
9

E
p
d
p
f
J
S
E
J

aF _ aF
=0: = 0
aX
r

aX2
(4.80)
The acaI expression for e gradients obtained from Equation (4.80) is left as
cxercise for the student.le Kuhn-Tucker conditions are applied by identifying the
various cases based on the activeness of various sets of constraints.
For this problem n = 2 and m = 4. Thcre are 2= 2
4
= 16 cases that must be
investigatcd part of KuhnTucker conditions given by Equation (4.73). While
some of these cases are trivial nevertheless it is a formidable task. This problem was
solved graphically in Chapter 2 and the same graphical solution can be exploited 10
identify the pticular case needs to be solved for the solution. Visiting C hapter 2
and sc ning Figure 2.9 it c be identified at cons ints 81 and 83 are active (as
confirmed by e zoomed solution in Figu 2.10). The solution is xT = 0.68 m and
xi = 0.65 m.
lf 81 and 8J are active consEn e m L ul
B y thc sa m e reasoning the m u ltipliers associated w i inactive constraints 82 d
E4' tiS132 and 134 must bc set to zero. T his information on the active constraints
can be used to solve for xj xi is is a sy m o f t w o uations in two unknowns.
This d not however complete sfaction ofe Kuhn-T ker conditions. 131
and sJ must be solved and verified that they are ve. 82 and 84 must be evaluated
and verified that ey are less than zero.
Sec4_5_2.m is the code segment atis used to solve particularcase discusscd
abovc. Thc problem is solvcd symbolically. T w o vcrsions of the problem are solvcd:
the original problem and a scaled version. 1objective function (4.76) is lJlodified
to have a coefficient of unity. This should not ch ge the value of the design variables
(why?). You can experiment with code and verify if is is indeed e. Note c
values of the Lagrange multipliers during is cxercise and infer the relation between
the m tipliers and the scaling of objective function. T he following discussion
assumes that the code has been run and values available.
Referring to e solution of thc casc above the optimal va)ues for design
variables are xj :::: 0.6773 m and xi = 0.6443 m. This is very close to the graphical
solution. Before w e concludethe solution is achieved take note of the multipliers:
= 5.2622e - 010 d = -0.1270. If the multipliers are negativeen it is not a
minimum. The values of e constraints at the optimal values of the design are 82 =
-4.5553e + 007 and 84 =0.0320. Both values suggest that the cons nts are
inactive. At least e solution is feasible-all cons ints are satisfied. The value of
st is zero and itis a1so an active con
.
}
198 N O NLlN E A R P R O G R A M M I N G
considered active. 1e solution however violat the side consaints.'he graphical
solution is rather explicit in identifying the minimum at epoin established in the
first solution. 1e vial case of the multiplier being zero when the constraint is active
is observcd with respect to the first two constraints express stress relations. ln
generalc tcrms in these inequalities have large orders of magnitude especially if
there are othrequations at express constraints on displacementsasor the like.
Large values exert a large influence on the problem so much as to cause lower value
constraints to have little or no signifi ncc for e problem. T his is usually a severe
problem in numerical investigations. T he remedy for is is scaling of the functions.
Scallng: Considcr e order of magnitude in Equations (4.77a) and (4.77c).
Numerical calculations are iven by larger magnitudes. lnequality (4.77c) will be
ignored in relation to the othcr functions even though the graphical solution indicates
is active. sis a frequent curren in all kinds of numerical t hniqu. T he
standard approach to minimize e impact of large variations in magnitudes among
diffe 1t equations is to normalize e relations. In practice this is also extended to e
variables. is is referred to as scaling lhe variables and scaling the funclions. Many
current software w i11 scale the problm without user intervention.
S ling Variables: 1e prcsencc of side constraints in problem formulation al10ws
a natura1 dcfinition of scaled variabJes. The user-defined upper and lower bounds are
used to scale each varlable between 0 d 1. T herefore

=--'; x:= scaled ith variable
X i - X i
X =x;(.-x:)+X:
(4.81a)
(48Ib)
In the original problem Equation (4.81 b) is used to substitute for the original variables
after which the problem c be expressed in terms of scaled variable. A n altemate
formulation is to use on1y the upper value of the side constraint to sca1e thc design
variable:
si=xi/4e
xj=xj
(4.82a)
(4.82b)
While is limits higher scaled va1ue to 1 it does not set e lower scaled value to
zero. For e example of is section there is no necessity for scaling the design
variables since their ordeJ' of magnitudc is one which is exactly what sca1ing atlempts
to achieve.
Scaling the Constraints: Scaling of the functions in e problem is usually critical
for a successful solution. Numerical techniqucs used in optimization are iterative. In
4.5 E X A M P L E S 199
c h iteration usually egrient of the functions at thc current value of the design
variables is involved in the calculations. This gradient exp ssed amatrix is ca1led
e Jacobi matrix or simply the Jacobiall. Sophisticated scaling techniques [78]
employ the diagonal enes of this matrix as metrics to scale the respective functions.
T heseenare evaluated at the starting values of the design variables. T he function
can also be sca1ed in e same manner as Equations (4.81) and (4.82). For the former
an expected lower and upper values of the constraints arnecessary. In this exercisc
e constraints will be scaled using relations similar to the relations expressed by
ualities (4.82). T he scaling factor for each consnt will be determined using E
starting value or the initial guess for the variables. A starting value of 0.6 for both
design variables is selected to compute e values ncessary for scaling the functions.
The scaling constants for the eqons are calculated
gO = 293888.4; g20= 102561.12; g30= 170E-03; g40= 1
1e first three constraints are divided through by their scaling constants. The last
uation is unchanged. 1e obj tive function has a coefficient of one. The scaled
problem is
Minimize 1-4
(4.83)
8.:
(4. 84a)
If
2
: (0. 88 + 1.4619x)(+x+ ) 166.5641 (xi - 0 (4. 84b)
: 1.0868x. + 0.3482 - 28.4641 (x1- 0
C 4 : X 2 - X I +0.001
0.02 XI 1.0; 0.02 5; x 2 1.0
(4.84c)
(4. 84d)
(4.79)
Sec4_5_2_scaled.m: The code (could have becn part of Sec4_5_2.m)
investigates the scaled problem in same way t the original problem was
investiga. Primarilyis exploits information from the graphical soI on. 1t is
expected t the solution for the design variab1es will be the samc although the
m u1tiplier va1ues are expected to be different . From the information in eWorkspace
window the optimal values for the design variables are xi = 0.6774 and xi = 0.6445.
A Il of constraints are feasible. The optima1 values ofe mu1tipliers have changed.
T he is no looger a zero multipIier-it has tumed positive. T here is still one
m u1tiplier with a negative sign implying the point has not met Kuhn-Tucker
conditioDS. Actually it should have been anticipated since changing oflhe sigll ofthe
multiplier is not possible with scaling.
200 N O NLlN E A R P R O G R A M M I N G
There is stilI the matter of inconsistency betw n the graphical solution and the
dissatisfaction of the Kuhn-Tucker conditions. Exploring this particular example has
been very torturous but has provided e opportunity to explore important but related
quantities Iikc scaling. Another important consideration is e scopc of the
Kuhn-Tucker conditions. These conditions apponly at reglllar points.
Regular Polnts: Regular points [8] arise when eq.uality constraints h(X) are
Psenteprob1em. leyara1so extended to active inI ityconsnts(pseudo
equa1ity consnts). Kuhn-Tuckcr conditions arc valid on1y for regu1ar points. le
two csscntia1 features of a regular point X. are
ThcpointXis feasible (satisfies alI constraints).
. Thc gradients of the ua1ity nstrints w clI as the active inlity
constraints at X. must form a line independent set of vectors.
In the sca1ed problem of Examp1c 4.3 thc constraints g I and 83 are active. The
solution xj = 0.6774 and xi = 0.6445 is feasible. The gradient of g I is Vg. =
[-37.631435.454T. The gradient of g2 is V g
1
= [-34.3119 30.4786]T. lt appears
that the two gradicnts arc a1most paral1cl to each othr. This is also cvident in the
graphical solution. The gradients thereforc are not Iinearly indepcndcnt-
Kuhn-Tuckcr conditions cannot bc applied at this point . This does not mean the
point is not a local minimum. T hris graphical evidence that it is indeed so.
KuhnTucker are the on1y available forma1 conditions for recognition of the
optimum va1ues. It is univcrsa1ly applied without regard to regulari. Some
additiona1 considcrations should bc kept in mind [8].
If equality constraints are present and al1 of e inequa1ity cons'aints are
inactiveen the points satisfying c Kuhn-Tucker conditions may be
minimum maximum or a sadd1e point. Highcr-order conditions are neccssary
to idcntify thc type of solution.
If the mu1tipliers of the active inequa1ity constrnts are positive the point
cannot be a 10cal maximum. It may not be a local minimum either. A point may
bc a maximum if multiplicr of an activc inuality constraint is zero.
R E F E R E N C E S
1. Stcin S. K. Calculus and Analytical Geometry. M c G m w - Hi I I New York. (1987)
2. Hostetler G. H. Santina M. S. and Montnlvo P. 0. Analytical. Nwnerical. and
Computational Methods for Science and Engineering. Prenti-Ha1I. Englewood CIi
NJ. (1991)
3. Molcr C. and Costa. P. J. Symbolic Math Toolbox-fore with MATUB-Users Guide
MathWorks Inc.. MA.
P R O B L E M S 201
4. MATLABDemos Symbolic Tool Box-Introduction. online resource in MATLAB.
MathWorks Inc. MA.
5. Burden R. L. and Faires J. 0. Numerical 'sis. 4th ed.. PWS_KENT Publishing
Company Boston. (1989)
6. Fox R. L.. Optimiwtion Methodsfor ElIgineering Design. Addison-Wesley Reading
MA. (1971)
7. V derplaatsG. N. N /lmerical Optimiwtion Techlliques for ElIgineering Desigll.
M c G m w -Hi11 New York. (1984)
8. AroraJ. S.lntroduction to OptimaI Design. M cG mw -Hill New York. (1989)
9. Kuhn H. W. and Tuckcr A. W. Nonlulear Programming. Proceedings 01 the Second
Berkeley Symposium on Mathematical Statistics alld Probabi/i. J. Ncyman (cd.)
Univcrsity of California Prcss 1951.
P R O B L E M S
(In many of the problems be1o w you are required to obtain the numerical
solution.)
4.1 Definc two nonlinear functions in two variables. Find thcir solution through
contour plots.
4.2 For thc functions in Problem 4.1 obtain the gradients of the function and UI
Jacobian. Confirm them using the Symbolic Math Toolbox.
4.3 Define the design space (chooses side constraints) for a two-variable problcm.
Dcfine two non1inear functions in two variables that do not have a solution
within this space. Graphica1ly confirm thrcsult .
4.4 Define a design space for a two-variab1e problem. Definc two nonlinear
functions in two variables have at least two solutions within e spacc.
4.5 Define a nonlinear function of two variables. Choose a contour value and draw
thc contour. Idcntify a point on thc contour. Calculate the va1uc of the gradient
at that point. Draw the grdient at the point using the computed valueCalculatc
Hessic above point .
4.6 Using the relntion in Equation (4.23) establish tthe gradient is normal to the
tangent [a physical interpretation for Equation (4.23)].
4.7 Define a nonlinear function of variables. Choose a point in the design
space. Find e gradient of function at the point . Ca1culate the Hessian
matrix at the samc point.
4.8 Exprcss the Taylor series expansion (quadratic) of the functionftx) = (2 -3x +
x2) sin x about point x = 0.707. Confirm your results through the Symbolic
Math Toolbox. Plot the original function and e approximation.
4.9 Expand the functionftx.y) = 10 (1 - x2)2 + (y - 2)2 quadratica1ly about the p ~ i n t
(11). H o w w il1 youdisplay the information? Draw the contours of the origina1
function and the approximation.
202 NONLlNEAR PROGRAMMING
4.10 Obtain the solution graphically forthecase when Equation (4.10) is added to
problem defined by Equations (4.36) (4.39) and (4.37). Obtain e solution
using e Symbolic Math Toolbox.
4.11 Obtain c solution to thc optimization problem in Scction 4.4.4 using MATLAB.
4.12 1n Section 4.5.1 obtain the coefficients for a quaC fit to e problem.
4.13 Use an example to show optimum values for the design variables do not
cb gethe objective function is multiplied by a const . Prove the same if a
constant is added to e function.
4.14 H o w does scaIing of the objective function affcct the valuc of e multipliers?
Usc an cxamplc to infer the sult. H o w does scaling the constraint affect the
multipli s ? Check with an example.
5
NUMERICAL TECHNIQUE8 -
T H E ONE-DIMENSIONAL
P R O B L E M
'he one-dimensional unconstraincd optimization problem is often introduced d
solved numerically in most courses at deaI with numerical analysisJtechniques. It
appears indirectly notably as finding the root of a nonlinear equation which is e
same as satising e F O C for an unconstrained optimization problem. T w o of the
popar techniques are e Newton-Raphson and e biscction technique.'hey E
introduced in Section 5.2 to provide a comparison with the methods used in
optimization. In engineering dcsign e single-variable optimization problem is
problya vialitybut one-dimensional optimization is a criticaI component of
m tivariable desi and is discussed in Son 5.3. Meanwhile the cIassical root
finding methods identified above do not play any significant part in numerical
optimization techniques as they are considered computationally expensive and not
robust enough over a large cIass of problems. A n important considcration in nonIinear
design optimization is that during the initial iterations when you are typically far away
o m e solution accuracy c bc ded for speed. In these iterations it is also
important to move through to c ncxl itcration instcad of faltering at e current one.
T hese.criteria have quid that thc one.dimensional techniques be simple in concept
as well as easily implemcntable. T w o different techniques the polynomial
approxlation and e golden section provide popular support for most optimization
softwe. Very often the two techniques are implemented in combination. T hey are
discussed in Section 5.2.
203
204 NUMEICALTECHNIOUE5 -THE ONE.OIMENSIONAL PROBLEM
5.1 P R O B L E M D EFlNITION
In order to connect with the discussions later the one-dimensional variable will be
identified nsinstead of X or XIIn the fol1owing chapters is is a1so referred to ns
the onc-dimensional stepsize computation.
E xample 5.1 T his example is similar to one in Chapter 4:
Minimize f() = ( _ 1)2(- 2)(- 3)
Subjectto: 0 S S 4
Theproblemdoesnotrepresentanyrealdesign.Itwasconstructedtohavemultiple
local minimums within the area of interest. Expression (5.1) reprcsents an
unconstrninedproblem.Thesideconstraintsaccompanyal1problcmsinthisbookto
con vey theidea that therearenotru Iy u nconstrained prob lems. Sideconstr ain tsalso
define an acceptable design region for all problems. Section 5.2 will explore this
prob lemthroughthcvar iou smethodsmen tionedcar I ier .
5.1.1 Constralned O ne-Dlmenslonal Problem
The only that Example 5. 1 a single-variable problem can a ommodate is
inequaliconstraint . From thc previous chaptcr there can be m o onc such
cons int . A single-variable problem cnnnot have uality constraint (Chapter 4).
In a constrnind multivariable optimization problem it will often be necessary lo.find
e stepsizesuch econsintjust becomes active. It is possible to defme
problem.
Example 5.1a
Minimize f(.)=(1)2(- 2)(- 3)
Subject to: g(): 0.752-1.5I S 0
(5.1)
(5.3)
O S S 4 (5.2)
In actual applications it is more I ikely at cxpressions (5.1)(5.3) establish two
problems that provide different solutions to c stepsizc. T he first is e solution to
expressions (5.1) and (5.2) which is the samc as Exnmplc 5. 1. T he second is the
solution to
g(.):0.752- 1.51 = 0
O S S 4
(5.3)
(5.2)
(5.1)
(5.2)
5.2 SOLUnON TO THE PROBLEM 205
The second solution is useful only if expression (5.3) is an active constraint Otherwise
the solution to Example 5.1 w il1 prevai1.le final solution chosen ensu sexpressions
(5.1)(5.3) are satisfied. A graphical investigation in e next section should illustrate
this feature c1early.
5.2 S O L UTlO N T O T H E P R O B L E M
The solution to Examples 5.1 d 5.1ais examinedgrphicaIIy. Next the c1nssical root
finding tec iqucs ofNewton-Raphson and ebisection method are presented. T hese
M followed by the polynomial approximation and e golden section mcthod.
5.2.1 Graphlcal Solutlon
T he example has already made its appearance in Chapter 4. Figu 5.1 is a plot of the
objective function in Example 5.1. It also includes c solution Example 5.1a. If
expression (5.3) represents an incquality constraint the feasible region for one-
variable functions is quite different from the two-vruiable problems encountered in

J
" '
/
3
g()

'
h
{

)
-
constralned minimum

1
.

H J " ' l g l o b a l
unconstrained minimum
.
;.. - -

-2
4

2
a
3.5 0.5 1.6
gure 5.1 Graphical solulion: Example 5.1.
206 NUMEAICAL TECHNIQUES THE ONE.OIMENSIONAL PAOBLEM
::tz;;:;321i222211;
1 F Z Z ; i t m t
Example5.1
=2.6484; f* =0.6195
(5.4a)
Example 5.1a
* = 2.5275; f. = - 0.5815
(5.4b)
5.2.2 Newton-Raphson Technique
T he Ncwton-Raphson technique also rcferrd to as Ncwton's technique is a gradient-
bascd n to finding c root of a singlc nonlinear eq on. Root finding
lli t U 1 2 2 Z Z ; : : r he equatia ;:;L
()=0
wher() is a nonlinear equation in the variable .
le technique has additional featurcs:
It hasag metric undeinning.
It usthe Taylor series expanded I inearIy.
It is iterative.
It has the property of quadratic convergence.
Any iterativc tcchnique addrcssing Equation (5.5) can be summarized as
Generic Algoritlrm (A5.1)
Step 1. Assum
Step 2. Calculate .1
Step 3. Update a = + .1
If converged ( () =:0) en exit
If not converged ((:;f: 0) ena
go to Step 2
I
p
iI
10 be downloaded fwm thc web sile indicaled by boldfncc snns serif Iype.
(5.5)
5.2 SOLUT10N TO THE PAOBLEM 207
The above sequen is fairly standard in iterative techniques for many classes of
problems. It also caplures the essence for most of the techniques to follow in this book.
Slep 1 indicates the primary f i rc of iterative m eods-the starting solution. T his
is an initial guess provided by the user 10 slart the iterative process. The iterative
process is continued 10 the solution by calculating changes in the variable (.1.). T he
value of the variable is updated. Convergence is checked. If Equation (5.5) is satisfied
then the solution has been found. If convergence is not achieved the updated
value provides the value to proceed with next iteration. Of course .1can be
calculated in many ways and Newton-Raphson is just one of the ways establish the
I change in the variable.
l j
Calculatlon 01 A(Newton-Raphson): Let b e the current value of the
variable. It is assumed that Equation (5.5) is not satisfied at is value of the variable
(e ason for us to iterate). Let a be a neighboring value. Ideally we would like 10
achieve convergence at this valu(even w e are aware it rnight take us several
itcrations 10 achievc convergnce). Using Taylor's theorem expanded to the linear
terrn only
d
() = ct(+ ) = ()+25A=0
A =1=-'Icb(o.)
Id
l

(5.6)
(5.7)
For Equation (5.7) to bc effcctivee gradient of the function should not be zcro. lt
is a1so inevitable at changes inwill be large where ct is flat and smal1 where the
slope is large. For the Newton-Raphson method to be effcctive the iterations should
avoid gions where the slope is small . This is a scrious disadvantage ofis mcthod.
Figure 5.2 ilIustrates the geometrical construction of tbe techniquc at the point = 3.
Example 5.1 T he Newton-Raphson tcchnique is applied to the first-order conditions
(FOC) of Example 5. 1.
Minimizc f()=(1 )2(2)(- 3) (5.1)
Subject to: 0 S S 4 (5.2)
T he FOCis
{ = = l - 2 2)( -3) )+( 1)3 )+( l) r-2 2
(5.8)
208 NUMERICAL TECHNIQUES -THE ONE.OIMENSIONAL PROBLEM
10
8
6
2

-2
0 0.5 1.5 2 2.5 3

Flgure 5.2 Development 01 Newton.Raphson technique.
3.5 4
J e Technlque: T w o iterations are shown here. T he complete application is done
through MA11AB codesecs-2-2.m.The code uses symbolic calculation.In rumino
code the figure window displays rnation of the iterati ravel6 0 ;
Also shown is the change in 6with each iteration. T he iterative results are
summarizcd in a table 8t e cnd. The ca1culations requirc the gradient of():
'()=2(-2)( - 3) +4*(1)*(3) + 4*(1)*(- 2) +2*(1)^2
Iteration 1:
=3; f()=0.0; () = 4 I t'()=16 6 = -0.25
I t( + L1.) = 0.875-not converged at least another iteration neccssary
Iteration 2:
=2.75; f(1.)= -0.5742; (.)=0.0875; .)=9.25; = O.46
(+6) =0.1040 not converged at least anothcr iteration nccessary
5.2 SOLUTlON TO THE PROBLEM 209
'he code S ec5_2_2.m ruires five iterations to converge the va1ue of I t to
1.0 E8 (considered zero here). Ts vaIue of 1.0 E08 is termed the convergence (or
stopping) criterion. T hc acccleration of convergence or rate of convergence to thc
solution is formally defincd in many different ways. One of thc ways is to monitor the
ratio of ( + 6 )1(). A nothcr istrack 6. ln applying and understanding
numericaI techniques it is important to develop an instinct and feel for the numbers
corresponding to various functions e iterations progress. The speed or equality
of convergence can be gauged by e shift in the dccimaI point in the quantity of
interest through the iterations. Consider the vaIue of I t through the five iterations
[4.00000.87500.10400.00230.0000]. Definitely e changcs in are nonlinear.
T he Newton-Raphson technique can be direcUy applied to Equation (5.3)-after
aIl the meUlod is designed to operate on these kinds of problems. It is left as an exercise
for r der. lere is a very important feature of most itcrative techniques that
exp!ore m tiple solutions-as is e case of Example 5.1. Rerun the code
Sec5_Z2.m with a starting vaIue of 0.5 for a.. After six iterationsconverges to
e vaIue of 1.0 which is a locaI minimum. W here would iterations lead if the start
vaIue is 1.5 ? These answers are sy to s graphicaIly in a one- or maybe a
two-variable problem. In a typicaI engineering design problem significant experience
and intuition are required to recognize Ulesc situations when cy occur. A healthy
S picion of the solution must aIways be in the back of one's mind when exploring
multivariable problems. In summary most iterative numerical techniques are
designed to converge to solutio1S that are c/ose 10 where they startfrom.
5.2.3 Bisectlon Technique
T his is another popular tcchnique to find tof a function. Itis also caIled a binary
search or intervaI haIving procedure. Unlike the Newton-Raphson method this
procedure does not quire the evaIuation of e gradient of e function whose root
is being sought. One of the reasons the method is being included is to compare it with
the golden scction method discussed latcr. The numericaltechnique in is method is
basedontheoning therl

t(or zero) of a function is trapped or bound between

a positive and a negative vaIue of the function. T he actuaI solution achieved by E
method is an intcrval in which ezero or the solution can be located. Thc finaI interval
is typilIy dctcrmined to be a vcry smalI value callcd thc tolerance of the ordcr of
lOE-08. Sce is is a root finding procedure when applied to optimization it is
applicd to gradient of ottive which w e identificd as (.) Equon (5.5).
Establishing the minimum of the objective is thereby translatcd to locating e rootof
e gradient function. It is possiblc to devclop a bisection tcchnique for directly
handling the minimization problem.
T he method ruires two points to start sayn and bThe vaIue of It at E
points must be opposite in sign.'hesc can be the sidc constraints of the
one-dimensionaI optimization problem. lt is assumed at Icast one solution exists
in is initiinterval. During each iterationis interva1 is haIved with e midpoint
of the interval replacing cither orbwhile kccping thc root still trappcd bctween
210 NUMERICAL TECHNIQUES -THE ONE-DIMENSIONAL PROBLEM
amction values ofopposite sign.The itemtiveMmique is expressed as the following
aIgorithm (a set of procedures e repeated).
AlgoritIJm!or Bisection Metllod (AS.2)
Step 1: Choose and to start . Let
Step 2: Set = + ( )12
Step 3: If(.) = 0.0 - Converged Solution - exit
se If (b-n)< lO E04 - tolerance met - exit
Else If(.)* ()>0.0; en
Elseb
go to Step 2
W e consider Example 5.1 which is reproduced here
Minimize f{) = ( _ 1)2(- 2)(- 3)
Subject to: 0 S4
ThcFOCis
(
(5.1)
(5.2)
(5.8)
S 5 _.3.m: T his AB code runs the bisection method on ple 5.1. It
iIlustrates the trapping of e mullrnum by the two values on either side Of the mot.
le 17 iterations for a tolerance of 1.0 E04 which terminated thc program.
Compa his with 5 itcratis for the Newton-Raphson 1 hod. for a bettcr solution.
U lower number of iratJons c be to quaJity of ir'nnation ed to
upde d variable changcnc gradient-bascd information uscd in thc
Newton-RaPIEBon tEChmque ls more sophisticated.though cgradient computation is
an additionaJ work load.
T K problem described by Equation (53)cmbe handled directly by thEbisection
od (.) is same (). In a w a Y. t he
opp o utY to rev1 1 sit prior knowlcdge. parti arly with resp defining and
slating an algori into IUDng code. There is u su a stron g c orre spo nd l
betwecn e stcp-by-stcp d cription of e numericaJ technique (the algorithrE
e c od e at will implement e ps effect a solution (for example.
:22tit-B;EP;521111:ZZ220:22LtJbre
followlng two methods are used extensively foronedimensional mimization. They
handlc the minimization problem directly.
5.2 SOLUTION TO THE PROBLEM 211
5.2.4 Polynomlal Approxlmatlon
The m eod is simplc in concept. Instead of minimizing a difficult function of one
variable minimize a polynomial at approximates the function. Thc optimal vaIue of
e variabl minimizes the polynomial is thcn considcred to approximatc the
optimal vaJuc of e variable for t1Je originaI function. It is rare for t1Je dcgree of t1Je
approximating polynomiaJ to excecd threc. A quadratic approximation is standard
less e ird degree is w ranted. It is c1ear that scrious errors in approximation are
expected ife polynomial is to simulate the behavior ofe original function ovcr a
l.ge range of vaIues of variable. Mathematical t1Jeorems exist justify a
quadratic representation of t1Je function with a prescribed degree of error witIJin a
smaJl neighborhood of t1Je minimum. What is ensures is at t1Je polynomial
approximation gets better as e minimum is being approach. Example 5.1.
repfuced for conveniepce is used for iI1ustration of the procedure.
Example5.1
Minimize f(.)=(_ 1)2(- 2)(3) (5.1)
Subjcct to: 0 S S 4 (5.2)
A quadratic polynomiaJ P() is used for the approximation. T his polynomial is
expressed as
P()=bo+b+ b
2
2
(5.9)
T w o elcments n d to understood prior to t1Je fol1owing discussion. le first
conccms e evaluation of t1Je polynomi. and t1Je second concerns e inclusion of
expssion (5.2) in e discussion. Thc polynomiaJ is completely defined if the
coefficients b
o
b. and b
2
arc known. T o determine m. t1Jree data points [(1'/.)
(.2'!z). (.313)] arc generated from Equation (5.1). This sets up a I incar system ofthree
equations in e unknowns by quiring t1Je vaJucs of t1Je function and the
polynomiaJ must be the same at t1Je t1Jrce points. T he solution to t1Jis system of
equations is values of t1Jc coefficients. 1 consideration of expression (5.2)
depends on t1Je type of one-dimensionaJ problcm being solved. If the one-dimcnsionaI
problem is a genuine single-variable design problemen expression (5.2) needs to
be present. If e one-dimensionaI problem is a subproblem m e
muJtidimensional optimization problem t1Jen expression (5.2) is not available. In that
c e a scanning proccdure is used to defin1and 3
Scannlng Procedure: le pris started from lower I imit for. A vaJue
ofzero for is vaJue be justified since it refers to vaJues current iteration.
A constant intcrval for 6.is also identified. For a w el1-scaled problem t1Jis vaIue
is usual1y 1. Starting at t1Je lower I imit intcrvaI is doubled until tlJree points are
.
213
5.2 SOLUTON TO THE PROBLEM







.




t














r

NUMERCAL TECHNQUES-THE ONE.OMENSONAL PROBLEM
detennined such at the minimum is bracketed between them. W i respect to
Example 5. 1 the scanning procedure generat the following values:
212
10
=0; / = 6 = 0; j{O) = 6;
B
= 1; = 0 = 1; j{l) = 0;
this cannot be the minimum is n01 yet pped = 2; j(2) = 0;
a

2

/2 = 18
A process such as that illustratd is essential sS it is both indifferent to the problem
being solved and can be programmed sily. The important requirement of any such
proc s is to ensuat the minimum lies between e Iimits established by the
procedure. lis procedure is developed as M ATI.AB m-fi)e UpperBound_1Var.m.
T he set of I inear equations to estabIish the coefficients of the polynomial in
Equation (5.9) is
6 = b
o
+ b(O) + b
2
(0) (5.108)
=4; = 4; j(4) = 18;
(5.lOb) 0::: b
o
+ b(l) + b
2
(l)
4 3.5 3
2

1.5
0.5
(5.lOc) 18 = b
o
+ b(4) + b
2
(16)
polynomial approm a1ion.
de development is termed a block structured approach-In this way-largcode
development is possible usmgsmallerpieces ofexisting code.
Rgure5.3
(5.1 1)
Using M ATI.ABesohon to e equations is
ho=6.0; b =-9; b
2
= 3
IC approximate solution to Equation (5.1) is 1.5. Figu 5.3 dcribes the polynomial
approximation procedure. lt is clearat the approximation leaves much to be desired.
O n the other hand if the datn for the polynomial coefficients were around a smaller
region near= 2.6 then results would be more impressive. Note at ram
usc a higher-order polynomial a better set of data is p.
The scanning procedure and e polynomial approximation techniques will find
significant use exploring multidimensional optimization problems. The stegy for
devcloping MATLAB codc from now on must include e possibility of code reuse. le
same code segments can be used in many dierent numerical techniques. Such generic
upperBound1Var.m:This codcsegment implements e detcnnination of e
upperboundoilhE on of one variable. The input to the function is the ncofthc
:
c o t ZZ 3 ; 4 2 ; ; ; 2 2 d E m u s t h a M B m -fi)ein esame directory
tZTftU3231:;:;Z?5512-twhich thc function
P(.}=6-9+32
d P
= - 9 + 6 = 0 ; :== 1.5
0 r
and the polynomial is
The minimum for the polynomial is the solution to
(5.12)
cod segment implements e polynomial
function of one variabl This function uscs
upperBound_1Var('EXam
p
le5_1' 0 1 10)
PolyApprox_1 Var.m: lis
approximation method for a
Usage:
214 NUMEAICAL TECHNIQUE5 -THE ONE.OIMENSIONAL PAOBLEM
UpperBound_1 Var.m to detennine the range of e variable. T he input to e
function is the name of the function; the order (2 or 3) of the approximation;
lowbound-the start valuc of the scan passed to UpperBound_1Var.m;
intvlstep scanning interval passed to UpperBound_1 Var.m; inials
number of scanning steps passed to UpperBound_1Var.m. The output of e
program is a vector of two values. The first element of the vector is the location of the
minimum of the approximating polynomial and the second is the function value at
this location.
T he function ferencedby ecode must be a M A TI.AB m filein the same directory
(Examp5_1.m). 1e input for Example5_1 is the value at which the function
needs to be computed. and its output is the value of the function.
Usagc: Value PolyApprox_1Var(' Example5_1' 2 0 110)(5.13)
The two programs use switch/case and the /eval statements from MATLAB. The
code also ilIusates calling and retuming from othernctions.
5.2.5 Golden Section Method
Thmethod of golden section is thc crcam of the family of interval ducing methods
(for example. e bisection method). It reduces e interval by thc same fraction with
each iteration. The intervals e derived from the golden seclion ratio. 1.61803. This
ratio has significanceasthetics as well as mathematics [34]. The method is simple
to implement . It is indifferent to e shape and continuity properties of e function
being minimized. Most important Lhe number of iterations to achieve a presibed
tolerance can bc established before Lhe iterations start.
Algorithmfor Golden Sectioll Method (AS.3)
Step 1. Choose low
= 0.38197 (from Golden Ratio)
e = tolerance = (L)Onal/(up_low)
N = number of iterations = -2.078 ln e
Sp2.1 =(1 -)low+ ;fl =/(1)
=ri
O W
+ (l)U
P
;f2 =)
nc points are eqdistant from bounds
Step 3. If ( i < N )
If/I >/2
10" .;1 2;fi 12
~ = I o w + (1 -)= / ( )
ii + 1
G o T o Step 3
Ifh>/.
Il' J
5.2 SOLUTION TO THE PROBLEM 215
up /:2;1;12 A
1 = (1 - 't)a!ow + ;/1 =/(.)
; r i + 1
G o T o Step 3
s iiiZ!
only callatee values. T he algo m is implemented in a different file.
ExampleS.l
(5.1)
Minimize /()=(_ 1)2(-2)(-3)
Golden Section Method
St 1. (1.10" = o.o;low = 6;u
P
=4;!"'= 18
L<Xj!nol = 0.001; E = (0.00114); N = 17
Step 2.. = 1.5279;/. = 0.1937; = 2.4721;12= 0.5401
F1gure54 i1lustrates th layout of information at this stage.It can bCVErind at
..2 are located symmically withmspect to outer bounds dO"p.
Step 3. i = 1
/1>12
(1.lolV = 1.5279;low = 0.1937; = 2.4721;/. = -0.5401
2 = 3.0557;12 = 0.2486
d
i
j
j
Y
3
5
2
3
3

i=2
Step 3. continued...
12>13
a.up = 3.0557; =0.2486;2 = 2.4721;= - 0.5401
1 = 2.1115;12 =0.1224
i=3
G o To Step 3 etc
In each itcration Lhere is only one ofcallation to evaaevariable and
value ofthe function.The rest ofthEComputauonts masSIgnsng extsting informatIon.
The differcnou bomds isalsochangingwitheach kmuon-hExmpleal
Start: L=4.0
Iteration 1: L= 4.0 - 1.5279 = 2.4721
217
vector of four pairs of variable and function values after the final iteration. With a
small tolerance y one of lhese pairs provides minimum location of the function.
It is recommended at the second pair be accessed for this puose. 1e function
referenced by the code must be a MATLAB m-file in the same dictory
(Example5_1.m). The inpul for Example5_1.m is the value al which the function
needs to be computed and its output is the value of the function.
U .ge:
Value GoldSection_1Var (' ExampleS_1' 0.001 0110) (5.14)
There are no new commands. Thc organizing and displaying of information in the
Command window is worth noting. If this code is part of a larger program the
printing of information in the command window needs to be switched off. It c be
med on for debugging.
Compar/son wlth Polynom/al Approxlmatlon: T he two popular methods for
miniouzing a function of one variable are polynomial approximation or golden
section. The algorilhms for polynooual approximation and golden section have
significant differences. T he former is a one-shot approach and is accompanied by
significant error in the estimation of the minimum which will improve as the data for
approximation get better. Implied in the implementation of the polynomial approach
is the continuity ofe function. T he golden section method on the other hand is
iterative technique. The number of iterations depends on etolerancexpected in e
final sult and is known prior to startof the iterations-a signifi nt improvement
relative to the Newlon-Raphson method where number of iterations cannot be
predicted a priori. The implementation is simple and the results impressive as it is able
tohomeinon oun1um. It is indifferent to the nature of the function.
There is no reason why the two cannot be combined.be golden section can be
used to establish four data points with a reasonable tolerance (instead of a low value)
and a cubic polynooual can be fit 10 identify the approximate location of minimum.
1 one-dimensional subproblem in a multivariable optimization problem is
employed for e determination of the stepsize after thc search direction has been
identified. It is easier 10 undcrstand these lerms by recognizing the generic algo m
for unconstrain optiouzation. In order to focus the discussion rather than e
general objcctive function representation a specific one Example 5.2 is introduced.
I M P O R T A N C E O F T H E ONE-DIMENSIONAL P R O B L E M 5.3
(5.15)
f(xlX
2
'X
3
) = (xl )2 + 2 { ~ - x
3
)2 + 3(x
3
- 1)2
Example5.2
Minimize
5.3 IMPOATANCE OF THE ONE.DIMENSIONAL PAOBLEM
5
up
' t
'
E
a
' J
s
NUMEAICAL TECHNIQUES -THE ONE.DIMENSIONAL PAOBLEM
20
18
18
216
14
12
10
8
6
{
6
}
h

4 3
Golden 5ectlon-starting valuas.
d = 3.0557 - 15279 = 1.5278
2

Flgur5.4
Iteration 2:

4
-2
-1
2

The final iteration is reproduced by running the following code:

iteration 17
2.6395
-0.6197
2.6406
-0.6197
2.6402
-0.6197
2.6399
-0.6197
le lolerance2 I in the final iteration is about 0.001 as expecled.
GoldSection_1Var.m: The codc lr slates e algorithm for the golden s tion
m od(A 5.3) inTLABcode. The input 10 the function is ename of thc function
tnCtname) whose minimum is being sought; etolcrance (tol) of the approximation;
the value (low bo und) of e s c paed to U p perB o und_1Va rr.m m n1;

scanmnginterv(u t 1) pa55edtoUp perBou md_1Va r'.m;th e m u mE

ps11 p to UpperBound_1 V .m. T he output of program is ;
218 NUMERICAL TECHNIQUE5 -T H E ONE-DIMENSIONAL P R O B L E M
A cory glance at Example 5.2 indicates atthesolutionisatxl = l;x2= I;X3= 1;
and the minimum value offis O. T hree variables are chosen because w e arnotgoing
to use any graphics for illustration (it is essential to follow the vector algebra).
Generlc Algorlthm (A5.4): The generic algorithm is an iterative one and is also
refer d t o a search algorithm. as the iterations takplace by moving along a search
direction. These search directions can be determined in several ways (Chapter 6). T he
algorithm without any convergencelstopping criteria can be expressed as
Step 1. Choose X
o
Step 2. For each iteration i
Determine search direction SJ vector
Step 3. Calculate sX; =jSj
Note: sX; is now a function of the scalar clj as Sj is known from Step 2
clj is called the step size as it establishes the length of s Xj
X j 1 = X / + s X
i
is determined by Minimizing F(Xi+l)
Iis is referred to as one-dimnsional step size computation
ii + 1; 0 0 T o Step 2
Applicalion 0/ Generic A hm(A5.4) EXIlITlple 5.2
~ [ g }

..

n
u
x

-
-
-

a
u
r
J

e
t

o

t

n

9
u
w
M

V
e
b

1
I
l
l
1
t
1

1
0
0

d
r
l
l
M
L

E
B
E
B
E
E
-
-
E
E
E
J

.
0
0
6
.
g
r
i
l
l
-
L

u
.
n

J
L
z
r
E

-
'
E
E
E
E
E
L
-

x
U
1
1
1
1
1
1
1
1

M
M
M

F
1
1
1
1
1
1
1
L
=

=
'
J

g
u
A
U

3
n
r
n
r

e
e

-
-
p
a
n
a

f(X1) = f(1) = (0 - 0)2 + 2(0 - 61)2 + 3(61 1)2 = 72l+3(6J - 1)2
Minimizing j{1)c value of 1 = 0.1. This is a one-dimensional
optimization problem.
While the solution above was obtained analytically because it was a simple linear
exp ssione polynomial approximation or the golden section method could have
been used instad.
5.4 AODITIONAL EXAMPLES 219
X l = l L
(5.16)
and so on.
5.4 ADDITIONAL E X A M P L E S
h is section additional examples of single-variable optimization problems arc
explored. T he available code for the polynomial approximation and golden section
should help solve any and all single-variable optimization problems. T his is largely
left to the reader to exploit . In this section only extensions modifications or creative
appIications of the one-dimensional optimization problem ar considered. First
Example 5.2 is revisited to modify the golden section method so at it can be used to
lculate Spsize in multidimensional problems. Example 5.3 is a solution to the
Blassius problem. lis is an example of the exact solution to e NavierStokes
uation for flow over a flat platc. Example 5.4 is an examination of the inequality
constraint in Examplc 5.la by the golden section method so that equality and
inequality constraints can be handled.
5.4.1 Example 5 . 2 -l Ilustratlon of General Golden Sectlon Method
'he example from e previous section is visited again to modify e golden section
method to dtcrmine e stepsize for multidimensional problem. is book is also
about getting MATLAB to work for you. In a scnse ssubsection is largely an exercise
in developing the codcwill see considerable use in the next chapter. A nuivalent
development with respect to e polynomial approximation method is left as an
exercise for the student.
'he goldcn section method requires the establishment of the upper bound.
'hreforetwo functions w il1 need to be changed to handle dcsign vectors and the
search direction vectors. The code is available UpperBound_nVar.m d
GoldSectlon_nVar.m. T he modifications are nOl very challenging and can be
inferred by comparing the usage ofe two functions.
For the single variable:
Usage: UpperBound_1Var (' Example5_1' 0 1 10) (5.12)
For many variables:
U ge:UpperBound_nVar('Example5_2' x s Ol 0) (5.17)
x: current position vector o r design vector
s: prescribed search direction
220 NUMERICAL TECHNIOUE5--THE ONE-DIMENSIONAL PROBLEM
O f course Example5_2.m retums the valuof e function corresponding to a
vector of design variables. lis implies the function call to Example5_2.m can
only take placafter the design variables are evaluated from e value of stepsize
applied along e prescribed search di ction from the current location ofe design.
The usage of GoldSection_nVar.m must accommodate function (5.17) and e
need to deal with a design vector. Comparing with the single-variable case
U ge:
V a l u e GoldSection_1Var('Example5_1' 0.001 0 1 10)
(5.14)
U ge:
V a l u e GoldSection_nVar(' E xample5_2' 0.001 x s 0 1 10)
(5.18)
Visit code to s thed15.
Compar/son w/th Sectlon 5.3: The code GoldSectlon_nVar.m was run from
e Command window using the fol1owing listing:
> > x [0 0 0 ];
> > S [ 0 0 6 ] ;
>> GoldSection_nVar('Example5_2' 0.001 x s 0 1 10)
After 11 iterations the final Jine in the Command window was
a n s CI
0.1000 1.2000
T he above is interpreted as:
o o 0.5973
1 = 0.1000; f(1) = 1.2 x
1
=0; A2 =0.0;
which matches Section 5.3 and Equation (5.16).
x
3
= 0.5973 (5.19)
5.4.2 Example 5.3 -Two-Polnt Boundary Value Problem
A rea1 example for one-variable optimization can be found in e numerical solution
of the laminar flow over a flat plate [5]. le problem usually attributd to Blassius
represents an example of the exact solution of the formidable Navier-Stokes cquation.
le mathematical foml Ulation allowing for similarity solutions is exp ssed by a
third-order nonlinear ordinary differentia1 equation with boundary conditions
sp ified at two poi - a two-point boundary value problemPBVP). W hat
folIows is the essentia1 mathematical description. le inte tedreader can follow the
details in most books on fluid mechanics inc1uding suggested refe nce.
5.4 ADDITIONAL EXAMPLES 221
Mathematlcal Formulatlon
ff" + 2f" = 0
x=O; f(O)= 0; f(O)=O
x = f( } = 1
T he solution nondimensional velocity in the boundary layer is obtained as
=f(x)
(5.20)
(5.21a)
(5.21b)
(5.22)
T he solution to Equations (5.20) to (5.21) is largely through special iterative
techniqucs. These techniques use a numerical integration procedure l i ~ e
Runge-Kutta method to integrate the system by guessing and adjusting the missing
boundary conditions at the initia1 point such the final point boundary conditions
are rea1ized. This is due to e fact that the integration procedu s usually solve initial
value problems only. In the Blassius problemis would imply the missing initial
value can be regarded a design variable.
('(0)=
The objective function therefore will be
(5.23)
Minimize: () 1)2 (5.24)
Implied in the formulation is lf() is obtained by integrating the direntialuations
(5.20) (e can be considered as differential cons'ain) . lere is one othcr
consideration to take into account.le integration methods require the ublem to bc
expressed in state spa formwhich r n- ord differentia 1 uation 10 be
expreed as a system of n l-ordcruations. le conversion is fairly standard and is
done through ul Sformation and inu ucing additional vles. For this example:
y= f
y ~ = f = Y 2
Y;=f' =Y3
Y;=f" ='/2
T he optimization can be formulated : Find
Minimize: [Y2() lf
using the Runge-Kutta method (or y other method) to integrale
(5.25)
222 NUMERICAL TECHNIQUE8 -THE ONE.DIMENSIONAL PROBLEM
-
E
E
E
E
E
E
E
E
E
E
E

3 v
d

E
J
U
J
U
J
g
J
I

A
U

r
i
-
-
I
r

-
-
J
I
1
1
1
1
1
J

r

E
E
E
E
E
E
a
E
E
E
E
E
J
o
o

r
E
E

-
E

-
E
L
-
-
1
B
E
E
t
-
E
E
E
B
E
E
E
E
J

)
)
)

n
v
n
U
4

v
i

(
(
(

-
a

'
-

f
w
l
w
l
v

(5.26)
From published work [5J the value of=0.33206. Inslead of e nal point at it
is considered 10 be 5.0. Leaving is value as a variable can define a corresponding
two-variable optimization problem. In e implementation to followe power d
convenience of M A1 L AB ar readilyapparent
Example5_3.m: This m-file is lIsed to implement expression (5.25). It calls the
MATLAB built-in function ode45 10 use the RungeKutta 4/5 order method. The
function ode45 requires the state equations to be available in another m-file. It a1so
Euires e definition of the interval of integration as well as the initial condition. T he
following line from the coddcscribes the usage (e code is w elI commented too)
[t y ]
ode45('Ex5 3 state' tintval bcinit);
t is the independenl variable and y the dependent vector.
Ex5_3_state.m: The state equations in (5.26) are available in is file.
stst [ y (2) 0.5*y(1)*y(3)]' ; y(3)
Refer to lhe code for details. GoldSectlon_1 Var.m is used from the command line
for is problem as
U ge:
GoldSection_1Var('Example5_3' 0.001 0 0.1 10)
Start iteration:
start
alphal (low)
o
1.0000
alpha( )
0.1528
0.1904
alpha(2)
0.2472
0.0378
The final iteration:
iteration 12
alphal (low)
0.3359
0.0000
alpha(l)
0.3364
0.0000
alpha(2)
0.3367
0.0000
alpha(up)
0.4000
0.0161
alpha(up)
0.3371
0.0000
5.4 ADDITlONAL EXAMPLES 223
The result expected i s = 0.33206. However the tolerance specified was only
0.001. The initial bounding interval was only 0.4. T he golden section has
produced an impressive solution to e TPBVP. Also note the convenience of
modular pl'Ogrumming. N o changes w e made to UpperBound_1Var.m or
GoldSectlon_1 Var.m to run this exarnple.
5.4.3 Example 5.4 -Root Findlng with Golden Sectlon
T he application of the golden section method to the root finding problem is important
to accommodate cquality and inequality constraints. Consider Example 5.4:
g():0.752 _ 1.51 = 0
0 $ $ 4
(5.27)
(5.2)
is is e inequality cons int of Exarnple 5.la except tit is transformed to an
equality constmint . In is form thenecessary to make it into an active constrainl
can be established. T he simplest way to accomplish this is to convert the problem to
a minimization problem for whie numerical techniques have already bcen
cstablished. Squaring E quation (5.27) ensurcs that the minimum value would be zero
(if it exists). Therefore the solution 10 Equation (5.27) is the same as
M mize: ): [0.75a
2
- 1.5-1]2
(5.28)
which can be handled by GoldSection_1Var.m. W hile is was quick and painless
therear some attendant problems using this approach. the nonlin rity has
increased. T his is usually not encouraged in numerical invcstigations. Second the
number of solutions has satisfying F O C increased. Figure 5.5 repsents the functions
in Equations (5.27) and (5.28).
'
Solutlon UslngGoldSection_1Var.m: T he following discussion is based on
the experience of attempting to run thc golden section method with the lower bound
at the value of 0e case with e othcr examples. T he is a problem since the
solution cannot be found. T he problem app rs to be in the ca1culation of the upper
bound. ln many of the numerical techniques there is an assumption at the objective
function being dea1t with is modalat is has a single minimum in the region of
iaterest. For this to happen. it is assumed at e function being minimized w i1l
always decrease from the lower bound. Take a look at e functionfin Figure 5.5. It
is increasing at thc va1ue of zero. UpperBound_1Var.m assumes that e initia1
interva1 is too large and continues to bisect it to find a nction va1ue is less an
the one at the lowcr bound. This drives the intervaI to zero and the function is exited
with the lower bound and with no solution.
"
4
R e m e d y 1: The easiest way out of the difficulty is to changc the value of the lower
bound and try again. It is useful to undcrstand that problem is with the calculation
224 NUMERICAL TECHNIQUE5 -THE ONE-DIMENSIONAL PROBLEM
18
-
E
-
-
-
e
-
-
a

' a
-
-
a

'
'
o
a
-
-
-
-
-
a
-
-
v

J
-
M
h
e
-
e

.
16
14
12
10
(

(
H
M
)

s
'
p
-
t
'
-
-
e
4
solutlon
2
g
0.5 2.5 3 3.5
Flgure 5.5 Exampl 5.4.
of the upper bound (bracketing the minimum) raer than with golden section
method. Changing the lower bound to 1 and rerunning th golden section m eod
provides a successful sult which matches e graphical solution. While success was
realized. such a pr edure needs user intcrvention well as some knowledge about
thproblcm-in this case there was e graphical solution_ These are not acceptable
qualitifor an automatic proc lure.
R e m e d y 2: Sioce the positive slope at the lower bouod is indicative of the
problem. it should be possible to sense this positive sign 01 the slope and move the
lower bound to the point where the slope is negative. Such a point can be
discovered by the samscanning procedure. T hc slope computation is numerical
and is based on forward first difference. Oncc morc is change in the lower bound
is only needed for the uppcr bound calculation. UpperBound_m1Var.m is the
modified upper bound calculation which relocates the l o w ~ r bound to the point
where the slope is negative. The usage and information from running the
GoldSectlon_1 Var.m incorporating the chaoged upper bound calculatioo are
shown below. The samc usage with thc old UpperBound_1Var.m does oot
produce any useful rcsults.
4
PROBLEMS 225
U ge: GoldSection_1Var(' Example5_4' 0.001 0 .1 20)
start
alphal(low)
alpha(l)
alpha(2)
alpha(up)
0.9931
1. 6069
2.6000
1.0000
3.0624
2.1720
0.0289
iteration 16
alphal (low)
alpha(l)
alpha(2)
alpha(up)
2.5266
2.5270
2.5273
2.5278
0.0000
0.0000
0.0000
0.0000
R E F E R E N C E S
1. Burden R. L.and Faires J. D. Numerical Analysis. 4th ed. PWS_KENT Publishing
Company Boston 1989.
2. Hosleller G H..samina-M.s..and MontaIOU-p.D.AnalyticalNwnericdd
computational Methodshrscience and Engineeringprmticc-Hall-Enslewood Cliffs.
NJ.1991.
3. V 1 G. N.. N u Oplil 11011 Teclmiq for Engineering Desigll
McGraw-Hill New York. 1984.
4. Huntlcy. H. E. The DlvIlle Proporlion: A Sludy in Muthemalir :al Beauty. Dover
Publicauons New York. 1970.
5. Schlichung. H. Bounda-Layer Theory McGraw-HiU. Ncw Y o 1979.
P R O B L E M S
5.1 Extend Example 5.1 to includc two inequality constraints for which a solution
exists-Display fu l (
5.2 Extend Example 5 .1 to incIude two i nuaality constraints for which t herl1s no
solution. Display the function graphica11y.
5.3 Reprogram e code Sec5_2_2.mobtain e values of f and ct' through
MA'1.AB function programs d apply to Example 5.1.
54 22J;;13222:l:::Z347tt
5.5 Identify the value of where e constrnt (5.3) bcomes active using
Ncwton-Raphsoo technique.
5.6 Implemcnt a bisection procedure for the rninimization of e function of onc
variable. Apply it to solve Example 5.1.
5.7 For the polynornia1 apprOldmation implementation cxtend c code to display
e original funcon d e approximating polynomial.
5.8 For c golden section implcmentation extend the codc to display the original
function and thc anal external bounds on the variable.
226
NUMERICAL TECHNIQUE5 -T H E ONE.DIMENSIONAL P R O B L E M
5.9
Verirat e bounds on the variable are decreasing by e same ratio. W hy is
this better an the bisection method?
5.10 H o w would you set up the golden section method to detrmine the zero of
function instead of the minimum.
5.11 Combine e golden section d e polynomial approximation to find the
m m i m u m o fe function.
5.12 Improvthe accuracy of the solution to the BI ius 1over that presented
in the text .
5.13 Obtain the solution to
approximation method.
the Blassius quatlOn usmg the polynomial
6
N U M E R I C A L T E C H N I Q U E S
F O R U N C O N S T R A I N E D
OPTIMIZATION
This chapter iIlusates many numerica1 techniques for muJtivariable unconstrained
optimization. While unconstrained opation is not a common occurrenc in
engineering design nevertheless the numerica1 techniques incIuded here demonstrate
interesideas and also capture some of the early intensive work inearea of design
optimi tion [1]. T hey aIso provide the means to solve constrained problems after they
have been transformed into an unconstrained one (indirect m ods Chapter7).
This book is unique for the presentation of nongradient techniques (Section 6.2)
which can be empowered by the ubiquitous availability of incrdible desktop
computing power. Until very recently. application of design optimization required
mainframe computers that relied on large programs written in FORTRAN. Today's
desktop can run programs atreqre only Iimited programming skills. The desire for
globaI optimization has brought into focus numericaI techniques at are largely
heuristic have limited need for sophisticated gradient-based methods require limited
programming skiHs. and require limited programming resources. Usually such
techniques require the solution space be scanned using a large number of
iterations. T hese methods can ideally be run on personaI desktops for unlimited time.
T he press of limited computing resources is no longer a significant constraint on
these techniques. While they may compromise on elegance and sophistication it is
important to recognize opportunity to engage desktop resources for greater
usefulness.
6.1 P R O B L E M DEF1NITION
The unconsained optimization problem requires only the objective function. The
book has chosen to emphasize an accompanying set of side constraints to restrict
227
228 NUMERICAl TECHNIQUES F O A UNCONSTRAINED OPTIMIZATION
solution to acceptable iacel gion. Most numerical iqu in S
chaptEr agpoe ?ns ints in I ng out method. It i s u su dIy
e
Ponsib iJity 0 fth 1e d eto ver t he dc co nlIInts as part of his exploration
ofoptimum. is checking ofcsidensntcan be easily programmed in the
code. T hus the problem for is chapter can be defined as
Minimize f(X); ] "
Subjectto: xf $; XI $; xr; i = 12 n
6.1.1 Examp'e 6.1
; ; 2 3 1 2 : : z z ; 2 2 2 Z 2 ; 2 J Z e mrpossible hicaJ descriptior
Minimize
f(X) =f(xlX2) = 3 + (XI - 1.5x2l
2
+ (X2 _ 2)2
Subject to:
O S X ( S5;
OSx2 S 5
(6.4)
Figure 6.1 provides the contours of e problem and e)lution can be located at
fJi Z L 2 ; a r v a l u e i s f = 3 n adratic he solution
6.1.2 Necessary and Sufflclent Condltlons
2;trzz;::2jr2zs;tt:!sZ;:?
conditions) are e gradlents must vanish at the solution
21=2(Xt-15) = 0
1) + 2(X2 - 2) = 0
(6.5a)
(6.5b)
Equons (6.5) cb e v e d to obtain xj = 3xi= 2. T his matches e graph
2iljitM223iC :tt; s z
=l
(6.6)
(6.1)
(6.2)
(6.3)
6.1 P A O B L E M O EN ITlO N 229

'jJ
T
I
l
!
l
l

'
l
l
i

4
4
w
h
H
M
l
'
I

ta--
0.5 2.5
X
1
5 3 1.5 2 3.5 4 4.5
Flguro 6.1 Example 6.1.
The Hessian is positive definite considering the detennants alone. Sec6_1_1.m
'
provides Ede for the symbolic calculation and also displays g phi ldcription
the problem. T he results ofecalculation arc displayed in the command window.le
satisfaction of the S O C identifies solution for the design is a minimum.
6.1.3 Elements of a Numerlcal Technlque
'he elements of a typi 1num1techniquc or algorithm can bc associated with the
generic aorin induced during e discussion of the relevance of the
one-dimensiona1 opnization for multivariable problems (Chapt 5). us algorithm is
iterative and is also referred to as a search algoritlvn as the iterations take place by moving
along a search direction or a search vectorduring an iteration. T heshdirections can
bedetem.ed in many ways. T he different hniques at are p sented in this chapter
primarily differhow thc search dir tion istablished. le maining elements of the
algorim ex pt for convergenccJstopping criteria are the same for almost all of
the methods. T he aJgorithm can be exprl scd as follows.
IFileslO be downlonded rrom Ihc web si nre indicaled by boldrace sons serir Iype.
230 NUMERICA l TECHNIOUES FOUNCONSTRAINEO OPTIMIZATION
General Algorithm a
Step 1. C hse X
o
Stcp 2. For each iteration i
Determine search direction vector 8j
(lt would be nice if the objective decreased along this direction)
Stcp 3. Calculate sX=si
Note: a Xj is now a function of the scaIar <<j as S is known from Step 2
is cal1ed e step size as ittablishes e length o f A Xj
j is dtermined by Minimizingf(whcre X'+I = X j + X 1
Here the convergcnce criteria (FOC/SOC) and the stopping critcria (Is
design changing? Exceeding iteration count? etc.) must be checked.
ii + 1;
O o T o Step 2
ln this chapter this algorithmic structure will be uscd for aIl of the methods.'he
one-dimensional optimization w il1 be implemented using the golden section metbod.
Example 6.1 w il be used to dcvelop thc differcnt numerical methods.
6.2 N U M E R I C A L T E C H N I Q U Es - NO N G R A O I E N T M E T H O O S
'hese methods are aJso caJled zero-order methods.'he order refcrs to e ordcr of thc
derivative of the objcctive function necded to establish the search direction during
any iteration. Zero order signifies that no derivatives used [2] which in tum
implies only function values are used to Sblish tbe search vector. There is
another important significance that accompanies tbe missing derivative
computation-the F O C should not be applied in these methods. If w c were
computing derivativcs w e might as well incude this information for establishing the
search vector. Only the changes in e objectivc function or the design variables can
providconvergence d /or stopping criteria.
e tcchniques are included in this section. The first is a heuristic one based on
random search directions. The sccond one cycles through a search dircction based on
each varlablc. le last is Powcll's method which has tbc property of quadratic
convergence.
6.2.1 R a n d o m Wa lk
T he search direction during each iteration is a random direction. Random numbers and
sets of random numbers are usualavai1able through software. MATLAB includes the
abty to generate severaJ typcs of random numbers and matrices. Most
computer-generated random numbers arcalledpseudorandom numbers as they cycle
after a suffjcient number of them havbeen created. The onedimensional stcpsize
computation is donc by thc GoldSectlon_nVar.m nction (Chapter 5).
62 NUMERICAI.TECHNIOUE- NONGAAOIENT METHOOS 231

H
i
l
A orlthm: R a n d o m Walk (A6.2)
Step 1. Choose X
o

N (number of iteratlons)
Set i = 1
Step 2. For each iteration I
8
1
= Random vector
Step 3. X j+
1
= X j + Sj
'j is determined by minimizingj{X i+I)
ii+ 1
lf i < N 0 0 T o Step 2
elsc S P (irations exceeded)
EZ2222zteszz:;12rz::;;i
l;f:;
sign based on a test of random number.
R a n d o m W a l km : T l m r a n d o m walk algorithm tA62)is impiementcd in this
OP6miadon-TMcodeof the random walk function also includes thE following
1'1
I ~
~ t
e contour of thc objective is drawn.
For two-variable problems tn
FortwovariablEproblems the successful iterations M e tracked on the contour
d the iteration" number identified.
. 2 ; : u 1 6 v md10 C d for
the one-dimensional stepsize computation.
. T he multivariable uprbound calculation (developed in chaptcr5)is used for
. ;?::;E :::2:zr1;z;;11iIn1 here 0 h d di iI1 r ec
.:::i : :;:e L ign variables and ctions for all of
thc itcrations arc printed.
232 NUMERICAl TECHNIOUES FOR UNCONSTRAlNED opnMIZATION
Use of previous code iIIusates code reuse and modular programming. Once more
r ders ar strongly encouraged 10 visit the code to understand thanslation of the
aJgorilJlm and subsequent application.
U ge:

Z
RandomWalk('Example6_1' [0.5 0.5] 2 0 0.001 0 1 20) (6.7) . ~

Input: (They are thitcms enclosed within the outermost parenthesis in expression
(6.7).) a
'Example6_1:
[0.50.5]
20
0.001
0
20
lC file Example6_1.m where lJle objective is described
starting dcsign veclor (dvarO)
number of iterations of the method (niter)
tolerance for e golden section proced e (tol)
initial stepsize for scanning upper bound (Iowbound)
slep interval for scanning (intvI)
number of scanning steps (naIs)
Output:
A vector of n + 1 vaJues of design variables at the last iteration and the vaJue of the
objectivc function.
Executing function (6.7) in lJle Command window produces the following
information (e comments are added later):
- T h e design vector a n d function during the iterations
0
0
1
1
1
1
2
2
8
8
4
4
4
4
4
4

0
0
9
9
9
9
8
8
0
3
0
0
0
0
0
0

u
n
u
n
u
n
u
n
u
n
u
n

3
n
E
n
o
n
o
n
u
n
D
n
D

=
R
J
R
d

'

J
n
u
n
v
n
u
n
u
n
u
n
u
n
u
....
4

m
0
0
0
0
0
0
0
0
2
3
3
3
3
3
3
3

a

0.5000
0.5000
1.0440
1.0440
1.0440
1.0440
0.9756
0.9756
2.1093
2.0481
2.0350
2.0350
2.0350
2.0350
2.0350
2.0350
5.3125
5.3125
4.6481
4.6481
4.6481
4.6481
4.6349
4.6349
3.0663
3.0025
3.0020
3.0020
3.0020
3.0020
3.0020
3.0020
start
d i d n o t move
decreased function
d i d n o t move
d i d not move
d i d n o t move
decreased function
etc
6.2 NUMERICAL TECHNIOUES -NONGRADIENT METHODS 233
3.0804
3.0804
3.0804
3.0804
ans
3.0804
2.0350
2.0350
2.0350
2.0350
3.0020
3.0020
3.0020
3.0020 20 iterations
2.0350 3.0020
Compare vaIu t o e actual ution of xi=3xi=2j* =3. Ts is
remarkhle for a method at has no sense of preferred direction. If the execution 15
resated with Menucalinputm entimnew solution will beobtained.Figure62tracks
th; Iges ofthe d n vari s on the co lf plot. The m o v e mnt of ilie sign
variables are paired with the corzesponding iteration number.Therandom walk
function can bEExECUted for several diffErent starting sets of design variables.It
should not disappoint. T he GoldSection_nVar.m is e sarne one as in Chapler 5
except ilie printing has been suppressed.
While executing e RandomWalk.m the printing of iteration m b ?
cmates a mess as you approach the solution.You can switch it offby commenttqgme
()statement-Thgeodein Randomwalk-m also seIves as a templateforother
algoriilims. lt is worth exploring in detail.
k
0.5
1.5
Flgure 6.2 Random walk: Example 6.1.
i
j
L
i
l
i
-
-
1
1
J
i
l
l

234 NUMEAICAl TECHNIOUES FOUNCONSTAAlNED OPTIMIZA TION

6.2.2 Pattern Search
The Pattem Search method is a minor modification 10 lhe Utlivariate method with a
maJor Impact . As the Pattem Search uses the Univariate method the latter is not
discussed separately. In the Univariate method a1so known the Cyclic Coordinale
Descenl method each design variable (considered a coordina) provides a search
direction. This is a1so refeed to as a coordinate directiond is easily expressed
through t h1e un 1
nu m br o fvariabless
desig n variables) i teratio ns onefor achofthe n search directions. Itcan be shown bv
appIication tfor problems with considerable nonlineathe Univadatemet130J
tends to get lock intoazigz agp attEemofsm a ma
soluJt 110 n.
llePatmSearchprc dureattempts to disrupt this zigzag bebavior by executing one
additional iteration for each cycle. In each cycle at eend of 11 Univariate directions the
n + l s e ah direction is assembl1 as a I II n 1 arc o m 11b
dcons d e optimum ue of t hle p ze fo at direction. one-dim ional
optimal stepsize is en mputed d the next cycle of iteration begIl1s.
A orithm: Pattern Search 6.3)
Step 1. Choose X
1
N
c
(number of cycIes)
j (1) =j(X) ;X
c
(1) = X
E E2: tolerance for stopping criteria
Setj = I (initialize cycle count)
Step 2. For each cycle j
Fori= 111
S;=(univariate p)
X;+ = X ; +
/ is determined by minimizingj(X;+)
(store values of.i and S/)
cnd of For loop
n
= :;S;aX - X (Pattcm
X j = Xn+ + CiSj
X + 1) X s J"+ 1) =j() (storcycle values)
Step3. 6j=1rU + 1)-1rA X = X.u+ I ) - X
c
If l.1jl S E" stop
IfAX
T
.1x S ~ . stop
Ifj = N" stop
X ~
6.2 NUMEAICAl TECHNIOUES-ONGAAOIENT METHODS 235
j j + l
GoToStep 2
ThePatternSearchmethodhasafcwadditionalstoppingcritcria.Thcscarebasedon
how much thefunction isdecreasing for each cycleand h o w much change is taking
placein thcdesign var iableitself. For thelatter thisIl1formation isbased on thelength
ofthechangein thedesign vector for thecycle. Onceagain theapproach issimpleand
the implementation is direct. Programming such a technique will not bc too
forbidding.
PatternSearch.m: T o conlInue eexposure 10 M ATu.B programming ismethod
is implemented very different 1y from the other algorithms that have bcen translated
into code so fi. It is a stand-a1one program. Therc are prompts for er inputs wi
appropriate default values in cas the user decides not to enter a value. The
one-dimensional stepsize computation is implemented by e multivariable golden
section mcthod. The interesting featu sare as fol1ows.
. It is a freestanding program.
le file contaIl1ing the objective function is selected through a list box.
Scveral program control par neters are entered from the keyboard after a
suitable prompt indicating defllult values.
Dcfault values arc initialized if the user chooscs not 10 entcr the values.
Thc design vcctor is tracked and printcd aftcr thc program is stopped.
Since the upper bound calculations arc based on positivc valucs of stepsize thc
search direction is vcrs if tumed stepsize is e same as the lowcr
bound.
Iteration counts are recorded.
For two-variable problem the objective function contours are plotted.
. T he first nine or Icss iterations are traced on contourplot . Thc Univariateand
the pattem steps are co)or codcd.
U ge: PatternSearch (6.8)
The prob)em is Example 6.1.le fi]e is Example 6_1.m.
Output (MATl.AB Command window): With e default va)ues and the starting
vector [4 3] the following are written to the Command window.
> > PatternSearch
T h e function f o r w h i c h t h e m i n i m u m is sought m u s t b e a
MATwa function M - File. G i v e n a v e c t o r dependent variable
it m u s t return a scalar value. T h i s is the function t o b e
MINIMIZED. Please select function n a m e i n t h e dialog b o x
a n d h i t return:

236 NUMEAICAL TECHNIOUES FOA UNCONSTAAINED OPTIMIZATION

T h e function y o u have chosen i s ::Example6_1
maximum number o f cycles [1000]:
convergence tolerance for difference in f[le8] :
convergence tolerance o n change i n design x [le-8]:
Input the starting design vector. T h i s is mandatorv a s
there is n o defaul t vector setup. T h e length o f o u r
vector indicates the number o f unknowns. Please enter it
n o w a n d hit return:
[4 3]
T h e initial design v e c t o r [4.00 3.00]
Convergence in f: 3.999E 009 reached i n 10 iterations
N u m b e r o f useful calls t o the G o l d e n Section Search
Method: 31
Total number o f calls t o the Golden Section Search
Method: 40
T h e values f o r x a n d f a r e
4.0000
4.4992
4.4992
4.3868
4.1432
4.1432
3.8229
3.3338
3.3338
3.1750
3.1980
3.1980
3.1944
3.1473
3.1473
3.0346
2.9958
2.9958
2.9971
2.9972
2.9972
2.9972
2.9977
2.9977
2.9999
2.9998
3.0000
3.0000
2.6924
2.7616
2.7616
2.5287
2.2224
2.2224
2.1543
2.1321
2.1321
2.0916
2.0980
2.0980
2.0682
1. 9969
1.9969
1.9981
1. 9981
1. 9981
1.9986
1.9988
1.9988
1.9990
1.9999
1. 9999
4.2500
4.0000
3.6916
3.6398
3.5801
3.4022
3.2890
3.0494
3.0343
3.0180
3.0175
3.0121
3.0118
3.0096
3.0067
3.0016
3.0000
3.0000
3.0000
3.0000
3.0000
3.0000
3.0000
3.0000
3.0000
3.0000
0
0.4996
0.3081
0.2254
0.2446
0.2339
1. 3148
0.4896
0.0681
0.3248
0.0234
0.0405
0.1579
0.0471
0.0298
2.3916
0.0391
0.0013
0.0322
0.0001
0.0005
0.4563
0.0005
0.0002
4.9305
0.0002
6.2 NUMEAICAL TECHNIOUE9 - NONGAADIENT METHODS 237
2.9998
2.9998
2.9998
2.9998
2.9998
1.9999
1.9999
1.9999
1.9999
1.9999
i
t
i
l
l
-
-
0.0000
0.3398
0.0000
0.0000
0.2020
3.0000
3.0000
3.0000
3.0000
3.0000
To verify e program is executing the a1gorithmsc the design veclors E
first l Wo columns. The univariale steps can brecognized w hn one component of the
vec10r changes while other maintains its value. The pattern directions can be
;1. recognized when both components change.
~ : The number of iterations is a significant amount . One reason is that the tolerances
are much smaller. It can be noted that difference belween lwo ilerations ch ges
I inearly. Such methods are expec1ed to run for a large number of iterations. The
1 : program con01 parters control the acccy ofe results as well as e number
of iterations. The solution is more accuratthan the Random W a1k m eod.
Figure 6.3 iIIustrales e a1gorithm graphically. The univariate and the paUern
directions are quite distinguishable. W hen executed on your machine th steps are
explicitly color coded. Readers are encouraged 10 go over the code. and modify i1 to
E
4.5
0.5 1.5 3
Rgure 6.3 Pallem search: Example 6.1.
238 NUMEAICAL TECHNIQUES FOA UNCONSTAAINED OPTIMIZATION
refl tthirpreference. The number of lines is around 250. W e atrting to get into
senous programmmg.
6.2.3 Powell's Me'o d
lf there were only one zero-order method must be progranuned the
overwhelming choice would be Powel1's method [3]. The principal reason for e
decision would be that it has the property of qlladratic convergence namely for a
quadratic problem with n variables convergencwill be achieved in less than or equal
to n Powel1 cycIes. A quadratic problem is an unconsined minimization of a
function that is expressed as a ql.dratic polynomial-a polynomial with no term
having a degree greater than two. Example 6.1 is an example of a quadratic
polynomial in two variables. Engineering design optimization problems are rrely
described by a quadratic polynomial.sdoes not impl y that you cannot use Powel1' s
method. W hat this means is that the solution should not be expected to converge
quadratically. For nonquadratic problems as the solution is approached iteratively
e objective can be approximated very well by a quadratic function. It is at is stage
at thequadrc convergencproperty is reaJized in the computations.
The actual aJgorithm (A6.4) is a simple (the word is not being used Iightly)
modification to the Pattem Search algorithm. ln each cycJe (aer e first) instead
of using univariate directions in the first n iterations. the search directions are
obtained from the previous cycJe. The new search directions are obtained by I
shifting the directions of the previous cycJe. In is way a history of the previous
search directions is used to establish the next . With this change. Powell's method
should converge at most in three cycJes for Examplc 6.1 rather than the several
listed in the previous section.
A orithm: Powell's Method 6.4)
Step 1. Choose X" N
c
(number of cycles)
Jtl) = j{X
1
); Xc{l) = X
E E2: tolerance for stopping criteria
Setj = 1 (initialize Powell cycle count)
Fori= 1 n
S; (univariate step)
Step 2. For each . cycle j
Fori=l.n
If j 2S; S;+
X;+ =X;+ISI
; is detennined by minimizingJtX i+')
end of For loop
Sj=S;+ = I ;S; = X
n
+ -XJ (pattem step)
6 2 NUMEAICAL TECHNIQUEONGRADIENT METHODS 239

1
1

1
f
H
{
X f = Xn+ +j
X cU+ 1) Xf; fcU+ f(Xj) (s cycIe
Step 3. !:if= fcU + 1)U); d1{ = XcU + 1) - Xc(j)
lf ldfl S E" stop
lf d1{T d X S E2. stop
If j=N
c
sp
XX
J
j

j + l
GoToStep2
APPMeatton of poweII's Method:The translation of the algorithm into MATLAB
COdeisleft as m exercise for the student.Step-by-step applicaGon toExample 6.1 is
shown here. Example 6.1 is resdas
Example6.1
Minimize f(X) = f(x" X2) = 3 + (x -1.
5 X
2)2 + (x2 (6.3)

X E =
S =[6]; S2
(6.4)
j = l
Step 2. j = 1
X 2 4 3 1 + l = ? ? l
2. = 1/4; X 2 =l

4 4 4 6 1 5 ; X 4 z j ; ) =4.5576
S) = 0.25 [6]
X 4 = [ 0 3 2 5 1i5]
i = 2:
i=3:
241 6.3 NUMERICAL TECHNIQUEs - GRADIENT.BASED METHODS
i
l
i
l
l
i
t
-
-
-
In light of the previous definition. these would be referred to asfirst.order m ods.
The search directions will be constructed using the gradient of the objective function.
Since gradients are being computed. e Kuhn-Tucker conditions (FO C) for
unconst ned problems. Vf = O. can be used to check for convergence. T he S O C are
hardly ever applied. One of the reasons is at itwould involve the computation of an
n X 11 second derivative matrix which is considred computionally expensive.
particularly ifeevaluation of the 0tive function quiresa call to a finite element
method for generating requir informalion. Another reason for not calculating the
Hessian is at the exislence of e second derivative in a real design problem is not
certain even though it is computationally possible or feasible. For problems that can
be described by symbolic calculations. M A11.AB should be able to handle computation
of second derivativat the possible solution and i eigenvalues.
Without S O C these methods require user's vigilance to ensure the solution
obtained is a minimum rather an a maximum or a saddle point. A simplway 10
verify is is to perturb the objctive function through peurbation in thdesign
variables at the solution and verify it is a local minimum. This brings up an important
prope ofthesemeods-they on1y find local optimums. Usually this will be c10se
to th design where the iterations are begun. Before concluding the design
exploration. it is necess Y to execute the method from several starting points to
discover if other minimums xist and select the best one by head to head
comparison. T he bulk of existing unconstrained and constrained optimization
methods belong to is category.
Four mcthods are presented. The first is thc Steepest Descent method. Whilthis
method is not used in practice. it provides an excellent example for understanding thc
algorithmic principles for gradient-based techniques. The second is the Conjugate
Gradient technique which is a classical workhorse pacularly in indus usage. The
third and fourth belong to the category of Variable Metric methods. or Quasi-Newton
m eodsthey are also called. These methods havc been popfor some time. d
will possibly stay that way for a long time to come.
Thc gcneral problem and specific example arproduced for convenience
N U M E R I C A L T E C H N I Q U Es - GRADIENT-BASED M E T H O D S 6.3
240
NUMERICAL TECHNIQUES FOR UNCONSTRAINED OPTlMIZATlON
L=11:570;
f(X
4
) = 4.4843
= - 0.14111'' Y .
8558
1
1.2 = 11.01041; f(X
2
) = 4.4145
; = 0.4235;
Step 3. l.dfl = 1.353> EI' continue
ld X
T
d Xl = 1.8992 > Ez. continue
} = 2: One cycle over
GoToStep2
Step 2:}= 2
x J 0 . 8 5 5
J =11 1 c'7n 1 :
11.l570 l '

M
-
-
g
u

111 iZ] ; S 2 2=[0 =[0 0 22 5 J =0 4 0 3 5
i= 3:_
v _1 0.95671 r 0.1011 r'll
- U.1966 J = /2/; f(X
4
) = 3
::i ; ; 3 2 ! e m d C utations in iercise stopped
i t E : 2 7;:Jr:atJ;;::::would it here? You aged to
- l e method only took two cycles to converge (as expected)
T he extent of compu on is same as the Pattem S ch b le numberof
iterations is significantly smal 1er.
2 : ; ; 2 1 2 S 2 2 ;: r s e c o n d m l e i S tive. H E n ce
The stepsize for the last itcration is 20. The supporting algorithm for
;;:::!1gand golden secuon mustbe able tO tp the minimum a t e lar
Whi ileno otilh u Islmt d hmerE.th helsaximl 3 u 1m n m Ilu u mbe ?r '0 fcyc1es toconver:gen 1ceflor
a
z::rL ::::z21;f::f g point . Y
(6.1)
(6.2)
(6.4)
(6.3)
j(X); [X]"
i=I.2.....n
j(X) = f(XI. X2) = 3 + (XI - 1.5x2)2 + (X2 - 2)2
OSXI S5: o S X 2 S 5
Minimize
Subject
Subject to:
Minimize
2jz:32ifi:5132ti:;
matrix. lf sea directions S . jand dieco n E
c on.ugacyarc
Steepest Descent Method
This method provides a natural evolution for the gradient based techniques [4].
The gradient of a function at a point is the direction of the most rapid increase in
6.3.1
(6.9)
SJ[U]Sj= 0
Veriif it is truc for thc example.
242 NUMERICAl TECHNIQUES FOR UNCONSTRAINED OPTIMIZATION
6.3 NUMERICA l TECHNIQUEs - GRADIENT-BASED METHODS 243
the value ofthefunctionat tllat point. T he descentdirctioncan be obtained reversinl!
the gradient (or multiplying it by -1). T he next step would be to regard tl1e descent
veclor as a search direction after all w e are aUempting 10 decrease tl1e function
Output (written to e Command window): A couple of iterations at e start and at
EEnd (out of20iterations)Me copied b-low.
through successive iterations. This series of steps give rise to e Steepest Descent
algorithm j
A 'orithm: Stl P tDescent 6.
Step J. Choose X J N (number ofiterations)
!.(l) = j{X 1); X.(l) = X
1
(store values)
EJ E2' EJ: (tolerance for stopping criteria)
Set i = 1 (initia1ize iteration counter)
Step 2. Sj = -Vf(X/)(is is computed in Step 3)
X;+l
is determined by minimizingf(X;+I)
Xs(i + 1) X 1; (i + 1) = f(X
i
+
l
) (store va1ues)
Step3. aJ=!.(i+l) a X = Xs(i + 1) - Xs(i)
IflaJl (function not changing)
If a X
T
a X E2; stop (design not changing)
[fi + 1 = N; stop
If V f(X;+llVf(X
i
+
l
) E3; converged
ii+ 1
GoToStep2
SteepestDescent.m : This is an m-file a1 execut a1gorithm A6.5. It uses e
golden section and tl1e upper bound scanning PI1ess. leares ofthis program are
as follows.
For two variables it w iJI draw tl1e contour plot .
For two variables the design vector changes can be seen graphically in slow
motion witl1 steps in different color.
The design variablese function va1ue and e SqUar of tl1e lengtl1 of
gradient vector(ca11ed KTvalue) at each iteration aredisplayed in tl1e Command
window at completion of tl1e number of iterations.
The gradien1 of tl1e function is numerically comput using first forward fisIte
difference. The gradicnt computation is tl1erefo automatic.
From a programming perspective basic string handling to change line color is
introcluced.
Usage: SteepestDescent('Example6_1' (0.5 0.5]
2 0 0.0001 0 1 20) (6.10)

j
0.5000
0.5000
5.3125 5.2969
0.5996
0.9483 4.7832 2.8411
1.0701
0.8429 4.3766 3.1379
2.6901
1.8659 3.0298 0.0507
2.7455
1.8502 3.0233 0.0462
2.7585
1.8960 3.0181 0.0316
Figure 6.4 rep sents the graphical rnotion of the design variables for 20 itErations.
The graphical picture and the changes in the design arE not imPEESSiVEThe Steepest
Descent mEthod iS WOEfully MadequatE Compazed to Powell-s method even ifthe latter
is a zero-order m etl1od. M O[1veris conclusion is drawn with rpect lo an sy
problem (quadratic) Example 6. 1.is performance justifies tl1e lack of serious
interest in the Stecpest Dcscent method.
5
4.5
0.5
gure 6.4 Slepest desnt : Example 6.1.
244 NUMEAICLTECHNIQUES FOA UNCONSTAAINED OPT1MIZATlON
r J;?12::JI:2412EZ
xpected to overcomthis pattem. In the case of the Univariate m od this was
achEeyed through patFrn Search method in the zero-order family.An iteration
breaki 0 of the zigzag pa m (or preventing getting locked into oInEPi5
::: : r r o : liz::: : 22:e m e eod Su ch
6.3.2 ConJugate Gradient (Fletcher-Reeves) Method
TheCougate Gradient method originalIy dueF1etcher and Reeves [5] is a s maIl
e izi i ;?:;; 5 525g:;:2;qi:?:1izi5;;
c onverge ncb ecau se th e s ea rch d l Ir.ec tu1o E n 3s aMrc0 1 Ij jugate w ith sptto the Hessian
::?xi:E: :: lu uo n. A qu la
A 'or/lm : ConJuga Gradfj ta
Step 1. Choose X" N (number of iteration)
!s(l) =j(X); X.(l) = X (store values)
E E2E3: (tolerance for stopping criteria)
Sel i = I (inilialize iteration counter)
Step2. If i= ISj=-Vj(X
j
)
Elses= '(xjfVj(X
j
)
V j(Xj_l) T Vj(Xi_')
Si=-Vj() + sSj_
X'+I = X / + t
I is determined by minimizingj(X
i
+)
X.(i + 1) X;+;(i+ 1)=j(X) % (s Vs)
S 3. !J.j=fs(i +l) 1); X = X .(i+ 1X.(
Ifl!J.jls EI; (fu mJn n E c ti ion nE
If!J.X' !J.X:$E2 stop (design not changing)
Ifi+ I = N ; 5top
IfVj(Xi+f V j ( X) :$ E3; convergd
i+ 1
G o T o Step 2
c; : : P 2 2 J t : : 2 J U l z ; i t t r t t ; 1 2 3 t z r : r n ;;;:;
6.3 NUMERICAL TECHNQUEs - GRAD'ENT.BASED METHODS 245
computation--s. and th succes5ive adjustment of search direction
incorporating this value. represents the ratio of the square of the current gradient
vector to the square of the previous gradient vector. The first thing to nOlice is at
a degree of robustness is built into the method by carrying information from the
previous iteration. This is Iike maintaining a history of the method albeit for just
one iteration. Since the F O C is based on the length of the gradient approaching
zero at the solution. is P ticular fonn of incorporation of is ingenious. If the
previous ileration is close to the solution. then s is large and previous iteration
plays a significant role in the current iteration. O n the other hand if s is large.
suggesting the solution is still far away then the current value of the gradient
determines the new search direction.
Appllcatlon 0 'ConJugate Gradlent Method: T his method is important to
warrant working through the iterations. T hese caIculations similar to Powell's
m thodcan be done using a hand caIculator. using a sp adsh tor usingMATLAB
itself. T he author recommends the caIculator as e process is slow enough to develop
an intuition working with the paltem of numbrs. In the followingand unlike
Powell's m thodon1y the relevant caIcu1ations are recorded. The reader should
follow aIoog verifying the numbers are iodeed correct.
10.51
Step 1. X 1 =1 o:sl; f(X) = 5.3125
1 0.5 1 _. ___ .... . ___. __ 10.59961
Step 2. S = = 0.1993; f(i) = 4.7831; X 2 =1 O:94841
t = 2 - 1 . 6 4 5 9 1
Step 2: s = 0.5351
11.91351 131
2 = I O:83831;i = 1.2544; f() =3.0; X 3 =121
Con)ugateGradlent.m: T his m-file will execute the Conjugate Gradient method.
For two ariable problems there will be a contour plot over which e iterative
progress in the desi space is recorded. The property of qua8tiC convergence can
be observed the solution is obtained after two iterations. T his is definitely
impressive compared with St pest Descent m od. K p in mind at the
modification to the aIgorithm is minor. As e various codes are run in order. some
changes in the figure can be observed. T hese are largly due to the use of some string
processing functions. Y ou are encouraged to see how ey are implemented. The
author is certain there are ings you would do differently. D o not hesitatto yas
is the on1y way to learn to use MATLAB eff ctively.
U ge:ConjugateGradient( Example6_1' [0.5 0.5] (6.1 1)
20 0.0001 0 1 20);
246 NUMERICAL TECHNIQUES FOUNCONSTRAINED OPTIMATION
Output (Command window)
The problem: Example6_1
The design vector function value and K T value during the iterations
0.5000
0.5996
2.9995
0.5000
0.9483
1.9982
5.3125
4.7832
3.0000
0.2490
2.8411
0.0001
Keep in mind the solution is dependent on the tolerance for the one-dimensionaI
stepsize search and that the derivatives are computed nurnerically. The execution of
e code should display Figure 6.5 in the figure window.
6.3.3 Davldon-Fletcher-Powell Method
The Davidon-Fletcher-Powell (DFP) [6] method belongs to e family of Variable
Metric Methods (VMM).It was nrst introduced by Davidon and seVEral yEMB later
was developed in its current form by Fletcher d PowelI. B ing aware of the
Conjugate Gradient method these methods would not m erIt incIusion in this book if
they did not have the property of qu.adratic convergence which they do. Generally
2 3.5 4 4.5
Flgure 6.5 Coniugate gradlent method: Example 6.1.

5
6.3 NUMERICAL TECHNIQUE- GRADIENTBASED METHODS 247
this farly of methods go beyond that. As the solution is approached they behave like
Newton's method (Section 6.4). The quadratic convergence of Newton's method is
quile impressive-it will locate solution to a quadratic problem in olle iteration.
This Newton-I ike behavior of the V M M as the solution is approached has given them
anothernamtermed Quasi-Newtoll or Newton-like methods.
The D F P method is presented here for historical interest d because it is a little
easier to understand thanothers. In fact Huang (7) documents a generic procedure
from which most ofe popular methods can be obtained and from which you could
also derive your own.
T he Conjugate Gradient method' s irnprovement over the Steepesl Descent method was
possible because of the inclusion of the history m the previous iteration. ln the
quasi-Newton methods th history from aIl previous iterations is available. This
informon is collected in an n X n m ix called e metric. The metric is updated wi
eacb iteration d is used to establish the sh direction. A n initial choice for the metric
is also ud. It must be a tet'c positive deftnite matrix. For the m od to
converge the metlIc must hold on to its positive definite property through the iterations.
h e D F P method the metric approaches inverse of the Hessian at Ule solution.
A orithm: Davidon-F 'cher-Powell (A6.
Step 1. Choosx.. [A1l (initial m eic}N
Eh e2' eJ: (tolerance for stopping cri ia)
Set i = 1 (initializiteration counter)
Step 2. 8=[Aj]Vf(Xj)
X
i
+
1
= X
i
+su A X = j S
i is determined by minimizingf(Xi+l)
Step 3. If Vf(Xi+I)T Vf(X
i
+
1
) e3; converged
If ! f(X
i
+
l
)f(Xj)1 ::;; el; stop (function not changing)
IfA X
T
(design not changing)
If i + 1 = N stop (iteration I imil)
Els
Y = Vf(Xj+l) - Vf(Xj}
z= [A;]Y
A X a X
T
[B]=---:-::::;:::
A X ' Y
ZZT

V ' Z
[Ai+Jl = [A ] + [B] + [q
ii + 1
G o T o Step 2
248 NUMERICAL TECHNIOUES FOR UNCONSTRAINED OPTIMIZATION
In the above e matrices are enclosed by square brackets. The initial choice of the
metric is a positive definite matrix. The identity matrix is a safe choice.
DFP.m: The D F P algorithm (A6.7) is coded in is m-file. Similar 10 the previous
programs for two variables e design changes are tracked on a background contour
plot Figure 6.6 . T he default initial metric is Ihe identity matrix generated using a
MATLAB built-in function.
Usage: DFP(' Example6_1' [0.5 0.51 4 0.0001 0 1 20); (6.12)
Output: The output from the above invocation (copied from the Command
window) can be obtained by removing thc scmi-colon after the statement . The variable
definitions match those defined in the algorithm. For Example 6.1
0
1

=

n
U
A

4 4.5
FJgure 6.6 DFP Melhod: Example 6.1.
5
6.3 NUMERICAL TECHNIQUE3RADIENT.BASED METHODS 249
iteration number: 1
s
0.4990 2.2467
delx =
0.0996
0.4483
y
-1.1458
2.6153
z
-1.1458
2.6153
B
0.0094 0.0422
0.0422 0.1899
c
0.1610 0.3676
0.3676 -0.8390
A
0.8483 0.4097
0.4097 0.3509
iteration number: 2
s
1. 2443 0.5446
T h e problem: Example6_1
The design vector function value a n d K T value
during the iteations
0.5000 0.5000 5.3125 5.2969
0.5996 0.9483 4.7832 2.8411
2.9996 1.9987 3.0000 0.0000
6.3.4 Broydon-Fletcher-GoldfartShanno Method
If you were to program only one gradient-bascd m thod. then thc Broydon-
Fletcher-Goldfarb-Shanno(BFGS)181method would be the one.It is a
quasi-Ncwton method and currently is the most popular of th Variable Mctric
methods. It enjoys lhe propcrly of quudratic convcrgencc and has robustness by
carrying forward information from I13E previous iterations.The differencEbEtween
D F P and B F G S is the way the melric is updaled. The former converges to the
inverse of the Hessian. while the latter converges to the Hessian itself. In a sense
e B F G S is morc direct. The B F G S has replaced the Conjugate Gradienl
techniques as a workhorse in solving nonlinear equations.

H
j
i
l
-
-
J
4
1
J
H
A
H
i
l
l
-
J

250 NUMERICAL TECHNIQUES FOR UNCONSTRAINED OPTlMIZATlON

For convergence the metric must be positive definite. A n initial choice of
positive definIl matrix for the metric is usualJy sufficienllo ensure this property
for quadratic problems. The identity matrix is usually a default choice.
Algorithm: BroFer-Goldfarb-Shanno (BFGS) Method (A6.
Step 1. Choe X " [Atl (initial metric) N
E" E2E3: (tolerance for stopping criteria)
Sct i = 1 (initialize iteration counter)
Step 2. The search direction is obtained as a solution to
[A;]S = -Vf(X )
C l S ; dJ( = S
i is detennined by Minzingf(X)
Step 3. If Vf(X)T Vf(X) E3converged
If I r(Xi+l) f(X)1 EIsp (function not chunging)
If !1X
T
dJ( E2; stop (design not changing)
If i + 1 = N stop (iteration limit)
Else
Y = Vf(X
i
+
l
) - Vf(X)
yyT
[B) =
y'dJ(
Vf(X ) Vf(X?
[ C ] =
Vf(X;)'Sj
[Aj+d = [A;] + [B) + [c]
if- i+ 1
G o T o Step 2
Applictlon 0 'B F G S Method: The method is applicd to Example 6. 1. Only
cssential computations are shown.
cpI X I = l s l ; 1) = 5.31; 1 Vf(X1)l

s t e p 3 Y = 1 2 J r i ; m = l t m l
6.4 NUMERICAL TECHNIQUEs - 5ECOND ORDER 251

R
U

iEc l = 2 m - 3 2 1
-3.0429 6.5095
S 2 4 2 2 1 ; = 144; X 3= [ H =3.0; )=[g]
Converged
The translation of the B F G S algorilhm into codc is left as an exercise for the
student. It is recommcnded that Ule calculation of the search direction in Stcp 2 be
accomplished as a solution to a set of linear equations (as listed) raUler than inverting
the matrix A. One other thing to note for Example 6.1 is that if the polynomial
approximation was used for the one-dimensionul process the calculation of -
would have becn exuct (why?).
6.4 N U M E R I C A L T E C H N I Q U ES -S E C O N D O R D E R
Scond-ordermethods for unconstrained optimization are not uscd bcausvalualion
of the Hessian matrix during each iteration the Hcssian matrix is considered to be
computationally expensive. A second-order method with e property of quadratic
convergence is very impressive. A n n-variable problem can convergc in one iteration.
A s mentioned before for real design problems where decisions are required to be
made on Ule discrete nature of some of e variables existence of the first
derivative not to mention second derivatives is questionable. Moreovcr Ule
quasi-Newton methods of last two sections are able to effcctively function as
second-order methods as ey approach solution-and they do not need the
estimation of second derivatives. One second-order method is prl sented here for the
sake of completenss.
Indeed there is only one basic second-order technique for unconstrained
optimization. It is based on the extension of the Newton-Raphson technique to
multivariable problcm. Many diffcrent cxtensions are available but in this text a
253 6.5 ADDITIONAL EXAMPLES
(iteration I imit) stop If i + 1 = N
Else
ii + 1
G o T o S tp 2
Application of Modified Newton Method: Algorithm (A6.9) is used to solve
Example 6.1. Only essential computations are included:
f(X
t
) =5.3125
-
E
E
E

-
E

0
5

&
n
U

Q
U

s =1 1 : u = 1.0: X? =1 1 ; 1 = 1.0; X;
1 1.5000 l ' U ot - J .v -1 2 1
Asexpected only a singleiteration wasnecessary toobtain thesolution. Translation
ofthealgorithm intoMATLAB code is once again left as an exercise.
converged f(X
2
) = 3.0;
Three additional examples are presented in is section. With many of the numerical
techniques already programmed obtaining the solution is a matter of judicious
application of the numericaI procedures. This section also presents some creative use
of the methods and explores nonquadratic problems. le frrsl example is e
Rosenbrock problem [9]. lIs example was among those created to cha11enge
numerical techniques for unconstrained minimization. It is also sometimes referrd to
as the bananafunction. Ifyou have a copy ofthOptimi<.ation Toolbox from MATLAB
you w i1l see is as part of toolbox demo. Y o u w iJ1 also note at the toolbox
contains many of techniques developed in this section. The second example is a
solution to a nonlinear two-point boundary vaue problem that is due 10 eNavier-Stokes
uations describing flow due to a spinning disk.1last is an unusua data fing
example using Bezier curves. Once again the Optirnization Toolbox presents its version
of the data fitting example when running optdemo at the M A TLAB prompt
A D D I T I O N A L E X A M P L E S 6.5
(6.15)
Example 6.2-Rosenbrock Problem
f(xlX2) = 100(x2 - xT)2 + (1 - XI)2
-2::;;xl ::;;5;
S t e p 1 4 3 1 ;
izl
-2::;;X2::;;5
6.5.1
The Rosenbrock problem is
Minimize
Subject to:

3
3

3
3
3
3

4
1
1
3
1
4
1
1
f
f
i
t
s
?
h
i
t
i
t
i
-
-

1
i
i
f
t
J
NUMERICAL TECHNIQUES FOUNCONSTRAINEO OPTIMlZATION
direct extension recast in e general algorithmic suct (A6.1) is presented. Once
again the general problem and specific example are as fol1ows.
252
(6.1) f(X); [X]n
xlXi xjt
Minimize
(6.2) i= 12... 11 Subject to:
(6.3)
(6.4)
f(X) = f(XI' X2) = 3 + (XI - 1.5x2)2 + (X2 - 2)2
O::;;XI:5; 0::;;x2::;;5
Minirnize
Subject to:
Modifled Newton's Method: The Newton-Raphson method used for a single
variable solves the problem (X) = O. This equation can present e F O C for an
unconstrained problem in one variable (Chapter 5). The iterative change in e
variable is computed through
(6.13)
dx=L=
'(X) f"(x)
where f is th single-variable objeclive function. The multivariable extension lo
computing a similar change in e variable veclor is
(6.14)
The originaI Newton-Raphson is not known for its robustness or stability and
Equation (6.14) shares the same disadvantage. T o control the design variable changes
and to bring it under e schemof a1gorithm (A6.l) the left-hand side is defined
the search direction vctor S followed by a standard one-dimensionaI stepsize
computation. This is terrned the modified Newton method. The complete algorithm is
as follows.

Algorim: Modlfled Newton M eod{A6.9)
Step 1. Choose X J N
E" E2' E3: (tolerance for stopping criteria)
Set i = 1 (initiali7..e iteration countr)
Step 2. The search direction is obtained as a solution to
[H(Xj)]Si = -Vf(X
1
); [ is e Hessian
X i = Xi+ /Si; A X = .SI
" i is deterrnined by MinimizingJ(X
i
+
l
)
Step 3. If Vf(X )TVf(Xi+I) E3' converged
If I r(X 1) f(Xi)1 ::;; EI' stop (function not changing)
IfAX
T
s X ::;;E2SlOp (design not changing)
254 NUMERICAL TECHNIQUES FOR UNCONSTRAINED OPTIMIZATION
The side constraints are used for drawing contours. The solution to this problem is
x ;= 1.0; x ;= 1.0; I = 0.0
Thc problem is notorious for quiring a large number of ilerations for converg ces.
Changes in design are small as the solution is being approached. Here the Random
Walk and Conjugate Gradienl methods are used to examine the solution. Both these
methods are changed 10 draw appropriale conlour levels. T o avoid c1utter disable
printing lexl information on lines representing design changes. Also sincese
methods are run for 200 iterations the fmal printing 10 the Command window is
avoided. T he tracofthe design variables on the contour plot is provided as the outpul
from lhe program along wlth Ihlinal values. A s the figurc is being drawn it is c1ear
that the progress to the solution is slow.
Solutlon Using R a n d o m Walk Method: Figure 6.7 presenls the resull of
executing R a n d o m W a lk.m for thls example. Thprogram invocation and output from
e Command window is
N
M
-1 2 3
X
Flgure 6.7 Random walk method: Example 6.2.
4 5
6.5 AODITIONAL EXAMPLES 255
>> RandomWalk('Example6_2' [3 2] 200 l. Oe-08 0 l 20)
T h e problem: Example6_2
a n s
2.0000e+002 1.0000e+000 1.0000e+000 1.2327e-007
Since thre are no stopping criteria bui1t into the method (was suggested as an
exercise) the method ran for 200 ilerations (first value). te final value for the design
V orand e objective function are very impressive:
random walk : x ;= 1.0; x ;= 1.0; I = 1.23e - 07
Solution Uslng C ougate Gradlent Method: The invocation and output from
the Command window is
njugate Gradient(' Example6_2' 2) 2001. Oe 08 01 20)
T h e problem: E xample6_2
No. o f iterations: 2 0 0
a n s
9.6B07e-001 9.3700e-001 1.0222e-003
The solution is
conjugate gradient : x; = 0.96807; x; = 0.937; r = 1.0222e - 03
This appears no match for the Random Walk method for is investigation. Figure 6.8
tracks e design changes on the contour plot.
6.5.2 Example 6.3Three-Dlmensional Flow near a Rotating Disk
This example represenls another of Ihe exact solutions to c Navier-Stokes equations
from fluid mechanics. The nonlinear two-point boundary value problem describes a
viscous flow around a flat diskat rotates about axisperpendicul to its plane wi
a constant angular velocity. After suitable defmition [10] the mathematical
description of the problem is the following set of coupled nonlinear equations with
boundary conditions at two poinls:
2 F + H ' = 0
(6.16a)
F
2
+ F' H - G
2
- F " = 0
(6.16b)
2 F G + HG' - G " = 0
(6.16c)
256 NUMEAICAl TECHNIQUES FOR UNCONSTRAINED OPTIMIZATION
. . .
M

x
t
2 3
Flgure 6.8 Conjugate gradlent method: Example 6.2.
4 5
F(O) = 0; G(O) = 1; H(O) = 0
(6.l7a)
F() =0; G( ) = 0
(6.17b)
Equations (6.16) are convertedstate spa fonn (s Chapter 5 Example 5.3). The
design variables arE the missing boundary condi6orts on F'(O)G'(O)and the final
value of the independent variable (in lieu of). The objective function is e squar
crror in the integrated vaJue al the final condition.
The Optlmlzation Problem
Minimize f(xl' x
2
x
3
): YI(X
3
)2 + Y3(x
3
)2 (6.18)
where thc slalc variab1es are oblained from the solution of thc following initial value
problem:
6.5 ADDITIONAL EXAMPLES 257
YI
Y2
YI(O) O
Y2
Y i + Y
Y2(O)
~ I I
Y3(0)
(6.19)
Y3 Y4
Y4 2YIY3 + Yv'5 Y4(O)
x
2
Y; -2YI Y5(O)

Thc statc equations (6.19) .c intcgraled using MATLAB function ode45 .

Solutlon to the Problem: The three-variable problem is solved using the D F P
method. Example6_3.m rtums the objective function. It calls the function
Ex6_3_state.m which rcturns the system quations in state fonn. The call to the
D F P method with e initial design vector of [1 -1 8J with 20 iterations of the D F P
method and a tolerance for the golden section of 0.0001 is shown below. AJso in the
call is the input to lhe upper bound calculation.
Usage: D F P (' E xample6_3' [1 1 8 ] 2 0 0.0001 0 1 20) (6.20)
T he final value for the design after 20 iterations was
x;() = 0.5101; .t;(t.t) = -0.6157; xj(= 10.7627; j = 5 08e06 (6.21)
111e tolerance for the golden section was only 0.0001. h is possiblc lhc objectivc
cannot decrcasc unlil a smallrtolerance is specified.
The published values for the design variables ar
x ;= 0.5101; ( ) = - 0.6159 (6.22)
which is remarkably close to Ihe values obtained by the D F P method. The vaJue of
xj is ILslLally not specified bUI is about 7. Since it is a substitute for infinity the Jarger
number should not be a probJem.
Example 6.3 is not trivial problem. The integration is highly sensitive to the
initial values. A reasonable starting point is essential to prevent the integration
from generating N aN's (nol a number). The second design variable musl have a
ncgative sign. The exampl illustrates the application of standard optimization
technique to soJve a nonlinear differential system. A similar appJication can be
made to problems in syslem dynamics and optima1 control . It is essential to
understand the algorithms outlined here are numerical too1s at transcend any
particuJar discipline.
T here are numericaJ techniques at address two-point non1inear boundary value
problems more efficiently. However computing resources especially on a P C are not
a concern. Therefore t11e proccdure adopted in is cxample is an acceptab1e approach
258 NUMERICAL TECHNIQUES FOR UNCONSTRAINEO OPTIMIZATION
for this class of problems. Even the Random Walk method is impressive in is
mstance.
6.5.3 Example 6.4 -Fittlng Bezier Parametrlc Curves
The example introduces a nontraditi al and an unusual curve fiuillg procedure.
Given a set of data points a Bezier curve 01 user-chosen order is fil 10 the data. Bezier
curves are parametric curves a special case of uniform B-splines. Bezier
parametrization is based on the Bemstein basis functions. Any point P on a
Iwo-dimensional Bezier curve (any parametric curve) is aClually obtained P(x(v)
y(v)) where 0 S v S 1.le actual relations are
P (v )=[x y ) =B Jn.1( v
;=0
JII.;(v) )(1 -)".1
(6.23)
whereB; E e vertices ofe polygon that determins the curve. B; represent a pair
of values in two-dimensional space. The order of the curve is lI- t hhighest power
in lhe basis functions. The aclual computations of the poinlS on Ihe CUlve'e ellsier
using matrix algebm and are well explaind and documented in Reference 11. Figure
6.9 a cubic B zier curve. The following are some 0 1 ' lhe most useful
prop ies of the curve as obsrved in the figure:
The curve is completely defined by the polygon obtaincd by joining the vertices
in order.
'he degree of epolynomidefining the curve is one less an the number of
verti s of the polygon.
. T he firsl and last points of the curve are coincidcnt with the first and last vertex.
Thc remaining vertex points do not lypically I ie on thc curvc.
. T he slopcs al the ends of the curvc have the same direction as the sides of the
polygon
. T he curve is contained within the convex hull ofepolygon.
In this exnple the convenient matrix representations for thc curve arc used. T hey are
not discussed here however. but are availablc in the codc. Foris example. the dcsign
variables are the inside polygon vertices. For n = 5 thcre arc 6 vec. The first and
last vcrtex are known from the data points. This leaves 4 vertices or 8 dsign variables
forn = 5.
Example6_ 4.m: lis m -fiIc is a stand-alone program to run thc cxamplc in this
scction. Typing Example 6_4 al the prompt should start thc program. Actually the
0.8
0.7
0.6

8 0 5
p
0.4
0.3
0.2
0.1

6.5 AOOITIONAL EXAMPLES

'1
"
8
2
/
a '
/
/ i
f /
. / '
.

1 J .
- - -


1

1 / "
1 . / l '
1 . / .
1 / J
/
/
.' / C!I 1 /
-
/ E 3
' "
B

0.2 0.4
Flgure 6.9
x data
Acublc Bzlrcurve.
1.6
259
2
example is generic as most decisions a based on e text file that contains the data
to be fiued. 11 calls on m y m-files. The Bezier caIculations are fairly modular. This
program is not vial. A more important ason for its inclusion is atif understood
it signals a reasonable maturity with MATLAB programming. More importanlly it
exemplifies e way to exploil the immense power of built-in MATLAB suppo. T he
author encourages the user to exert somc effort tracing the progress of the calculations
rough e code. A 101 of program control is based on e user' s choice conceming
the order of the Bezier curve. Also the initial values for the design variables are
generated automatical" based on the data 10 be fitted. The function calling sequence
used in is example can be bettcr understood by the following structure:
Example6_5_3.m
CaUs uigetfilc to pick up e (* .txt) Ic at contains two-column data at will be
fit (Ex6_5_3.txt)
'hrows up an inputdialog box to obtain the order of fit from the user (0 = 5)
Calls coeff.m with the order information
Calls comblnatlon.m - calculation 0 1 ' combination for Bcrnstein basis
Calls Factorial.m - caIculates factorial of an integer
260 N U M E R I C A L T E C H N I Q U E S F O R U N C O N S T R A I N E D OPTIMIZATION
Calls Curve_fit.m which handles the optimi tion
C alIs D F P
CalIs BezSq_Err.m calculation of function
Calls gri'sdfun.m - calculation of derivativs
Calls Gold_SectionnVa r '.m - 1 D stepsize
Calls UpperBound_nVar.m - bracket the minimum
Since there are eight design variables a11 of the information except for iteration
number. convergence criteria and objctive function is suppressed from the
Command window to prevent scrolling and loss of comprehension.
le significant elements in is codear as follows:
Picks up e data file througb a file selection box.
Uses an input dia10g box to query the user on the order of the c ve.
Theordrdetermines the size of the coefficient matrix A and it is automatica11y
generat.
The number of design variables is automatically determined. Once again it is
basedon order of the curve.
T he initial values for the design are automatically generated based on the da
to be fit .
The D F P method is used to find the optimum.
The objective function is computed in the generic fiIe BeziecSq_Err.m. The
data specific to e xample are passed through e M A T L A B global statement .
The original data set and Bezier curve fitting the data ar displayed at the
end.
The obj tive function and thconvergence data for each iteration are printed
to the Command window.
ObJectlve Functlon: Thobjective function is the 1 st squared error over all of
the data points between e actual data and the fitted Bezier curve. It is expr'ssed as
Minimize:
IIDa
X): Iy -YBT
1<=1'" .J
(6.24)
The following information is copied from the Command window (the file is
Ex6_5_3.txt and e order is 5):
Bxample6_4
iteration number: 1
gradient length squared:
objective function value:
iteration number: 2
gradient length squared:
5.0001
0.5302
5.6890
6.5 ADDITIONAL E X A M P L E S 261
objective funcion value: 0.2497
iteration number: 3
gradient length squared: 0.0546
objective function value: 0.1830
iteration number: 4
gradient length squared: 0.1322
objective function value: 0.1605
iteration number: 5
gradient length squared: 0.0573
objective function value: 0.1467
iteration number: 6
gradient length squared: 0.0374
objective function value: 0.1437
iteration number: 7
gradient length squared: 0.0071
objective function value: 0.1433
iteration number:
gradient length squared: 0.2793
objective function value: 0.1415
iteration number: 9
gradient length squared: 0.4337
objective funcion value: 0.1401
UI
From the data above it appears that the eor is around 0.14 and is difficult to riuce
further. It also appears the reduction in the objective is accompanied by the increase
in the gradient. after a certain number of iterations.lt is moving away from satisfying
the FOC. The best value for the F O C is about 0.007 and starts to worsen with further
iteratIons. It is quite possible numerical inaccuracies may be a problem as there
are xtensive computations involved in each iteration. It is also likely at e
numerica1 gradient computation may need to be refined. Currently egr ients e
computed using forward finite difference with a step ofO.OO1. These are parameters
tltat n d to bxplored in optimization as there is no universal set of values
for all classes of problems.
Figure 6.10 illustrates e origina1 data d the Bezier curve aerelast iteration.
The polygons are not shown as our effort is to genrate the c ve.he fit is definitely
acceptable and there is probably littleeBezier curve can do about a couple of points
that are not very smoothly located. It is probably these points that do not a110w the
objective functIon to decrease further without increasing thgradient . T his concludes
another innovative use of unconstrained optimization. le author has used these
curves to optimally design airfoils for low-speed flight .
' "
'11'

I
d
-
-
R

4
3

t
m
i
t
t
H

E
l
i
-
-
Note o n F O C for Unconstrained Minimization: example has alerted us
to the question of F O C V f = O. If analytical computation of the derivative is possible
for a problem. then F O C has the likeliltood of being m et. ln design problems like
Example 6. 4 where the elements of the gradient vector have the form
51

oo
4
)o. e..._ 0
-e ...
9
2
263
5. Fletcher R. and Reevcs R. M. Function Minimization by Conjugale Gradients. The
Computer Joumal. V ol . 7 pp. 149-180 1964.
6. Davidon W. C. Variable Metric Methods for Minimizalion U.S. Alomic Energy
Commission Research and Development Reporl No. A N L - 5990 Argonne National
Laboratory 1959.
7. Huang H. Y. Unified ApproachQuac a J ly Convergent A1gorithms for Function
M i n .tion Joumal of O pu wtIon 'aory and Applicatio/ls V ol. 5 pp. 405-423 1970.
8. Vanderp1aats G. N. N l/lnerical Optimizatio/l Techniques for Engineering Design
McGraw-Hil1N e w York 1984.
9. Hock W. and Schittkowski K. Test Examples for Non Linear Programmil!g Codes
Lecture Notes In Economic and Mathematical Systems 187 Springer-Verlag Berlin
1980.
10. Schlichting H. BoundaI Layer Theory McGraw-HilIN e w York 1979.
11. Rogers G. F. and Adams J. A. Mathematical Elementsfor Computer Graphics 2nd ed.
McGraw-HilIN e w York 1990.
P R O B L E M S
NUMERICAL TECHNIQUES F O R UNCONSTRAINED OPTIMIZATION
E
d
-
t

a
u
-
d
u

f

o
- :

a
E
F
-
-
n
"
a
u
-
-

;

O
B
-
h
v
-
-
O

262
e
a
a

a
q

{
S
E
E
E

)
S

P R O B L E M S
Apply the Random Walk method to Example 5.2.
Solve using the Random Walk method:
Minimize !(X.x2) = x1-2 2 + x 1 + ~ - 2 x . + 4
Apply the Pattem Search method to Exampl5.2.
Solve using the Pattem Search m od:
Minimize !(.x.x2) = x1-2x1X 2 + xi+.q -2x. ...4
Modi the Pate m Search so that program control pneters for contour
plotting golden section and upper bound calculation can be set by er.
Include prompt and allow for default values in case the user decides not to
take advantage of it .
Translate the Powel1 method into working MATLAB code. Verify the solution to
Examples 6.1 and 5.2. Start fTom several points and verify that the number of
cycles to converge is esame.
Verify e search directions are conjugate with spectto the Hessian matrix
for Exarnples 6.1 d5.2.
Solve using Powell's m od d at verify any two search directions are
cougate wi respect to the Hessian matrix at the solution:
Minimize !(.x.x2) = x1-2x1X2 + xi+.q -2x. + 4
Solve Example 5.2 using the Steepest Descent Conjugate Gradient D F P and
B F G S m eods.
6.10 Solv using the Steepest Descent Conjugate Gradient D F P and B F G S
methods problem
ze f(x. OX2) = x1- 2x1X 2 + xi+.q -2x. + 4
6.6
6.8
6.9
6.7
6.3
6.4
6.2
6.5
6.1
2.5
and which are best computed numerically some specifications are needed regarding
derivati ve computations if they are to be used for enforcing the necessary conditions.
R E F E R E N C E S
1.5
x data (read from flle)
Orlglnal data and Bezler curve.
i )
1. Fletcher. R Practical MetllOdsfor Optimiwtion V o1.1Wiley N e w York 1980.
2. Brent R. P. Algorithms for Millimization without Derivativestice-Hall
Eng1ewood Cliffs NJ 1973.
3. PowelJ. M. J. D. NOlllinear OptimiWlion Academic Press N e w York. 1981.
4. Fox R. L.. Optimizatioll Methodsfor E l!gineering Design. Addison-Wesley. Reading
MA.1971.
Flgure6.10
0.5
11-

264 NUMERICAL TECHNIOUES FOR UNCONSTRAINED OPTIMIZATION

6.11 Verify the values in Sction 6.3.3 using a calculator. Verify if the matrix A at
e solution is e inverse of the Hessian.
6.12 Verify if the matrix A at the solution is the inverse of the Hessian for D F P
m eod for
Minimize f(XIx2) =x1-2xtx2 +xJ + x ~ - 2x1 + 4
6.13 Develop e B F G S method into MATLAB code and verify the caJculations in
Section 6.3.4.
6.14 Develop the Modified Newton mcthod into MATLAB code and apply it to
Example 5.2.
6.15 H o w will you incorporatee side constraints into the codfor aJl of the various
methods in this section? Implemcnt them in thc numerical procedures.
6.16 Solve Examples 6.2 6.3. and 6.4 by one other m thod of the section.
6.17 Identify and solve a system dynamics problem using any method ofthis section.
6.18 Identify and solvc your own curve fit problcm.
7
N U M E R I C A L T E C H N I Q U E S
F O R C O N S T R A I N E D
OPTIMIZATION
lis chapter explores aJgorithms/methods that handle e gcneral optimization
problem. Both equity and inequaJity constraints are included. For an engineering
problm this will involve bringing several nonlinear relations into the design space.
This is certain to iase the degree of difficulty in obtaining c solution. For the
designer ere is an additional burden of being more attentive to the design changes
and the corresponding numbers to coax the solution if the mathematical definition of
the problem is parlicularly severe. In aJl of lhese problems lhere are two outcomes that
algorithms seek to accomp ish. le first is to ensure at the design is feasible
(satisa Jlconstraints) and the second tit is optimal (satisfies the Kuhn lcker
conditions). W hile the focus is on determining e solution in times of difficulty it is
essential 10 remember Ihatfeasibiliis more important at optimality. Also unstated
in rest of e chapter is at e optimal solution must be feasible. T w o distinct
approaches w i11 be used to handle the constrained optimization problem. T he first
approach is termed e indirect approach and solves problem by transforming it
into an unconstrained problem. The second approach is 10 handle e cons ints
withoul nsformation-the direct approach.
The indirect approach is an expression of incremental development of the subject
to take advantage of e current methods. For exnpleit leverages the D F P method
to handle constrained optimal problems. T w o indirect methods are presented. th
Exterior Penalty Function (EPF) method and Augmented Lagrange Multiplier
(ALM) method. The direct approach handles the constraints and the objecti ve together
wioutany nsformation. Four methods p sented.le methods in isbookare
SequentiaJ Linear Programming (SLP) Sequential Quadratic Programming (SQP)
Generalized Reduced Gradient Method (GRG) and SequentiaJ Gradient Restoration
Algorithm (SGRA).
265
266 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
7.1 P R O B L E M DEFINION
The standard format of the nonlinear programming problem (NLP) is reproduced he
for convenience:
Minimize f(x" X2 X
n
) (7.1)
Subj tto: hk(x" X2' . . . X
n
) = O. k = 12 .1 (7.2)
gj(X" X2. . . X
n
) 0 j = 1. 2 . m
i=I.2..... 1t
In vector notation
Minimize f(X). [Xl"
Subject to: [b(X)]f = 0
[g(X)]" S 0
X1nwSX:;;X"1'
(7.3)
(7.4)
(7.5)
(7.6)
(7.7)
(7.8)
For this chapter the following indices are rcserved: It. number of variabl; 1. number
of equality constrainls; m. number of inequality constraints. Many of Ihe algorithms
and methods present a reasonable level of difficulty. A two-variable prob1em (for
graphical i1lustration). with a single equality and inequality constraint is used to
illustrate the various algorims. A l1 of the functions are nonlinear a1though the
constraint functions have simple geometric shapes.
7.1.1 Problem FormuJatlon-Example 7.1
Example 7.1 is a simple mathematical formulation. It does not present any
engineering prob1em. It is cons cted to have a solution of xj = 1 d xi = 1 for
nstrained prob1em and xj = 0.8520 and xi = 0.8520 for the uncons ined prob1em.
The two constraint functions have simple non1inearity. The uality constraint is a
cile while the inequality constraint is an elipse.
Minimize f(x" X2): x 1 - + xr + XI - 2x1 + 4 (7.9)
Subjecllo: h(X"X2): xr+xi - 2 = 0 (7. Oa)
g(x" X2): 0.25 xT + 0.75 - 1 S 0 (7.lOb)
O :5 xl :5 5; 0 :5 x2 :5 5 (7. Oc)
F

7.1 PROBLEM DEFINITION 267

9<0
3 4
F1gure 7.1 Graphicsolu1ion for Example 7.1.
Figure 7.1 displays the graphical solution to problem. It is clear from the figure
that the solution is at xT = 1. xi = 1 (why?). T he value of the function is 3. A simple
a roach to the solution of the problem is 10 uuation (7.lOa) 10 eliminate one ofe
variables m e problem ereby reducing it to a singlevariable constrained problem.
lis is not done he but it is a good exercise. FIg7 _l.m
1
will create the figure.
7.1.2 Necessary Condltions
le necessary condition-or the Kuhn-Tucker (KT) conditions for e
problem-was devcloped in Chapter 4. It is based on the minimization of E
Lagrangian function (. The condilions are
Minimize
F(xl .. 'X
n
' AI ... Af' sI ... s
m
) = f(xp

x
n
) + Ihl+ ++
P1+ + s"g" (7.11)
1Ft!10 be downloaded fromewebsi are indicaled by boldface sans serif Iype.
A
4

L
/
269
(7. l7c)
Equation st(7.17) is solved symbolically in MATLAB. Thcre are two solutions tha!
satisfy the sidc cons ints (7.l0c)
PROBLEM DEFIN'ON 7.1
xi + x i - 2 = 0
NUMERICAL TECHNIOUES FOCONSTRA'NED OPTlMIZATION
subj tto econsaints. le aren + 1 + m unknowns in the Lagrangian. The same
number of equations is requII to solve problem. T hese are provided by the F O C
or Kuhn-Tucker conditions. Here'l equations are obtained as
268
j = 1.2..... n
(7.12)
dF dr dh. . dh 0 d81. . 11 d8m

= + + + L - 4 1 + + PJ S 7 = 0 ;
axi a x a I a x t t a x i t axi dxt
x;= 1.4142;
x;= 1.0676;= -0.1217
= 0 X; =0;
x ;= 0.9275;
80th of these violate e inequalily constraint. In Figure 7.1. the first solution
definitely do but lhe second is a liltle difficult 10 see. Zooming should help. Case a
docs not provide a solution.
1 equations are obtained directly t1lfOugh e equality constrainls
(7.13)
k = 1.2.....1
h ixX2.... .xn)=O;
lis establishes four equations as g = O Case b:
8j<0 then
m cquations are applicd through the 2
m
cases
sj8j = O if = 0
(7.18a) 4xt + 0.5 s X
1
= 0
(7.18b) - 2 x f + 2 xx
2
+ 2 X2+ 1.5
(7.14)
> 0 lhen if 8j=0
(7.18c) xi - 2 = 0
The K T conditions apply only if the points are regular (see Example 4.3).
(7.18d)
Equations (7.18) solv symbolically and lhe only solution satisfying Ihe side
constrnts s
0.25 - 1 = 0
s " = O
which was expected (for the design variablcs not lhe multiplicrs). The valuc of s" = 0
is not expecled as lhis is an active constraint. The K T conditions can be considered 10
be weakly salisfied as s is not strict1y positive. As in Chapter 6. wherever possible.
e K T conditions are used to check for convergence.
i.." =-0.5000; = 1.0; xi= 1.0;
a

;
t
i
-
-
Appllcation 0' the K T Conditions: e Lagrangian
F(x"

-2xT x
2
+ x T + x -2x + 4 +
xi+ 0.75 - 1)
(7.15)
(7.16a)
(7.16b)
T hree equations are set up through
dF
E ; = 4 4 - 4 x t X 2 + 2 X I + x i - 2 + 2 h i + 0 5 P X l = 0
dF
= 2 x T + 2 xx
2
+ 2 i..x2+ 1.5 s X
2
= 0
C1X
2
Elements of a Numerical Technlque
There is a distincI dierence in the numerical techniques with incorporation of the.
constrainlS. even if lhe idea of using a search direction veclor is still useful . The major
differences in the a1gorithms of the previous chapter was way the search directions
were established. T o understand the necessary changes. the general approach for
unconstrained optimization as seen in the previous chapter was:
7.1.3
(7.16c)
h = x i - 2 =
With this information Equations (7.16) become
The fourlh equation is deve10ped through
s = O Case a:
General Algorlthm 6.1)
Step 1. Choose X
o
(7.17a)
(7.17b)
4x1 - 4 X x
2
+

-2xT + 2 x x
2
+ 2x
2
= 0
270 NUMERICAL TECHNIOUES FOR CONSTRAINED OPTIMIZATION
Step 2. For each iteration i
Detennine search direction vector Sj
(It would be nice if the objective decreased along this direction)
Step 3. Calculate A Xj =Sj
Note: A Xj is n o w a function of the scalar (Xj as Sj is known from Step 2
(Xj is called e step size as it establishes e length o f A Xj
a; is deterrnined by Minimizingf(X;+I)' where
Xj+1 = X + A Xj
As the above steps e reviewed in Step 2 a dilemma n ds to be resolved: Should
the search direction Sj decrease e objective function 0' should it attempt to satisfy
the constraints? (It is assumed at it cannot do both though very often it will.) Since
feasibility is more important an optimality it is sier to identify the latter as a
significant influence in deterrnining the direction. This is more difficuIt than it sounds
as each constraint can have its own favorite direction at the current point . Such
occasions call for a trade-off and to see if it is possible to negotiate a decrease in the
objective function as well. It is clear that Step 2 will involve investment of effort and
some sophistication.
Step 3 is not far behind in the qui ment forsp ial handling. Once the direction
is established (Step 2) what kind of stepsize will be acceptable1 Several results are
possiblc. First thc objcctive function can bdecreased along the direction. Second
current active constraints (including equality constraints) can become inactive
violated or can still p serve e activc state. Third current inactive constraints can
undergo a similar change 01 " state. Fourth probably most important current violated
consaints can become active or inactive. It is difficult to encapsulate this discussion
in a generic algorithm. Somc algorithms combine Steps 2 and 3 into a single step.
Some divide the iteration into a feasibility componenl and an optimality component .
In essence the generic a1gorithm for is chapter which is not particularly efulis:
Generlc Algorlthm (A7.1)
Step 1. C hs e X
o
Step 2. For each iteration i
Detennine search direction vector Sj
Step 3. Calculate AX; =jSj
X j+1 = X j + A Xj
7.2 INDIRECT M E T H O D S F O R C O N S T R A I N E D OPTIMIZAT10N
T hese m cods were developed to take advantage of codes at solve unconstrained
optimization problems. They are also referred to as Sequential Unconstrained
Minimization Techniques (SUMT) (1). The idea behind e approach is to repeatedly
call the unconstai1/ed optimizatio1/ algorithm using the solution of the previous
7.2 1NDIRECT METHODS FOR CONSTRAINED OPTIMIZATlON 271
r
t

4
4
j
5
3
2
iteration. le unconsained algorithm itself executes many iterations. This would
rui a robust unconstrained minimizer to handle a large class of probems. T he
B F G S method is robust and impressive over a large class of problems.
A preprocessing lask involves tr forming the constrained problem into an
unconstrained problem. This is largely accomplished by augmenting the objective
function with additional functions reflect e violation of the nstraints very
significantly. These nctions are referred to as e penal f ctions. le W
significant activity in is at one time which led to entire farnilies of different
types of penally function methods [2). The firsl of these was the Ex1erior Penalty
Function (EPF) method. The first version of A N S Y S [3] incorpora1ed
optimization in its finite elemenl program relied on the E PF. The E P F (presented in
e next section) had several shortcomings. T o address those e Interior Penalty
Function (IPF) methods were developed leading 10 the Variable Penalty Function
(VPF) methods. In is text . only the E P F is addressed largely due to academic
interest . In view of e excellent performance of direct methods these methods
will probably not be uscd today for continuous problems. They arc once again
important in global optimizatiorl techniques for constrained problems. The second
m eodpsented in this section lhe Augmented Lagrange Method (ALM) is lhe besl
of the S U M T . lts exceedingly simple implementation its quality of solution and its
ability to generate infonnation on the Lagrange multip1iers allow it to seriously
challenge direct techniques.
7.2.1 Exterlor Penalty Functlon (EPMethod
The transformation of thc optimization problem (7. 1)(7.4) 10 an unconstrained
problem is made possible through a penalty function fonnulation. The transfonned
unconstrained problem is:
Minimize F(X '11' 'g) = f(X) + p(X '11 ') (7.19)
(7.4) X: :S; X/ :S; i= 1 2 1/
where P(X 'h 'g) is the penalty function. '11 and 'g are penalty constants (also called
multJlies.)
The penalty function is expressed
p(X 'h' ') = 'h 1 L."k(X)21 + '81 ( m {O(X)})21 (7.20)
1"'1 lt1
In Equation (7.20) ife equality cons ints are not zero their value gets squared and
multiplied by the penalty multiplier and then gets added to e objective function. If
thinequality constraint is in violation it too gets squared and added 10 objective
function after being amplificd by penalty multipliers. In a sense if the constraints
272 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
are no1 satisfied then eyarepenizedhence the function's nne. It can be shown
thc lsfonncd unns n problem solves e original m ned problem as the
multipliers r" rg approach . In order for P to remain finile at e values of e
multipliers (1 dcd for a valid solution)e constraints must be satistied. In computer
implementation is limit is placed by a large value insad of . A nother facet of
mputer implementation of this method is alarge value of the multipliers at the t
ileration is bound to create numerical diffi lties. lesemtiplierare started with smal1
values d ua g m elly ch iteration. e unconslraInchniquefor
ex DFPw i1 solve Equation (7.19) for a known value of multiplicrs. The
solution ncd from the DFP can be consider as a function of thc multiplier and can
be thought of as
X. = X.(r"r
8
) (7.21)
T he S U M T iteration involves updating the multipliers and e initial design vector
and calling eunconstrained minimizer again. In algorithm it is assumed thate
D F P method is used (although any method from Chapter 6 can be used BF O S is
rcommended).
Algorlthm: Exterlor Penalty Function (EP M e t h o d 2)
Step 1. Choose X
I
N. (no. of S U M T iterations) N
u
(no. ofDFP iterations)
E;'S (for convergence and stopping)
r! (initial penalty multipliers)
Ch. c/I (scaling value for multipliers)
q = 1 (SUMT iteration counter)
Step 2. Call DFP to minimize F(X
q
r%
OutpUl: xq.
Step 3. Convergence for EPF
If hk = 0 for k = 1 2. . . . 1;
If gj :5 O. for j = 1. 2. . . . m;
If all side constraints are satisfied
Then Converged Stop
Slopping Criteria:
d.F = F
Q
- F 0-" d X = X
q
* - X(q-I)*
q q--I
If (d.Fl :5 EI: Stop (function not changing)
Else If d.X
T
d.X :5 EI: Stop (design not changing)
se lf q = N.: SlOp (maximum iterations reached)
Continue
q q + l
r:r1Cg
xqxq.
0 0 to Step 2
1
7.2 INDIRECT METHODS FOR CONSTRAINED OPTIMIZATION 273
The EPF is very sensitive to the starting value of the multipliers and to thscaling
factors as well. Different problems respond favorably to different values of the
multiplirs. It is recommended that the initial values of the multipliers be chosen as
the ratio of the objective funclIon to the corresponding tenn in penalty function a1
the initial design. This ensures at both the objective function and the constraints are
equally important in detenining the changes in the design for succeeding
lteration.
One reason for the tenn Exterior Penalty is that at the end of each S U M T iteration
design will be ilasible (until the solution is obtained). This implies that c
method detennines design values are approaching e feasible region from the
outside. This is a serious drawback if the method fails p maturelyas it will often do.
Thc information generated so far is valueless as the designs werc never feasible. As
seen in the example below. the EPF severely increases enonlinearity of the problem
creating conditions for the m eod to fail . Il is expected the increase in
nonlinearity is balanced by a coser starting value for the design as each S U M T
iteration starts coser to the solution than c previous one.
ln the following the EPF is applied to Exarnple 7.1 through a series of calculations
rather an through the translation of e algorithm into M AT l AB code. T here are a
couple of changes with respect to Algorithm (A7.1). To resolve the penalty function
with respect to e inequality constraint the constraint is assumed to always be in
violation 8 0 at ereturn from the mllx function is econstraint function itself. This
w i1l drive the inequality constraint to be active which we know to be true for this
example. Numerical implementation as outlined in the algorithm should allow
detennination of inactive constraints. Instead of numerical implementation of the
unconstrained problem an analytical solution is determined using MATLAB symbolic
computation. Example 7.1 is reproduced for convenience.
Minimize f(XjoX2): + 4 (7.9
Subject: h(x(X2): (7.lOa)
g(x( :5 0 (7.LOb)
o :5 Xl :5 5; 0 :5X2 :5 5 (7.l0c)
Figure 7.2 is thc contour plot of the transformed unconstraincd function for values of
rh = 1 and = 1. T he increase in nonlinearity is rcadi 1y apparnt . Figure 7.3 is the plot
for rh = 5 and rg = 5. Handling ese functions numerical1y even with the B FO S wil1
be a considerable cha11enge. Both the figures suggest sveral points at satisfy first
order conditions. Their closeness m it difficult for any numerical technique to find
the optimum.lt is cleartheEPF severely increases the nonlinearity ofthe problem.
Sec7_2_1-plolm contains the code tgenerates the plot . Since the code uses
symbolic manipulation. actually drawing tlle plot takes time. Evaluating the data
Dumerically will make a big difference.
274

4
NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZAnON

7..;..;..:
..--
x
t
z
. . .'"
3 4
Flgure 7.2 Eor Penalty Funcllon Melhod: Example 7.1 (n. = 1 rg = 1).
4
5
Appllcation 0 'E P F A orlthm (A7.2): The corresponding unconstrained func-
tion for is problem is




+4)+
+ xi - 2)2 + '8(0 x f + O1)2 (7.22)
S e c 7 _2_1_calc.m is an m -fiIe that will calculate esohon and the values ofe
function. for Example 7.1. for a predetermined set of values of the penalty multipliers.
It requires two inputs m the user at different stages. The first input is the values for
e multipliers for which e solution should be obtained. le I ist of solution (there
are nine for this problem) is displayed in Command window. The user finds e
solution that satisfies the side constraints (usually one) which must be entered ut e
prompt (note: it must be entered as a vector). The program then prints out e values
of e various functions involved in e example. The following is posted from the
Command window for both of the penalty multiplierset to 25.
Sec7 2 1 c al c
e n t e r value f o r r h [default 1] 2 5
3
l

N
M
7.2 INDIRECT METHODS FOR CONSTRAINED OPTIMIZATION
5
$ everc nonllnearlty wlth many locol mlnlmums
41
3
..
x
t
Flgure 7.3 Exterior Penalty Function Method: Example 7.1 (n. = 5. fg = 5).
e n t e r v a l u e f o r r g [default 1] 2 5
ans
-1. 4125 -0.1015
-0.8692 1.0980
-0.6263 -1.2107
-0.0278 -1. 3266
-0.0124 1.3267
-0.0090 -0.0000
0.7893 -1. 1267
0.9775 1.0165
1. 4184 -0.1318
Input t h e design v e c t o r chosen f o r evaluation
[0.9775 1. 01651
T h e d e s i g n v e c t o r [ 0.9775 1.0165 1
objective function: 2.9810
equality constraint: 0.0112
inequality constraint: 0.0138
275
5
277
Here ).. is the multiplier vector tied to the uality consaintsp is the multiplier
vector associated wilh the inequality constraints and rh and rs are the penalty
multipliers used similar to lhe E P F method. F is solved as an unconstrained
function for predetermined values of p r" and rg. Therefore the solution for
each S U M T iteration is
7.2 INDIRECT M E T H O D S F O R C O N S T R A I N E D OPTIMIZATION
N U M E R I C A L T E C H N I O U E S F O R C O N S T R A I N E D OPTIMIZATION
In the above run the equality and the inequality nstrnt ar not satisfied (
expected). Table 7.1 documents the results of the application of e E P F method to
Example 7.1 through Sec7 _2_1_calc.m. le scaling factor is 5 for both multipliers.
A glance al Table 7.1 clearly i1Iustrates the characteristics of the E P F method . A s the
values of the multipliers inc ase:
276
X.=X(pr
h
rg)
. T he design approaches e optimal value.
The constraint violations decrease.
The solution is being approached from oulside the feasible region.
At e end of the S U M T ileration lhe values of the multipliers and pnalty constants
are updated. The latter are usually geometrically scaled but unlike E P F do not have to
be iven to for convergence.
Algorlthm: Augmented Lsgrange M uipller LM)Method 7.3)
Step 1. Choose X
1
N. (no. of S U M T iterations)
N
u
(no. of D F P
E j'S (fi orc onv rgence and stopping)
r ;. r4; (Oin itia a1 pena aIty ml II u llti i Plie
C h c
g
(sca1ing value for multipliers)
value for multiplier)
.JJI (initial multiplier vectors)
q = 1 ( S U M T iteration counter)
Step 2. C a11 D F P to minimize F (Xq )..q p" rZ r:)
Output: xq.
Step 3. Convergence for A L M
If h
k
= 0 for k = 1 2. . . . 1;
lfg
j
:0for j = 1. 2 . . m ;
( >Ofor =0)
(IfVf+ k 'kVhk
If all sidconstraints are satisfi
Then Converged Stop
Stopping Criteria:
M ' = F q - Fq-1aJ( = xq* - X<q-I)
lf (M')2 S EJ: Stop (function not changing)
se If aJ(T d.X S EI: Stop (design not changing)
Else If q = N.: Stop (maximum iterations reached)
A n important note ds to be mad. le analytical computation using symbolic
computation appearsy. T he same cannot be said for the numerical computation.
especia1ly looking at Figure 7.3.
Augmented Lagrange Multiplier (ALM) Method
This is the most robust of e pena1ty function methods. More importantly it a1so
provides information on e Lagrange multipliers at e solution. This is achieved by
not solving for the multipliers but merely updating them during successive S U M T
iterations [24). It overcoms many of the difficulties associated wi the pena1ty
function formulation without any significant overhend.
7.2.2
Transrmatlon to the Unconstralned Problem: The genera1 optimization
problem (7.1)(7.4) is lsform i n od of Lagrange multipliers
(7.23)
(7.4)
Minimize
r8): f(X) d X ) z d)+
hk

(X) )
Table 7.1
i = 1. 2..... n
Exlerior Penally Function Melhod: Example 7.1
xlsS x !
Continue
q q + l
+ 2 rh h(X"O)
pq pq + 2 r
ll
(max(xq*).-pqI2r
8
])
r1.rrCh; r: r f Cg
s
0.0227
0.0295
0.0138
0.0037
7.5097e004
0.0687
-0.0302
-0.0112
-0.27
-5.9775e-004
f
2.9578
2.9651
2.9810
2.99
2.9988
X 2
1.0391
1.0364
1.0165
1.0044
1.0009
X
1
0.9228
0.9464
0.9775
0.9942
0.9988
r
8
1.0
5.0
25.0
125
625
rh
1.0
5.0
25.0
125
625
Iteration
'
a
q

J
A
-
-

J
278 NUMERICAL TECHNIOUES FOR CONSTRAINED OPTIMIZATION
x
q

x
q
*
0 0 to Step 2
The A L M algorim (A7.3) is available as a stand-alone program in ALM.m. le
program is long which is not unexpected. T hese a1gorithrns are not vial. To translate
algorithm (A7.3) into working code is a mendous accomplishment particularly if
yourexpoto M ATI.AB occurred through is book. T he reader should walk through
the code line by line. e organization and structure are kept simple and
straightforwal'd. Conunenls hav been liberally used as an cncouragement to
underst.1nd the code.
ALM.m: The code gaers input from the user through several prompts in the
Conunand window. The prompts and final iteration ar copied below. The
significant features of the code are:
The starting design vector is the first input to progrn. The number of design
variables is automatically extracted from this information.
le lower and upper bounds on design vables (side consnts)
obtained rough user prompts (though ey are not used in e prograrn).
Number of equality constraints and corresponding initial multipliers ()
promptcd.
Number of inequaJity nstraints and corresponding initial multipliers ()
promptcd.
Values at cach S U M T itcration printcd in the Command window. Wamings
generated by MATLAB switched off.
The DFP.m is used for uncons ined optimization. This program uses
GoldSectionnVa r
oe program exP cts the following esto be available i n e p a:
Objective function Ofun.m
Equality constraints Hfun.m
Inequality constraints Gfun.m
Unconstrained functIon F A L M . m
The program uses global statements to communicate multiplier values.
The initial penalty multipliers are computcd automatically.
Several parameters are coded into e program. especiaIly those needed for the
golden section and the upper bound calculation. These can be changed by
usr. if necessary.
Application to Example 7.1: Thc following represents the initial prompts
secn in the Command window and the values for thc starting iterations. Other
iterations are simi 1arly displayed. Also inc1uded below is a consolidated printing
7.2 INDIRECT METHODS FOR CONSTRAINED OPTIMIZATION 279
ofvalues for a11 ofthe S U M T iterations at the termination ofthe program. The initial
design vector is [32 ]. Initial = 1 and ~ = 1.
A L M
1nput the starting design vector
This is mandatory as there is n o default vector setup
The length o f your vector indicates the number of
unknowns (n)
Please enter it now and hit return for example [ 1 2
3 ...]
[3 2]
t
The initial design vector:
3 2
1nput the minimum values for the design vector. .
These are input as a vector. The default values are
3 * start value below the start value unless it is zero.
1 n that case it is -5:
[0 0]
The minimum values for the design variables are 0 0
-
1
!
c
i
-
-
1nput the maximum values for the design vector.
These are input as a vector. The default values are
3 * start value above the start value.
1f start value is 0 then it is +5:
[5 51
The maximum values for the design variables are 5 5
Number of equality constraints [0] 1
1nitial values for lambda 1
Number of inequality constraints [0] 1
1nitial values for beta 1
A L M iteration number: 0
Design Vector (X) 3 2
Objective function 64
Square Error in constraints(h g) 1.2100e+002
1.8063e+001
Lagrange Multipliers (lambda beta): 1 1
Penalty Multipliers (rh rg): 5.2893e-001 3.5433e+000
280 NUMEAICAL TECHNIQUES FOA CONSTRAINED OPTIMIZATION
(other iteration omitted)
X n o t changing O.OOOE+OOO r e a c h e d i n 5 iterations
T h e v a l u e s f o r x a n d f a n d 9 a n d h a r e
3.0000e+000 2. OOOOe+OOO 6. 4 0 0 0 e + 0 0 1 1.8063e+001
1.2100e+002
7.9603e-001 7.1223e-001 2.9443e+000 0 7.379ge-001
9.9937e-001 1.0092e+000 2.9995e+000 1.8223e-004
2.9434e-004
9.9607e-001 1.0013e+000 2.9962e+000 0 2.731ge-005
9.9940e-001 1.0002e+000 2. 9994e+000 2. 5872e-011
6.2772e-007
9.9940e-001 1.0002e+000 2. 9994e+000 2. 5872e-011
6.2772e-007
T h e v a l u e s f o r lamda a n d b e t a a r e
1.0000e+000 1.0000e+000
-9.0876e-001 0
-1. 6 4 9 3 e001 9.5663e-001
9.1192e-002 9.5616e-001
7.6430e-002 9.9221e-001
T hprogram took five iterations (lifth was not necessary) to find the solution as:
x;=O.94; x ;= 1.0002; r = 2.9994; h. = 0.0000; g. = 0.0007 (7.24)
which is very c10se to the actual solution. The values for e multipliers
A.' = -0.0763; s" = 0.9922
(7.25)
This satisfies the K T conditions. However in Section 7.1.2 the values of the
multipliers obtained using symbolic computation are quite different. It is difficult to
explain the discrepancy except to say the symbolic calculation failed to discover
this particular solution which is indeed possible. le values in (7.25) ar m o
acceptable an those in Section 7.1.2 considering e K T conditions.
T h e A L M m eodas demonstraled above is effecliveand useful method in sDite
of belonging to the class of indirect m eods. It has several significant advantages over
the other penally function formulations [2]. S o mofthem are lisled below:
The method is not sensitive to the initial penalty multipliers or their scaling
strategy.
T he method does not quire e penalty multipliers to be driven to extreme
values 10 tablish convergence.
The equality and active constraint can be satisfied prl isely.
3
7.3 DIAECT METHODS FOA CONSTRAINED OPTIMIZATION 281
. The starting design can be feasible or infeasible.
The initial choices for thmultipliers n be relaxed.
There is only a modest increase in the calculations.
At convergence the Lagrange multipliers w i1l be driven 10 tlleir optimum values.
This aJlows verification of K T conditions through the sign of the multipliers
associated wie active inequality consaints.
7.3 D I R E C T M E T H O D S F O R C O N S T R A I N E D O PTlM I Z ATlO N
:1
The direct methods inc1ude both the objective d .e constrnts to search for the
optimal solution. While e methods do not involve conversion to a different c1ass of
problems (like Section 7.2) most of them are b ed on linearization of the funclions
about the currenl desi poinl. Linearization is based on expansion of the function
about the current variable values using the Taylor seri(Chapter 4).
Unesrlzatlon: T he Taylor series for a l Wo.variable expanded function j{x y)
quadratically about ilie current point (xp' Yp) is expressed as:
1)= f(xf" y/) -1- [ J13{J+
! I + 2 + )
L _.. I (x.JI) ----" I (xpJ') -" I ( J
r

i
y
c
J
u
u
d

Ife displacements are organized a column v [ax 8y]T. expansion in

Equation (7.26) can be expressed in a condensed manner
f( ) = f
L[ A y ]T [H (;Xn.Yn IJ] ]I ( 7.27)
. . IsYI
For It variables. with X
p
the current point and 8 x the displacement vector
f) = f(X
p
) + Vflp)T 8 X + t d X T H )
Equation (7.28) c be wrilten in terms ofe difference in function values
=f(X
p
+ f(X
p
)= Vf(Xp)T d X + t d X
T
X
p
))
d f = O f + o
2
f
282 NUMERICAL TECHNIQUES F O R CONSTRAINED OPTIMIZATION
where of = VjT a X is tenned Jirst variation. o2jis the second varialion and is given
by e s ond tenn in the above uation. In linearization of the functionfabout the
current value of design X
p
' only the fi variation is used. T hneighboring ue of
the function can be expressed as
l(X
p
) = f(X
p
) + V f(X
p
) T a X (7.30)
All of the functions e problem c be linearized simil ly. It is essentiaI to
understand difference betweenf(X) andl(Xp)' This is ilIustrated in Figure 7.4
using the objective function ofExnmple 7.1 expanded about the current designx = 3
x2=2. T hcurved I inesf(X) are the contours of e originaI function. The sBight
lines J(X
p
) are the contours of the linearized function (lines of constant vaIue of the
function) obtained through the following:
Origina I ion: f(x). X2): x1- 2 xrX2 + xi+ X ~ -2x
1
+ 4
Linearized function: ](!J.XI' !J.X2) = +92 !J.X I - 6 !J.X 2
where the coefficients in second expression includes the evaluation of the
function and gradient atx = 3 X 2 = 2 and substituted in E ation (7.30). In Figure
7.4 several contours are shown although those of intrest are in the neighborhood
ofe current design. A n y nonlinear function can be expanded in the same manner.
If another point was chosen. then e slopes 0 1 " Lhe lines would be different .
The quadratic expansion of the function can be obtained by using Equon (7.28).
le expanded curves will be nonlinear (quadr c). leyappear ellipsin Fig w7.5
where severaI contours of e 0ective function of Example 7. 1 expanded
quadratilIy autXI = 3 X2 = 2. It is important to ognize that econtourwouldbe
different if another point w e chosen for expansion.
Four direct methods ar discussed The first is SequentiaI Linear Prograrnming
(SLP) where the solution is obtained by successively solving the corresponding
linearized optimization problem. The s ondSequentiaI Quadratic Programming
(SQP) uses e quadratic expansion for e objective function. Like e S U M Te
current solution provides e starting vaIues for the next iteration. The third. e
GeneraIized Reduced Gradient Method (GRG). develops a sophisticated sh
direction and follows it with an elaborate one-dimensional process. The fourth. Lhe
Sequentia1 Gradient Restoration Algorithm (SGRA) uses a two-cycle approach
working on feasibility and optimality a1temately to find the optimum. lere are
severa1 other methods due to many researchers but ey differ from ose listed in
small details. T o k p the length of this chapter reasonable. ey have not been
incJuded. With ecode d other procedures available from this section. it should not
bedifcult to program thadditional techniquesatare worth exploring for particular
cJasses of problems.
Except for special classes of problems in the following two chaplers. is chapter
is the principal reason for the book. It provides a resolution of the complete
optimization problem as poslated. For a long time now there have not been any new
or greatly improved numerical techniqu. lere is a lot of work on global optimization.
7.3 DIRECT M E T H O D S F O R CONSTRAINED OPTIMIZATION 283
N
M
5
4
3
t
2
2
2
d
t

4
A
f
F
d
-1
x
t
F1gure 7.4 lIIustratlon olllnearizatlon.
n Current interest in traditional desi optition (is chap) is directed toward
actual design appucadons and migration f m m themainfmme and U N I X Environment
to e P C (like is book)For completeness. the standard fonnat of the N L P is
reproduced here for convenience:
Minimize !(x1o X2. x)
(7.1)
Subject:
ht<X " X 2 . X
II
) = o. k = 1 2. . . . 1
(7.2)
~ X I X 2 . . . . ) O. j = 12.... m
(7.3)
Xl S: Xi S:Xr. i=1.2.... n
(7.4)
In vector notation
Minimize f(X). ] "
(7.5)
Subj tto:
[h(X)]= O
(7.6)
284 NUMERICAL TECHNIOUES FOR CONSTRAINED OPTIMIZATION

5
4
3 J



2"'"
-2
2
I







-1
x
v
2 3
FJgure 7.5 lIIustration 01 quadratlc 9xpanslon.
[g(X)]m S 0
X10wSXSXup
4
(7.7)
(7.8)
Example 7.1 will be used to ilIustrate the various algorithms.
Minimize f(x. xi): x1-2xix2+xi+xlrz-2xl+4 (7.9)
Subjectto: h(XI x:z): xi + rz - 2 = 0 (7.10
g(X..X2): S O (7.10b)
OSXI ::;5; OSX2SS (7. lOc)
7.3.1 Sequentlal Linear Programmlng (SLP)
5
In the SLP [5] all of the functions are expanded linearly. If X; is considered ecurrent
design v toren e linearized optimal problem can be set up as
7.3 DIRECT METHODS FOR CONSTRAINED OPTIMIZATlON 285
Minimize: l(aX) = f(X;) + Vf(X;) T a X (7.31)
Sutlo: aX): "k(X;) + vhI (X;)a X = 0; k = 12 . . . 1 (7.32)
(AX): (X;) + V8J{X;)T A X S 0; j = 12.. .. m (7.33)
Ax!ow ax; S a x)IP; i = 12 .. . n (7.34)
Equations (7.31)(7.34) represenl a Linear Programming (LP) problem (Chapter 3).
All of the functions in E quations (7.31)(7.34)except for aX. have numerical values
after substituting a numerical vector for X;. Assuming an L P program code is
available it n be called repeatedly after the design is updated as
X" = X+ A X
In order to include a search direction and stcpsize calculationX in Equations
(7.31)(7.33) is considered as a solution for S (a search direction at the currenl
design). Stepsize computation slragy is no longer simple. Ifre are violated
constraints a1pha attempts to reduce is violation. If e current solution is feasible
e stepsize will attempt to reduce the nction without causing the constraInts to be
excessively violated. Such strategies are largely implemented at the discretion of
algorithm or code developer.
A orlthm: Sequentlal Llnear Programmlng (SLP) (A4)
Step 1. Choose X' N
s
(no. of iterations)
E;'S (for convergence and stopping)
q = 1 (iteration counler)
Step 2. Call LP to optimize (7.31)(7.34)
Output: S
Use a constrnedcalculation (X; = xQ)
AX=.. S
X q+I = X q + A X
Step 3. Convergence for SLP
If h
k
= 0 for k = 1 2 . . . 1;
If gJ S 0 for j = 1. 2 . . . m ;
[f all side constraints are satisfied
If K T conditions satisfied
Then Converged Stop
Stopping Criteria:
A X = x q + l - x
q
If A X
T
A X S EI: Stop (design not changing)
If q = N.: SIOp (maximum iterations ached)
286 NUMERICAL TECHNIQUES FOR CONSTRAINEO OPTIMIZATION
Continue
q q + l
0 0 to Step 2
The SLP is applied 10 Example 7.1 below. In Chapter 3 no explicit program for solving
e L P program was developed although a command-by-command walk rough was
introduced. A program for applying eSimplexmeodpslmplex.m is included on
e website (without any comment or discussion). Using instructions from Chapter 3
adding somc program control and organizing some bookkeeping can provide a
MATLAB program for L P problems. Altemately M .... A B'S linprog can be used in a
loop to help lranslate e SLP algorithm to code. In is secon e linearized
subproblcm in each iteration is graphicalIy presented. The area of interest can bc
zoomed and the solution read from the plot . This is treated as d X (instead of using it
as S). The design is updated and another plot can be obtained by running the m file
for a new value of the design vector. Sec7 _3_1-p1ot.m assists in displaying the
I inearized functions from which solution can be obtained. The equality constraint is
e gr n line. le red is inequalily constrint le design vector is putto e
program. Thde uses symbolic calculation 10 solve and display the subproblem.
Appllcatlon of S L P
sP 1. X
l
= (32)
Step 2. Linearized subproblem r .
Minimize 1(dX) = 64 + [92 -6] I I
I
s X
21
ct II(dX): 11 + [6 4] I1=0
I !lX21
g(dX): 4.25 + [1.5 311 Iso
I !lX21
= a x
l
=1.335& 2 = ax
2
= -O.75.For
thenext iteration X2 = [ 1.665 1.25]. Repeat Step 2.
Step 2. Linarized subproblem
Minimi) = 6.7985 + [13 05 -1.382] 1
I LU21
S : ( ):23347+[333251lh=0
I
s X
21
lax.l-
g(d X 9 + 25 1.8751/ISO

7.3 OIRECT METHODS FOR CONSTRAINED OPTIMIZATION 287

5
4
-4 -3 -2 4 5 2 3 o
O X
1
Flgure 7.6 Example 7.1 SLP ileration 1.
-
-
i
3
i
i
f
z
j
-
From Figure 7.7 (zoomed) the solution is ax
l
=0.54ax2 =0.23. For the next
iteration X
3
= [1.125 1.02]. Repeat Step 2.
Step 2. Linearized subproblem
Minim 1(.) = 3.206 + [2.3957 -0.2363] I
l "'21
bject 10: (): 0.3060 + [2.25 2.04] 11=0
l "'2 l
E)::00 6
From Figure 7.8 (zoomed ) the solution i sdx (=0.125dx2 =0.02. For the next
iteration X
4
= [ 1.0 1.0 ]. Since this is the known solution further iterations
stopped.
le SLP took three iterations t. o find the solution. This is very impressive. It is true
only for th. ose cases where the number . of equality plus active incquality constraints is

l I i
288 NUMEAICAL TECHNIQUES F O R CONSTRAINED OPTIMIZATION
5
4
3
N
>< 0
< 0
-2
-3
4
-5
5
Flgure 7.7 Example 7.1 SLP l Ierallon 2.
4 5
lhe same as e number of design variables-a full comp[ement. The main
disadvantage of lhe melhod appesifere is no full complement. In lhat caseside
consints (7.34) are critical to e detennination of lhe solution as ey develop into
active constraints. In lhese situations lhe limits on lhe side constraints e called lhe
move limits. ln se of active side constrninls Ihe move limils establish the values of
lhe design changes. If lhe design changes are consider as lhe search di ction (s
algorilhm)en bo vaJue and direction are acted. For lhese changes to be small
as lhe solution is approachedemove limits have to be adjusted (lowered) with every
iteration. le strntegy for is adjustrnent may influence e solution.
Example 7.1 wilhout the inequality constraint but wilh e equality constraint
provides an illuslion of the move limits. Figu 7.9 represents lhe lin rization
about lhe point [ 3 2 J .The move limits in lhe figu (box) represent lhe side constraint
limits
-2 S 2
lt is clear from Figure 7.9 Ihat wilhoul Ihe move limits thc solution is unbounded
which is not very helpful . Using lhe solution obtained in lhe figure lhe next iteration
1
7.3 DIAECT M E T H O D S F O A CONSTRAINED OPTIMlZATION 289
5
4
3

3
1
-2
-3
-4
--55 L
-4 -3 -2 -1

2 3 4 5
S X
1
Flgure 7.8 Example 7.1 SLP l Ieration 3.
C be executed. Once again lhe move lirnits will influence e solution. If the move
limits are left at lhe same value the solution will always be on lhe sque with no
possibility of convergence. These lirnits musl be lowered with ch iteration. Usually
they are done geometrically lhrough a scaling faclor. lis implies lhe convergence
will depend on lhe stralegy for changing lhe move limits not an appealing situation.
T he S L P is not one of lhe popular melhods. 111e move lirnits are avoided wilhe next
m eod.
7.3.2 Sequentlal Quadratlc Programmlng (SQP)
The lack of robustness in eSLPdueto e need for move lirnits can be countel by
including an element of nonlin rity in e problem. The arc several ways lhis can
be achieved. One way is 10 require lhal search directions (remember in aca1
implementation s X is S) in Equations (7.31 ) (7.34) be limited by the unit circle which
can be described constrainl:
S T S S: 1 (7.35)
290 NUMEAICAL TECHNICUES FOA CONSTAAINED OPTlMIZATION
5
3
2
.
z o
-1
-2
-3
-4
-5;
-4 -3 -2 o
6 x
Flgu 7.9 lIIustratlon of move IImlts.
stepsize determination should then account for the actuchange in the design
vector. The problem expressed by E q (7.31)(7.35) is a quadmtic
programming (QP) subproblem. A n itemtive solution of this subproblem c beone
form of S Q P m eod. This od is popular and is used widely. While the Q P
subprogram is a simple nonlinear problem with linear consints. it is still a
multivariablc nonlinear problem. Fortunately Q P problems appear in many
disciplines and there are efficient procedures and algorithms to address these problerns
[6-10]. One of thse methods is an extension of the Simplex method of Linear
Programming.
le solution to e Q P is well documented. It would r luire much discussion and
therefore is not introduced hcre. Readers are referred to the various references. This
is also e primary reason why the Q P method is available in the Optimization
Toolbox in MATLAs. T he formal methods for Q P problems are based on efficiency in
the various stages of the implementation. A n inefficient but successful solution 10 the
problem can be obtained by using Ihe A L M of the previous section to solve the Q P
subproblem. After all. computing resources are not a constraint today. In lieu of formal
discussion. an intuitive understanding of e numerical method is attempd in this
book. T he translation of S Q P into code is not trivial and the method is usually the basis
of commercial software. For example S Q P is the only algorithm for constrained
7.3 DIAECT METHODS FOA CONSTAAINED OPTlMIZATION 291
optimum MATLAB implements (as of this writing). Having access to the
Optimization Toolbox allows N L P to be solved using SQP. A detailed discussion of
the solution technique is also available [11].
le Q P subproblem employed in this subsection is based on expanding thc
objective function quadratically about the current design. le constraints are
expanded linearly as in S L P [8]. is is called sequential quadratic programming
(SQP).
Minim 1(d.X) =f') + Vf(Xi
r
d.X +T V
2
fl d X )
Subj lto: ( ) : hk(Xj) + VhI (X) = 0; k = 1.2.....1 (7.32)
gj(Xj) + VgJ{ Xj)T d.X S 0; j = 1.2..... m (7.33)
sx}ow S s xj S i = 1.2 ... n (7.34)
In Equation (7.36) V (X) is the Hessian matrix. In actual implementation the real
Hessian will not be used. Instead a metric ] is updad wi each iteration is
used. s is based on success of the Variable M ec Methods (V.f) of the
previous chapter. Several researchers [6.9] have shown B F G S update for the
Hes8ian provides an eicient implementation of the S Q P method. This Q P is well
posed and convex and should yield a solution. Solution to this subproblem is at
heart of e S Q P method. A considerable amount of research has been invested in
developing efficient techniques to handle is subproblem. In a formal
implementation of S Q P d . X in Equations (7.36) d (7.32)(7.34) musl be
placed by the search direction S. The Q P for S al80 modifies the constraint equations
80 that a feasible direction can be found wirespectto e current active constraints.
Figure 7.10 illusS e concem about moving linearly to a point on an active
con8aint. XO the current design is on an active constraint . If the search direction SI
follows linearized function at XO (the tangent to the function at that point). any
stepsize. along Eh direction. however small will cause the constraint to be
violated. In order to determine a neighboring point that will stil satisfy the constmints.
a search direction slightly less the tangent is used. T his introduces a lack of
consistency as the deviation from tangency becomes a matter of individual experience
andpracti. Experiments have suggestedat90-95% of the tangent is a useful figure
although making it as close to 100% is recommended [2].be search direction finding
Q P subproblem is
Minimize 7(S) = f(X
1
) + Vf(Xj)T ST[H]S (7.37)
Subjecl: (S): c M X
i
) + VhI (X)S=O; k = 1.2.... 1 (7.38)
(S): cgi<Xi) + V 8.x)'f S S 0; j = 1. 2..m (7.39)
s}OWSSjSSr"; i=I.2.....n (7.40)
Repcing Hessian with B F G S Update: ln Eqln (7.37) calculating the
actual Hessian is strongly discouraged. Since the B F G S m od converges to the
Hessian the B F G S metric is considered to be an excel1ent replacement of the Hessi
[9]. T h e m ec is based on the Lagrangian ofe function [12]:
Calculating the Mulflptters: The popularity of the S Q P m od is that it
attempts to update e Hessian based on the Lagrangian of the problem.
Calculating the multipliers is necessary to verify the Kuhn-Tucker conditions at
thsolution. Calculating the multipliers for the current value of the design vector
is necessary for additional operations including setting up the descent function in
Equation (7.43) as well as updating the metric replacement for the Hessian
(follows next). The calculation uses thF O C based on the Lagrangian. The values
of the gradients of the various functions are computed at the current design. The
multipliers then obtained as a solution to a set of linear equations. T he
solutions to Q P and S Q P problems usually take advantage of thactive constraint
set. This includes all equality constraints and only those inequality constraints that
are active. This implies that S Q P iterations start at a feasible point. The
multipliers corresponding to inactive constraints are set to zero. The remaining
multipliers can be established considering a set of an appropriate number of linear
independent equations from Equation (7.12):
Equation (7.42) or (7.43) can be used to obtain e value g stepsize. T he function in
Equation (7.43) requires an important observation. Should the Lagrange multipliers
be calculated for each value of? Imagine using the golden section to solve Equation
(7.43) with e multipliers calculated o m the safaction of e Kubn-T ucker
conditions for each new value of X. Several updating strategies for multipliers in
Equation (7.43) can be found inelistdreferencs. A simple one is to hold the values
of the multipiers constant e previous iteration value and change them at eend
of the iteration.
(7.43)
(7.41)
.42)
293
i= 12.. . n
(7.12)
7.3 D I R E C T M T H O D S F O R CONSTRAINED OPTIMIZATION
n
u
x

J
R
u
r

m

J
r

A
J

+

X

F
J

=

A
V

.
n
u

m
A

m

m
T
L
-
H

r
e
+

4
x

t
n

+

.
r
'
'
E

J

X

A
V

e JF eJf " dh . oh n e}g 1 . . n e}g
= + + + L - + P I + + ~ n ! ! ! . = O ;
axi a x i f t a x i t a x i f l a x i i a a x i
xj=x;'

1+S
Minimize:
Minimize:
The side constraints are not necessary is is a well-posed problem. le factor c is
the adjustment for moving tangent to the constraint.le suggested value for c is
0.9-0.95. For the inequality constrainte value of c is 1 if the constraint is satisfied
(should be able to move tangent to the constraint).
Stepsize Calculation: O ne search direction has been determined it is
necess
Y
to calculate the stepsize. Since the methods of s chapter have seen an
enormous increasin the number of calculations the stepsize calculations are based
on sl Ultaneously decl'sing e obj tive as well as improving constraint
satisfaction. In order to accomplish this goal a suitable function must be developed and
e stepsize determination based on minimizing this unconstrained function. Several
authors refer to it as the descent fllnction. Thdescription of this function varies from
researcher to researcher. Some define is as the Lagrangian function in whicb case
Lagrange multipliers must be determined. Many others include only the
constraints with a penalty multiplier and adopt E P F form for function. Still
others include only the maximum violated constraint. T w o forms of the function az
isted below:
4 3
Movlng linear to an active constraint.
NUMERICAL TECHN1QUES F OCONSTRA1NED OPT1MlZATION
2
Flgure7.10
><" 1
-1
292
294 NUMERICAL TECHNIOUES FOR CONSTRAlNED OPTIMIZATION
p=Vf(X) + L A k Vhk(X 1) + sjVCj(X'')
1<=1 j=IVjeJ
-Vf(Xq) + ' k V + L s
j
V gi
x q
)
(7.44)
1<=1 j=lltjeJ
y = Xq+I _ X
q
[Hq+l] = J+
p'y y[Hq]y
j E J impIies the set of active inequality constraints. It is recommendedm ec
be kept positive definite as much as possible even if it is likely at it w iIl nol be
positive definite at the solution. One way of ensuring is is to verify that
p T y > O
for each iteration.
A 'orlthm: Sequential Quadratic Programmlng (SQ7.
Step 1. Choose X I 1(no. of iterations).
E/S (for convergence and stopping)
q = I (iteration counler)
Step 2. Call Q P to optimize (7.37)(7.40) (X
1
= xq)
Output: S
Use a constrained calculation
AX=" s
X q+1 = X q + A X
Step 3. Convergence for SQP
Ifhk=Ofork= 1.2....l;
forj= 1.2...m ;
If K T conditions are satisfied
n Converged. Stop
Stopping Criteria:
AX=xq+l_xq
IfA X
T
A X EI: Stop (design not changing)
If q = N
s
: Stop (mu m iterations reached)
Continue
UpdateMec [
q q + I
n n rn Stpn
l i
7.3 DIRECT METHODS FOR CONSTRAINED OPTIMIZATION 295
Application 01 SQP: Th S Q P technique is applied to Exarnple 7.1. There are
several ways to accomplish this. The application can use graphical support as shown
with the SLP (this is an exercise). The application can use the Q P program from
MATLAB (also an exercise).ln is book Q Pproblem is solved by applying e K T
conditions-wbich is the feature ofthis subsection. Atirall foris specific example
efunctions are analytically defined and eirstrucis simple. W e can hamess the
symbolic operations MATL to calculateederivatives and eactual Hessian. As
in e case of SLP the actual chang in tlX is computed [Equations (7.36).
(7.32)(7.34)] rather than the search direction S for the Q P subproblem.
Sec7 _3_2.m applies the S Q P m eod symbolically to Example 7.1. lt identifies
the solution using eLagrange method (Chapter 5). It promp for the current design
vector. It solves the Q P analytically applying the K T conditions and providing e
results for the two cases. C ea corresponds to s = 0 (g must be less an zero). Case
b corresponds to g = 0 (s > 0). The output from running the program is recorded below
for several iterations in is implementation of SQP. A na1ytical solution with user
intervention is us. Sec7 _3_2.m must be run for each iteration. You should be able
to copy and pastdesi vector in command window.
Iteration 1:
Iteration 2:
X
I
=[3
M =[2 l=[ ?
l
1;
d.x
l
=-1.0591; =1.1613
=0; g = -0.8225;
X
1
= 1.9409; =0.8387
X
2
= [1.9409 0.8387]
3.123; = U Z ;1 2 2 2
705; V h = [ Jg=0.4693; = 1 1
The soIution to the Q P problem is Case a:
d.x
l
= 0.6186; d.x
2
= -O.0041
h=O; g=1.8281; f
296 NUMERICAl TECHNIQUES FOR CONSTRAINED OPTIMIZATION
7.3 DIRECT METHODS FOR CONSTRAINED OPTIMIZATION 297
1= 1.3222; =0.7975
Iteration 3:
X
3
= [1.3222 0.7975)
f=4.2126; =J= 2 2 8 1
= 0 =!?0858; V g ;;;l
7.3.3 Generallzed Reduced Gradlent (GRG) Method
1 G R G method is anothcr popular technique for constrained optimization. le
origina1 method the Reduced Gradient Method [13] has seen severa1 different
customizatiODS due to severa1 researchers [14). Like SQP is is ao active set
method- deals with active inequalities-hence the method iocIudes uality
cons ints only. T he inuality cons ints E sformed to equa1ity coaints
usiog a linear slack variable of the type used in LP problems.ne genera1 optimization
problem is restructured as
T he SOIUtiOD to the Q P problem is Case a:
Iteration 4:
Minimize f(XI' X2 . Xn) (7.45)
d x
l
= -0.2685; d x
2
= +0.2042
Subject to: ht(XI' x2' x) = 0 k = 1 2 . . . 1 (7.46)
(7.47)
h = 0; g = -0.0190;
8J(XIo X2' . . . x) + xt+j = 0 j = 12 . . m
X
1
= 1.0537; = 1.0018
xlsx;sxr i= 12...n (7.48)
Xn+j 0 j = 12 .. m (7.49)
X
4
= [1.0537 1.0018]
f = 3.0685;
; v h = I 2 1 2 4 ; g = 0 = l T Z l

The total number of variables is n + m. The active !:et !:trntegy ao;sumes that it is
possible define or choose n subset of (11 - 1) independent variables o m the set of
(l + m ) equality constraints. Once these varinbles are estabIished lhe dependent
variables (l + m) can be recovered through consLraints. T he number of independent
variables is based on coriginnl design varinblcs and the original equa1ity constraints
of the problem. The fol1owing development is ndapted from Reference 2.
The set of the design variables [X] is partitioned into eindependentset[Z] d
dependent set [11.
dx. =-0.0523; L1xz=-O.0017; 0.9275
=0; g=O;

-
E
J
Z
Y

r
E

-
E
E
a
h
=

x

1. = 1.0014; = 1 .;
The constraints are accumulated as
f=3.0014; h=O.27; g = 0.0006
H=[:]
The solution is a1most converged and furthcr iterations are left to the reader. T he
next iteration should produce va1ues for convcrgence.
T he SQP here takes five iterations to converge. T his was based on computing the
actua1 Hessian. T he SLP took four. The A L M took five. For Example 7.1e S Q P
does not app r to have y significaot advaotage which is unexpected. T he A L M
slarts to appear qui attractive. T he SQP in Sec7 _3_2.m is implemented
analytical 1y based on K T conditions. It is very different than Algorithm (A 7.5) which
rep sents a numerical implmentation.
Re1ations (7.45)(7.49) can be recasl as
Minimize f(Z Y) (7.50)
Subject 10: H(Z Y) = 0
z!S z;Szf i= 12...n - l
(7.51)
(7.52)
298 NUMERICAL TECHNIQUES FOR CONSTAAINED OPTIMlZATION
A similar set of side constraints for Y can be expressed. 1n acal implmentalion the
inactive inua ity constraints are not inc1uded in problem. Also the slack
variables can b ng to the independent . In G R G lin zed 01ve is
minimiztd suhdect to thelinearizcd constraints about the cumnt design r.
Minimize ](l:J.z' l:J.Y) = f(X
i
) + V zf(Xi? d.Z + V yf(X
i
? l:J.Y (7.53)
Subject to: (!:J.Zl:J.Y) = H.(X
i
) + V zH.(Xi)T d.Z+ V.;.(Xi)Tl:J.Y = 0 (7.54)
lf the G R G began at a feasible point.eB(Xi) = 0e linarized constraints are aso
zeron iblearizedequations in ] detennine a set of linear equations in l:J.Z and
d.Z. Considering vector [H] the linear equation can be written
where
[A]d.Z+]l:J.y=o
r r VvHT 1
I Y ZIIJ I I ya &1 I
1 1 1 V
[A] = 1 - . - 1 ; [B] = 1 " - 1

I V
zm I I V)
(7.55)
From Eqtion (7.55) the change in e depcndent variable l:J.Y can be expressed in
lerms of the changes in thc independent variables as
l:J.Y = [Br
l
[A] d.Z
(7.56)
Substituting Equation (7.56) into Equation (7.53) reduces i t a n unconstrained
problm in d.Z:
]() = f(X
i
) + V zf(Xi)T d.Z - Vyf(Xi)T[Brl[A]d.Z
= f(X
i
) + {v zf(XT -VyflT[Brl[A]) l:J.Z (7.57)
=X +{Vzf(X
i
)([l[Alfvdxt)
= f(X
I
) + [GR]T l:J.Z
[G
R
] is e reduced gradietJt of the functionj{X)It provides the search direction. T he
izeis found by J1iring tngsin l:J.Y J1J1e5ponding 10 ch es
in h z ( S ) determine a f l sible dgn. This is implemented rough Newton' s
method as described below.Since thehnctions M E linearthem is no quadratic
'11111
7.3 DIRECT METHODS FOR CONSTRAlNED OPTIMIZATION 299
convergence associated wiis choice. The iterations start with a feasible current
design vector [Xi] = [Z Y
Stepslze Computatlon: A rlthm(GRG)
Step 1. Find G
R
S = - G
R
; Select a
Step2. q = 1
d.Z=S; Z = Zi +!:J.Z;
l:J.y q = [Br
I
[A]d.Z
Step 3. y q+l = Yq + l:J.y q
x1+1=1 ZI
Iyq
If [H(X)] = 0; Stop Converged
seq q + 1
l:J.Y 9 = r
1
[-H(Xi+I)]
G o T o Step 3
E ve agorim nei Z [A] [BJ noris nged. T hc singularity of [B]
is a concem and can be conOIled by thEselectionofindependent variables.ItapPE
a t a z zs responsible for decrEasein thE function while A Y is rEsponsiblEfor
feasibi1ity. The above algorithm is usual1y xecuted for lwo vaues of. A quadratic
interpolation between the thr points ( = 0 is availabJe from the previous iteration)
is to detennine e optimum psize. One f in al i tera on is performed for i
optimum stepsize.
A 0 1 m: Genele!llzed Reduced Gradient ( G R
Step 1. Choose X
I
(must be feasible)
N. (no. of iterations)
E;'S (for convergence and stopping)
p = 1 (iteration counter)
Step 2. Identify Z Y
C aculate [A] [B]
C acuJate [G
R
]
Calculate Optimum Stepsize(see algorithm)
Calculate X P+l
Step 3. Convergence for G R G
If hk = 0 for k = 1 2 . . . 1;
If 8j S; 0 for j = 1 2 . . . m ;
If K T conditions are satisfied
T hen Converged SlOp
300 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
Stopping Criteria:
LlX = X
p
+
1
- X
P
If LlX
T
LlX EI: Stop (design not changing)
If p = N.: Stop (maximum iterations reached)
Continue
p p + l
G o to Step 2
Applicatlon 0 'the G R G Method: The G R G algorithm is applied 10 Exnple
7.1. Once again the iIlustration is through a set of calculations genernted using
M A1 L AB. For a 5tart Example 7.1 is rewritten using a slack variable for thinequality
constramt.
Minimize f(Xh x2X3): x1- 2xrX2 + xt + - 2xl + 4
Sut H1(xlX2X3):
H2(XhX2): + X 3 = O

For is problem n = 2 1 = 1 m = 1 n + m = 3" -l = ll + m = 2.
Example 7.1 is not a good problem to iIlustrate the working of the G R G method
because only feasible point where g is active is also e solution. This makes Z
and refore G
R

a scal. The example is retained because of uniformity. In

implementation below Z = [XI] and Y = [X2 X3)T.
Sec7 _3_3.m: lIs m-fiIe handles Example 7.1 through e G R G method
symbolically. It requests a starting design which should be feasible. It sets up e Z
and Y automntically and computes [A) nnd [8]. Experimenting with the code indicates
that not all feasible points provide a nonsingular [8]. It then prompts the user for two
vuofwhich ituses to perform a quadratic inteolation. Itcomputes an optimum
slepsize and completes the iteration for d. Y. T his mpleles one iteration ofGRG.
Experimenting with e code iI1ustrates at the choices of need some thought for
the iteration 10 succeed. The following is transcribed from the Command window for
two iterations. Note e prompts for alpha:
lteraon 1
sec7_3_3
Input the feasible design chosen for evaluation.
[1.2 0.74833] % This is x
1
The design vector [ 1.2000 0.7483 0.2200]
function and constraints(f h1 h2):
3.6304e+000 -2.2111e-006 0
A
2.4000e+000
6.0000e-001
B
7.3 DlRECT METHODS FOR CONSTRAlNED OPTIMIZATlON 301
1. 4967e+000 0
1.1225e+000 1.0000e+000
s
-6.0183e+000 (Reduced Gradient)
Input the stepsize for evaluation.
(Prompt for first stepsize)
0.1
alpha
1.0000e-001
No. o f iterations and constraint error: 29
1.0000e-001 8.2187e-009
design vector: 5.9817e-001 1.2815e+000 -3.2110e-001
function a n d constraints (f h 1 h2)
3.3548e+000 7.2526e-005 5.4394e-005
s ' "
-6.0183e+000
Input the stepsize for evaluation.
0.05
alpha
5.0000e-002
(Prompt for second stepsize)
No. o f iterations and constraint error: 12
5.0000e-002 6.2635e-009
design vector: 8.990ge001 1.0916e+000 -9.5823e-002
function a n d constraints (f h 1 h2)
2.9702e+000 6.3314e-005 4.7485e-005
alpha
S 5 6595e 002 (quadratically interpolated}
=
-6.0183e+000
No. of iterations and Constraint error: 14
5.6595e-002 4.3237e-009
design vector: 8.5940e 001 1.1231e+000 -1.3072e-001
function and constraints (f h 1 h2)
2.9903e+000 5.2604e-005 -3.9453e-005
302 NUMERICAL TECHNIQUES F O R CONSTRAINED OPTIMIZATION
This mple oneiteration. This is far from satisfactory as X 3 the slack variable is
negative. The stepsize and design change calculation must be madsensitive to e
side constrints. In actual implementation of the G R G since g(X) Was inactive at e
start it would nol have been inc1uded in the calculations at all. Using the solution to
s t the next additional iterations can be obtained leading 10 the solution.
lteration 2
Rerun Sec7 _3_3.m with X
2
= [8.5940e-001 1.1231e)00]
The rest of e iterations are left as an exercise for reader. T o complete
Sec7 _3_3.m the calculations for K T conditions must be included. In the above since
g(X) is in violation the K T conditions were not computed. T his is currently one of the
more robust methods. For this pacular example the progress of solution is quile
sensitive to e stepsize chosen for interpolation.
7.3.4 Sequentlal Gradlent Restoratlon Algorlthm (SGRA)
The S G R A is a two-phase approach to e solution of N L P problems. Starting at a
feasible design the Gradient phase decreases the value of the objective function while
satisIIng linearized active constraints. T his will cause consaint violation for
nonlinear active cons ints. The Restoration phase brings back the design 10
feasibility which may establish a new d differenl set of active constraints. T his cycle
of two phases is pealed until the optimum is found [15 16]. The m eod incOJporates a
d cent property with ch cycle. Like other algorithms in is stion it uses an 8 Ctive
nstrint strategy. In comparis the G R G combines the two phes togeth.
method has been favorably comp81 to G R G and the Gradient Projection method [17].
T he S G R A as originally inducedin e listed references assumes the inequality
consaints of the form
gj(X)0; j = 12m
T o be consistent wie discussion in the book it is changed to the standard form of
N L P problems defined in is text. In the following development of the algorithm
only the salient features are indicated. They are also transcribed consistent with e
other algorithms in this section. It is commended the reader review the
references for a detaied description. T he general problem is
Minimize
Subject to:
[X]n f(X)
[h(X)]1 = 0
[g(X)]" 0
X10wsXSXUP
(7.5)
(7.6)
(7.7)
(7.8)

E
F
5
7.3 DlR E C T M E T H O D S F O R CONSTRAINED OPTIMIZATION 303
In S G R A only active inuality constraints are of interest. Equality constraints are
always active. Active inequality constrainls also inc1ude violated constraints. If 11
indicates the set of active constraintsuations (7.6) d (7.7) are combined into
a veclor of active cons ints ():
(X)=[J
(7.58)
The number of active inequality constraints is v - l. T he Lagrangian for the problem
can be expressed in lerms of active cons i alone [since multipliers for e
gJ{X) < 0 will be set to zero as part of K T conditions]. le K T conditions a then
expressed as
V x F(X) = V x. f(X) + [V (7.59a)
(X)=O (7.59b)
O (7.59c)
where [V X+1 = [V X+I V
xz Vx'!>v]; = l 1 .JT
GrBdient PhBse: Given a feasible design X' find the neighboring point
s= X i + A X
such that 5j < 0 = O. Imposing a quadratic constraint on the displacement d X
[16]e problem can be set up as an optimization subproblem whose K T conditions
determine
tlX = - a V Fx(X
I
A.= S (7.60)
T he interpretation of the search direction S is used to connect to e other
algorithms in is book. It is not in the original development. The search direction is
opposite to e gradient of the Lagrangian which is a novel feat. T o compute e
Lagrangian lhe Lagrange multipliers have to be calculated. This is done by solving a
system ofIinear equations
[V V xj(X
I
) + [V = 0 (7.61)
Steps;ze for Gradlent PhBse: T he stepsizecalculation is based on driving E
optimality conditions in Equation (7.59a) to zero. T herefo if
304 N U M E R I C A L T E C H N I Q U E S F O R C O N S T R A I N E D OPTIMIZATION
= X
i
-(J.V (Xii.v)
thcn optimumis found by cubic interpolation while trying satisfy
VxF(')..l'l V
x
F i..I')=o (7.62)
Care must be taken this stepsize does not cause significant constraint violation.
This is nfol"ced by capping the squared error in the constraints by a suitable upper
bound wJuch is set up
(g)T() S P
m
(7.63)
Reference 16 suggests tPm is related to anoer performance index Q which is
eerror in the optimality conditions:


(7.64)
Restoration Phase: It is expecd thatatndof the gradient phase the function
will have decreased but would be some constraint dissatisfaction (assurning
there was at least one nonlinear active constraint at the beginning of the gradient
phase). The restoration phase establishes a neighboring feasible solution. 1t does this
by ensuring that the I inearized constraints are feasible. Prior to is. e active
constraint set has to be updated (ii) since previously feasible constraints could have
beC9me infeasible (and previously infeasible constraints could have become feasible)
with design change caused by egradient phase. The design vecr and the design
changes for is phase can be written as
X = X g + dX (7.65)
The design change for this phase dXis obtained as a I st squaree.'or in Lhe design
changes subject to e safaction of the lin r consaints. Setting up N L P
subproblem this calculation for edesign changes is
d X
r
= - V xCT(X
g
) (7.66)
Here "is thevector Lagrange multiplier of the quadratic subproblem. The values
for the mu1tipliers are esblished through the linear equations
( g)-VAds)TVJdE)=0
(7.67)
Thefactor is a user-controlled p'ameter to discourage large design changes.

7.:3 D I R E C T M E T H O D S F O R C O N S T R A I N E D O PTlMIZATION 305

The Restoration phase is iteratively applied until
(lCT() S 8
1
(7.68)
where EI is a small number.
At the conclusion of the restoration phase econstraints are feasible and the next
cycle of Gradient-Restoration ph ec be applied again. A summary of the above
steps is captured in ealgorithm given below.
A orithm: Sequential Gradient Restoration A 'rithm (SGRA) (A7.
Step ]. Choose X
J
(must be feasible)
N. (no. of iterations).
8j'S (for convergence and stopping)
p = 1 (iteration counter)
Step 2. Execute Gradient Phase
Calculate Stepsize using cubic inteolation
Calculate xr
l
Step 3. Execute Restoration Phase
xP+J
Step 4. Convergence for S G R A
If h
k
= O. for k = 1. 2. . . . 1;
If gj S 0 for j = 1. 2. . . . /11;
If K T conditions are satisfied
Then Converged Stop
Stopping Criteria:
a X = XJ*I - X!'
IfaX
T
x S EI: Stop (dsign not changing)
If p = N.: Stop (maximum iterations reached)
Continue
p p + l
G o toStep2
Appllcation of the S G R A 7.: Example 7.1 is used to test the application of
the SGRA. The m file Sec7 _ 3 _ 4.m contains the code that applies the S G R A through
a combination of symbolic and numeric calculations. As in e previous exnples it
is customized for two variables. one equa1ity and one inequality constraint. Unlike the
G R G implementation it incorporates e active constraint strategy. It executes one
cycle of the S G R A which has a single iteration in Gradient phase and several
iterations in e Restoration phase. In the gradient phase the cubic interpolation of the
original technique is placed by the quadratic interpolation used everywhere in the
book. The starting input is a feasible design vector. In egradient phase the method
306 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
reques a value for the stepsize to scan for the optimum using quadratic
intolation. This is important for successive cycles start at the optimum value of the
stepsize for the previous cycl. The value ofin Equation (7.67) is set at 0.5. Printing
is controlIed rough the semicolon and e reader is encouragedfollow details by
flovmg 11.
In the folIowing the program is run wie starting design used for the G R G
method. 1 output from the Command window is pasted below with some bold
annolations 10 draw auention.
>> Sec7 3 4
n p uh e feasible design chosen for evaluation.
[1. 2 0.74833] (feasible design vector input)
The design vector [ 1.2000 0.7483]
function and constraints(f h g):
3.6304e+000 -2.2111e-006 -2.2000e-001
***********************
Gradient Phase
* * * * * * * * * * * * * * * * * * * * *
T h e Lagrange multipliers
-1.0812e+000
Input the stepsize for evaluation.
1 (alpha us ed to start scanning process)
alpha for quadratic interpolation
o 1.2500e-001 2.5000e001
(alpha that brackets the minimum)
function values f o r quadratic interpolation
1.0142e+001 6.6291e+000 1.1300e+001
(corresponding function value)
optimum alpha: 1.1615e-001
The design vector [ 1.0043 1.0622]
function a n d constraints(f h 1 h2) :
3.0077e+000 1.3681e-001 9.8328e-002
(both constraints violated)
The performance indices Q P
6.3181e+000 1.8718e-002
(P measures the squared error i n the constraints)
* * * * * * * * * * * * * * * * * * * * * *
Restoration Phase
* * * * * * * * * * * * * * * * * * * * *
Number of Restoration iterations: 21
The design vector [ 1.0000 1.0000]
/
!
l

7.4 ADDITIONAL EXAMPLES 307

function a n d constraints(f h 1 h2):
3.0000e+000 6.712ge-005 4.6741e-005
(Constraints satisfied - 9 active)
T h e performance index P
6.6911e-009
The solution has converged one iteration. The S G R A is of the same caliber as th
G R G (some comparisons show it to be better). It is difficult to compare different
algorithms because of differenl coding struclures and nonstandard implementation
details. Nevertheless the S G R A has considerable merit. It is definitely better than the
G R G as far as E xample 7.1 is concemed (Section 7.3.3). This comparison may not be
fair s ~ n c e the active constraint strategy was not used for the G R G even though it is part
of the GRG. This demonstrates the need for such a strategy in all algorithms atdeal
wi cons ined optimization.
7.4 ADDITIONAL E X A M P L E S
T hree additional examples are introduced here 10 ilIustrate and expand the eofse
techniques. During the early development of these techniqure was a significant
emphasis on efficiency robustness and keeping computational resources as low as
possible. Most of e techniques were developed on mainframes using FORTRAN.
Today Ihe search for global solulions (at are largely heuristic) coupled with simple
calculations rep led almost endlessly performed on dktop PCs wi incredible
computing power Ihrough softwru systems al provide a wide range of resource
has shifted the focus to experimentation and creativity. The author feels such an effort
must be coupled with knowledge and insight gained through a wide selection of
applications d understanding e progress of numerical calculation. In spite of all
e expos e so far in the for many real design problems the solution is not
automatic. D ling with nonlinear equations is a challenge and often surprising.
Trying 10 understand why the lechnique does not work is as much a learning
expel"ncea routine solution to the problem. Since lhese examples use the melhods
of e p vious section with differnt functions all of the code developed before is
revised with and the appropriate changes and renamed. se modifications quire
that few lines be changed from the original code. lis allows the user to make
incremental changes as needed. This is not an efficient approach bUl capitalizes on the
unlimited hard drive space available today. Students are urged to understand the
algorithm the flow of the code and e result being sought .
7.4.1 Example 7.2agpole Problem
This is the same problem which was solved graphicalIy in Chapter 2 analytically
in Chapter 4 and now considered for numerical solution. In is subseclion the
scaled version of Ihe problem is solved. From Chapter 4 Example 4.3 it is
reintroduced as
308 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTlMIZA TION
Example 2
Minimize (7.69)
gl: 2.55+O.l36lx
1
- 33.148(x1- 0 (7.70
g2: (0. 88 +

- 0 (7.70b)
1 .0868x
1
+ 0.3482 - (7.7Oc)
g4: x
2
- x
1
+0.001
0.02 x 1 1.0; 0.02 x
2
1.0
(7.70d)
(7.71)
Insht about Numerical Solution: In Chapter 4 it was investigated e
soIution does not satisfy the K T conditions because thsolution is not a regular point.
It was ascertained te gradients of e two active constraints tdetermine e
solution (810 g3) are nearly parallel in the neighborhood ofthe solution. H o w will this
affect the numerical search for the soIution?
TypicaJly what a numerical technique does is pick a useful direction and see how
best it can move along it. Suppose the technique decides to move along the gradient
of 83 to develop it into an active constraint . Thcre is obviously going to be a problem
in pushing gl to become active. Pushing a constraint to be active is a basic way to
reduce the objective function in well-formulated N L P problem. Therefore most
algorithms (unless ey are trained to compromise in some significant way) w ilI
underachieve by only pushing one ofthe constraints to be active and ensuring the other
is feasible.
A L M _ 7 _ 4_1.m: T his m-fiIe uses the A L M method 10 solve e problem. ln order
to preserve the old A L M . m to flect the discussion in Section 7.2.2 a copy of the file
is made. Ofun341.m Hfun341.m Gfun341.m and F A L M 3 4 1 . m reflecte
functions from Example 7.2. A1so ALM_7_4_1.m is adjusted avoid handling
equality consaints d provide some printing support. 0 erwise it is essentially the
same. Once again the output from the Command window is p ted below and
annotated to draw attention to some issues. Only initial prompts withe inputs
the initial iteration and the consolidated information after the final itemtion copied
below.
ALM_7_4_1 (invoke program)
Input the starting design vector
This is mandatory as there is n o default vector setup

7.4 AOOlnONAL EXAMPLES 309
The length o f your vector indicates the number o f
unknowns (n)
Please enter it now and hit return for example [ 1
2 3 ...]
[0.75 0.55] (initial design vector)
The initial design vector:
7.5000e-001 5.5000e-001
Input the minimum values for the design vector.
These are input as a vector. The values are
3 * start value below the start value unless it is
zero.
In that case it is -5:
[0.02 0.02] ( i n p u t lower bound not currently used)
The minimum values for the design variables are
2.0000e-002 2.0000e-002
Input the maximum values for the design vector.
These are input as a vector. The defau1t va1ues are
3 * start value above the start value.
f start value is 0 then it is +5:
[1 1] (input upper bound -currently n o t used)
The maximum values for the design variables are 1 1
Number of equality constraints [0]
(default value o f 0 is ok)
Number of inequality constraints [0] 4
Initial values for beta [1 1 1 1]
(input number o f inequality constraints a n d initial
beta)
A L M iteration number: 0
Design Vector (X) 7.5000e-001 5.5000e001
Objective function 2.6000e-001
(In the following violated constraints are shown i n
bold)
Inequality constraints -5.9180e+000 -3.5997e+001
5.2383e+000 -1.9900e-001
Square Error i n constraints(h g) 0 0
Lagrange Multipliers (lambda beta): 0 1 1 1 1
Penalty Multipliers (rh rg): 1 1
310 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
A L M iteration number: 4
Design Vector (X) 6.4726e001 6.1013e-OOl
Objective function: 4.6695e-002
Error in constraints (h g) 0 2.1366e009
Lagrange Multipliers (lambda beta): 0 0 4.8227e002
o
Penalty Multipliers (rh rg): 1000 1000
The values for x and f and 9 are
7.5000e-001 5.5000e-001 2.6000e-001 0
6.5647e-00l 6.0173e-001 6.8871e-002 0
6.4726e-001 6.1017e-001 4.6638e-002 1.5525e-006
6.4726e-001 6.1013e001 4.6695e-002 2.1366e-009
6.4726e-001 6.1013e-001 4.6695e-002 2.1366e009
(Note design and function are not changing hence
stopped)
The values for beta are
1.0000e+000 0 0 0
1.0000e+000 0 0 0
1.0000e+000 1.4062e-002 3.B9B3e-002 4.8227e-002
1.0000e+000 8.9252e001 1.7073e-001 0
(Each column represents values of beta for a n
iteration - beta for constraints 1 2 a n d 4 are zero and
it can b e seen below they are feasib1e b u t inactive.
Only the third constraint is active)
Inequality constraints 6.7858e-002 -4.9734e+OOO
4.6223e-005 -3.6136e-002
(Note third constraint is active. The first one is
almost activel
The analytical solution for is problem is
X;=O774; =0.6445; I = 4.3577E - 02;
g;-O.O; g;= -4.9837; g;=O.O; g;= -O.319 -02
In comparison the A L M has done a decen1job of geuing cJose 10 the solution befo
staIling. Some ofis can be traced 10 discussion prior 10 running e program.
7.4.2 Example 7.B e a m Deslgn
This sec1ion ruires e use of e Optimization Toolbox from MATLAB. More
specificaJly it uses 1he Q P function solve 1he Q P subproblem in the S Q P m ed.
The problem is solved using the direct proced e ilIustraled through the example in

7.4 ADDITIONAl EXAMPLES 311

Section 7.3.2.sectioncombines the symbolic capability ofMAtheresou
available in the Optimization Toolbox to implement an origina1 interpretation of the SQP.
example is edesign of an I-beam for use in a parcars ctura1 problem. is
a ditiona1 m hanicaVcivil engineering strucra1 optimization problem. It was
introduced in Chapter 1 for ilIus tion. It a1so appears in Chapters 8 d IO for iIlus tion
of additiona1 issues. Ineexample hereeI-b m wiI1rry a weight at Rr.le
ncloadg d edesi variables foreproblem are defu:tedF igure 8.7 (Chap
8).lefigure is not reproduced he butX I is the depth of the beamX 2 is the flange width
X 3 isthewickn. d X 4 i s ef1ange tJuckns. Other parameters (gW . Epy)
arenotr efinedhe.
le mathematica1 model is
Minirnize J(X) = pLAc
Subjec1 to: gl(X): W L
3
- 12EI 0.25 (detion constraint)
82(X): W Lx
I
- 8/1y 0 (normaI stress constraint)
g3(X): W Oc -21x
3
: y :S; 0 (shss cot)
Some geometric cons1raints to relate the various design variables are
g4(X): XI - 3 X 2 0
8'(X): 2x2 - XI 0
g6(X): X 3 - 1.5x4 0
g7(X): 0.5x4 - X 3 0
The side constraints on e design variables are
3 :S; x
1
20 2 15 0.125 :S; x
3
0.75 0.25 :S; x
4
:S; 1.25
T he following relations defineecross-seclIonal area (A
c
) moment of iner( d
moment of area abou1 the centroid (Qc) in terms of the design variables:
A c = 2X
4
+ x
1
x
3
- 2x
3
x
4
I

xf (- - 2x3)3

12 12
Qc=0.5x
2
I- xJ + 0.5x3(x 1 - xJ2
The design parameters for the problem are W = 1.4 x 2000 (1.4 is factor of safety)
L = 96 E = 29 E+06 p = 0.284y= 36 E+03:=21 E+03.
312 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
E X 3 4 3 . m : This is e mfile al will implement the Solulion lhrough the SQP.
While this is a traditionaI design problem. it is aIso nol trivial . It is rare to find a
published detailed procedu for solution. The solution is usuaIly obtained
numerically. In any event. anaIyticaI application of the K T conditions is usuaIly not .
contemplated because of 2
7
c(for e 7 constraints) even if many of those cases
are trivial . IncIusion of the side constraints will provide an additionaI 8 linear
cons(agrand to of 15 constraints) of the form (iIIus ted for X) only )
gs(x): -x
1
+ 3 S 0
gI2(X): x
l
- 2 0 S 0
The reader is strongly recommended 10 review code in the m-fiIe forefollowing
reasons:
The power of symbolic manipulation is ploited further using vector
definitions 10 make the implementation fairly easy. The symbolic calculations
are used to genemte e dala for the Q P subproblem.
T h symbolic calculations are effortlessly inlegraled with numerical
computations (thsoluLion to the Q P is numerical).
The implementation of S Q P is s ightforward and direct (and simple).
'he solution is siificantly better an the numerical solution by MATL
o w n S Q P program (Chap 10).
The program requires thstarting guess for the design vector. The functions for the
example ar hard coded. The final solution is printed to the Command window:
Initial Guess:
Final Design:
Objective:
n
u
n
u

'
a
n
u

n
u

J

n
U

4
z
J
'
i

n
u
n
U

4
n
u

r
o
o
n
u
z
u
t
J

n
u
U
0
6

m
5

A lI constraints ar satisfied:
Acive constraints: 1 (deflection). 7.9 (X2 lowerbound).10(.lowerbo d).
11 (X4 lower bound)
The reader is encouraged to explore w h y the solution obtained he is significant1y
better the one using the S Q P from the toolbox. granted that the implementations
are not the same. Some thought should be given to penormance of quasi-Newton
(meric) methods and their adaptation 0 constrained lechniqus if the Hessian is no
positive definite which in this case it is.
7.4 ADDITIONAL EXAMPLES 313
7.4.3 Example 7.4 - 0ptlmal Control
T his section introduces another cIass of problems C be addressed by design
optirnization techniques. It is the basic trajecory determination problem in optimaI
control . Given two points in a plane find a paat w i1I be trversed between them
using a steering v tor of constant magnitude. The path has to be aversed in unit
time. The problm c bedescribed using Figure 7.11. If the problem n be described
in e physical plane. the sl ring vector considered as velocity. en the kinematic
uations for a particIe moving on e pa can be described by the following
differential equations (d i optimization prefers to deal with a1gebraic functions).
whe X.Y locate e problem in physical space:
= V cos 8(t); 0 S t S 1
= V sin 8(1); 0 S t S 1
The initiaI and final conditions are (t11ese were the values chosen for the example)
(0) = 0; y(O) = 0
x(l) = 3; y(l) = 2
Here. 8(t) is regardd as a control variable. II is a continuous function of t between 0 S
t S 1 that will transfer the pac 1e from initial point 10 the final point . In contrast.
design optimization (i.e.. his book) determines scalar values of the design variables.
These types of problems belong to the area of optimal control . T h are special
numerical techniques al are available 10 solve Ihese categories of problems. One
approach to these problems is to use design optirnization techniques. T his is aided by
using a parameic representation for the control function. Man y different parnec
p sentations for the control have been used. such as cubic splines and Hermite
polynornials. In this sudy a Bezier curvc is used to rep sent the control. The
differential equations aren numerically inlegrated 10 esblish thjeclory.
y
x
FIgure7 Descriptlon of Irajeclory nlrol. 1.

which requires brachi.m to retum the state space definition of problem.

brachi.m also includes the Bezier interpretation code.
Figure 7.12 describes e Bezier veices of the control function at the stan and
final iterations. T he final continuous control is also plotted on the figure. The start
curve is a saight line through the vertices and is not shown. Figure 7.13 shows e
start trajectory d e final jectory. It is cl rfrom efigureeinitial and e
final conditions are satisfied for the final trjectory. Since e equations w e
integrated. the differential constraInts are therefore completely satisfied.
T he final trectory shown is only feasible it is one of many canp
betw n the two points. This opens the possibility at a further option c be
performed with spect tojectory. Forexample. find eshortest path in etime
provided. Not it is not e straight pa since for a given velocity magnitude the
destination may be ached sooner anp scribed. The cassical problem in optimal
ntrol is e brachislochrone problem. lt is e minimum time for the ple 10
trvel between two prescribed points dynamically (obeying principles of dynamics).
The reader is cncouraged to expand technique and e code to explore se
additiona interesting problems.
315 7.4 ADDITIONAl E X A M P L E S N U M E R I C A L T E C H N I Q U E S F O R C O N S T R A I N E D OPTIMIZATION
Solutlon Technique (adopted in is section)
Use a Bezier curve to parametrize the control function.
Integrate the differential equations forward using a Runge-Kutta method
(ode45.m)
Use uncons ined lechnique (fminunc.m) minimizer to ive e square of
eerrorin e final condito zero 0 st squar error).
314
Feaslble troJectory
This example nds the M ATI.AO toolbox to execute fminunc.m. Altemately. e
unconstrained minimizer can be among those previously developed. is translates
inlo an uncons ined minimum problem which therefore should have been handled
in Chapter 6. It is incuded here because e differential equations are actually
considered drential conslraints. The objective function therefore is
f{X}= [ 'finnl-x(l)f+ [Yfinal-y(l)J2
Ex744.m will execute the example and produce plots of the controI and the trj tory.
It uses two standard M A T L A O functions mentioned before (odc45.m and fminunc.m).
The Bezier curve generation r uires coeff.m. Comblnatlon.m. Factorlal.m. le
objective function is constructed in obLoptcont.m. It qui s the use of ode45.m.
M ze
The vertlces and 0 (1)
z





llnal polnl

3
3.5 2
1.5
2.5

'

-
H

- - -

-
2
>- 1.5
0.5
3.5 3 2.5
Flgu 7.13 Trajecloriestart nal .
2
x
1.5 0.5
-0.5
0

o
0.5

a
'

e

d
y

o

a'

-...
0.9 O. 0.7 0.6
Control funclion 0(/)
0.5
tlme
0.4
Flgu 7.12
0.3 0.2 0.1
316 NUMERICAL TECHNIQUES FOR CONSTRAINED OPTIMIZATION
R E F E R E N C E S
1. Fiacco A. V. and M cConnick G. P. Nonlinear ProgralllmiJlgSequential
Unconstrailled Minimizotioll Techniques Wiley New York 1968.
2. Vrpl G. N. Numerical Optimizotion Techlliques for Engineering Desigll
McGraw-HiII New York 1984.
3. ANSYS Software A nsys Inc. Pittsburgh PA.
4. Bertekas D. P. Constrained Optimizotion and Lagrange Methods Academic Press New
York1982.
5. KeJley J. E. The Cutting PlanM eodJoumal ofSIAM V ol. 8 pp. 702-7121960.
6. Gill P. E. Murray. W. and Wright M. H. Practical Optimizotion Academic Press New
York1981.
7. Boo1 J. C. G. Quadrauc Programming Sludies itl Mathematical and M Qllagerial
Economics H. Theil ( dVol. 2 North-Holland A msterdn1964.
8. Biggs M. C. Constrned Minimization Using Recursive Quadrauc Programming: Some
Altemate Subproblem Fonnulauons Towards Global OptIIzotionL. C. W. Dixon and
G. P. Szego (eds.) pp. 341-349 North-Holland Amsterdwn 1975.
9. Powell M. J. D. A Fast Algorithm for Nonlinear Conslrained Optimizauon CaIculalions
No. D A M P r P 77INA 2 University of Cambridge England 1977.
10. Han S. P. A G10bally Convergent Method for Nonlinear Programming Joumal
Optimizotion Tlreory and Applications V ol. 22 p. 297 1977.
11. Arora. J. S.. Intrnductiolllo Oplimwn De.risn McGrnw-HilINew York 1989.
12. Branch M. A.and Grace A.Optimizotiotl Too'oxUser's Guide MathWorks lnc. 1996.
13. Wolfe P. M ods of Nonlinear Programming Recenl AdvQllces i/l Mathemalical
Programming R. L. Graves and P. Wolfe (eds.) McGrnw-HiIlNew York 1963.
14. Gubriele G. A. Dnd Ragsdel J K. M.leGcnerolized Gradlcnt Method: A Reliable Tool for
Opa I Design. A S M E Joumal of Engineering and IndJ" SeT B Vol . 99 May 19.
15. MieleA Huang.H.Y.dHeidemanJ.C.SuentiaI Gradient Restorauon AIgorithm
for the Minimizauon of Constrained FunctionsO r dinary and Conjugate Gradient
Versions Joumal ofOptimizotion Theoand Applications V ol. 4 No. 41969.
16. Levy A. V. and G omez S. SequentiaI Gradient-Restoration AIgorithm for the
Optimizauon of a Nonlinr Constrained Function. Joumal'he Astronautical Scietlces
Speciallssue on NumicM eds in Optimizauon Dedicated to Angelo Miele K. H.
Well (sp. ed.) Vol . X X X No. 2. 1982.
17. Rosen J. B. The Gradient Projection Method for Nonlinear Programming Part D:
Nonlinear Constraints S I A M Joumal of Applied Mathematics Vol. 9 No. 4 1961.
P R O B L E M S
7.1 Modify Sec7_2_1.J)loLm to review contour values and plot automatically
selccted contours.
7.2 Modify Sec7 _2_1_calc.m to automatically calculate the initial multipliers
loop over pena1ty multipliers. and to automatically pick the design vector
satisfies the side constraints.
PROBLEMS 317
7 ~ (uses Optimization Toolbox) Develop e S L P by using the L P solver o m the
Optimization Toolbox and apply to Example 7.1.
7.4 Solve Exampl7.1 by solving e Q P problem graphica1ly.
7.5 Dcvelop code to find a feasible solution from a given slarting vector for
nsltined problems.
7.6 Develop a program to calculate the multipliers for the N L P problem at a given
design.
7.7 Develop a program for a constrained stepsize calculation procedure.
7.8 (uses Optimization Toolbox) Use the Q P program from the toolbox and develop
your o w n S Q P implementation.
7.9 (uses Optimization Toolbox) Use e S Q P program from MATLAB to solve
Example 7.1.
7.10 Develop S e c7 _3_2.m to automatica11y execute severa1 iterations to
convergence.
7.11 Implement a new version of S Q P where the Hessian matrix is maintained as the
identity matrix for a11 iterations.
7.12 Finish solving Example 7.1 using the G R G m eod.
7.13 Solve Example 7.1 with Z = [X2] and Y = [x x)]T.
7.14 Modify the G R G code to include consideration of active constraints and
compe perform ce.
7.15 Build in K T condition calculation into the SGRA.
7.16 Solve the Brachistochrone optimal control problem.
8
D I S C R E T E OPTIMIZATION
This chapter imroduces some concepts methods and algorithms associated wi
discrete opti zation. It is not possible to compress sut of Discrete
optimizauon to a single chapter as the topic itself is capable of spawning several
courses. Discrete optimization is very different difficult diverse and continues to
develop even today. These problems are largely combinatorial and are
compu ona1ly more time intensive corresponding continuous problems.
The problems addressed by the discrete optimiz on research community are mainly
in eaofop eratiosse chchacriz usually by linea rm o
optimizatio nmostly incorporat.esno nlinearre la tio n IS. From a real perspecti ve discrete
design vaItablesMBfundamental inengineedng optimization.h e beam desIlm
p lm in Chapr 1 a p ical solution should involv identifying an
off-the-shelf' beam as erolling rnill will probably make it prohibitively expensive
for a limited production of unique beam was identified the solution to e
continuous optimization problem. In the problem regarding e number of different
placement machines the solution was expected to be integers (discrete value).
rnilarlchoice of dian rslengths washers valves men componentsck
SlzeSG so onare usually guided by avoiding high procurement costs associated
wi nons dard components unless cost recovery is possible because of large
volumes. C o n uti IOl I l
item s to be completely manufactured i n"hou s (no ofllf com nenl is
necessary)
Practical enginering design quirlal some design variables belong ( 0 an
ordered set of values.-discrete variables.This makes it a discrete optimization
318
DISCRETE OPTIMIZATION 319
problem. However there are very few engineering problems are currently
solved as a discrete optimization problem. First discrete optimization algorithms
are difficull to apply. Second they are time consuming. Third most discrete
algorithms and code address linear mathematical models. Discrete optimization in
engineering w il1 necessarily involve adaptations from e currently available
techniques used by the decision-making communily. Such adaptations ar
uncommon. It is also rare to find y book on design optimization addresses
or includes the subject of nonlinear discrete optimization notwithstanding the fact
at it is enormous in extent. A modest effort is being made in this book to acquaint
the reader with the subjct area.
le typica1 approach for incorporating discrete design variables in engineering is
to solve the corresponding continuous optimization problem and adjust the optimal
design to nearest discrete values (is is sirnilar to the rounding process to arrive
at an integer number). According to Fletcher [1]there is no guaranteisprocess
is correct or a good solution can be obtained is way. Very often is rounding
may result in infeasible designs. Given the altemative is to solve the discrete
optirnization problem itself a systematic approach to is rounding process has
become acceptable. A rounding process is based on maintaining a feasible design
is available in Reference 2.
'his chapter and the next will differ significantlyo m the previous chapters in both
content and organization. Their primary focus is presenting new ideas and
content ratheran developing a technique or assisting in code devlopment . Discrete
problems require significantly different concepts e traditional continuous
mathematical models of the earlier chapters. Simple examples are used to bring out
e difference. For example derivatives gradients and Hessian do not apply to
discrete problems. B y extension search directions and one-dimensional stepsize
computation lose their relevance. While ere are sveral excel1ent references on
discrete optimization almost all of them deal with linear mathematical models. There
are few if any about nonlinear discrete optimization applied to engineering
problems.
Discrete optirnization problems (DP) are implied when the design variables are not
continuous. Extending concept of the side constraints these design variabl
teger value of 8 => binary value of 1000
320 DISCRETE OPTIMIZATION
Therefore if e design variable x is rescted to integers between 0 S; x S; 16 then x
can be placed by five 0-1 design variables [y.. Y2 Y3 Y4YsJ from which x c b
assembled as
x = YI (2
4
) + Y2 (2
3
) + Y3 (2
2
) + Y4 (2
1
) + YS (2
0
)
Such a transfoIIDation is not recommended if re is a large number of integer
variables. Engineering design problems can be expected to contain both continuous
and discre variables. These are termed Mixed Programming (MP) problems if e
mathematical model is line In this book the classication is extended to nonlinear
problems too.
In the interest of manageability only three methods e presented in is chapter.
These methods are only representative and in no way address topic of discrete
opnization sufcientlylet alone completely. T he third one has seen limited use but
with computional resources not bing a hurdle today it is definitely attracve. The
selection does not represent any priority though e methods are among those at
have been applied often in engineering. The methods of the next chapter have evolved
through their initial application to discrete optimization. They could also belong to
is chapter but have been kept sep ate because ey are the driving force in the
search for global optimum today. The methods of is chapter are (l) Eaustive
Enumeration (2) Branch and Bound (partial or selective enumeration) and (3)
D I amic Programming.
8.1 C O N C E P T S IN D I S C R E T E P R O G R A M M I N G
In the opening discussion some defmitions of discte programming problems
regarding the nature of discrete optimization were inoduced.ln is section the
concepts are detailed using a simple unconstrained minimization example. As a result
e optimality issu do not in de on presenting the important ideas in discrete
optimization including the treatment of finding a continuous solution followed by
adjusting the discrete design variables to neighboring discrete values.
Example 8.1 M inimizeobjective function given below wherexl is a continuous
variable and X2 X3 are discrete variables. X2 must have a value from the set [0.5 1.5 2.5
3.5J and x3 must have a value u m the set [0.22 0.75 1.732.24 2.78J.
Minimizef(x
1

x
3
)= (x
1
- 2)2 + (x
1
- ~ ) 2 + (x
1
- X
3
)2 + (x
2
- x
3
)2 (8.1)
The side constraints on the design variables can be set up as
x
1
E R (8.2a)
X
2
E .5 1.5 2.5 3.5J E xu (8.2b)
.1 CONCEPTS IN OISCRETE PROGRAMMING 321
x
3
[0.22 0.75 1.73 2.24 2.78] E X
3d
(8.2c)
The symbol E identifies at the variable on the left can have one of the values on the
right side. In Equation (8.2a) R stands for a real value. This represents a standard use
ofsymbol E d is an effective way of expressing the idea that the discrete values
C only be selected from a group of values. Tbis is a mixed programming (MP)
problem.
8.1.1 Problem Relaxatlon

P
4
4
M
J

E is were a unconstrained problem in continuous variables the solution could be

obtained by setting V f = 0 and sol ving e e equations for e valusof x..X2' and
X3Eqtions 8.2b and 8.2c h to be overlooked. Altemately for a continuous
problem inspection of the objective function yields e following solution:
x;=2; x;=2; x;=2; I = O (8.3)
For the original problem 'dXl is defined but not df/dx2 or df/dx3' since X2 dX3
are discrete values. Derivatives are defmed by taking the limit of the ratio of change
of objective functionfto the change in the value of the design variable as the change
in the variable approaches zero. The value ofobjective functionfis only defined
at selected combination of X2X3 in Equation (8.2) and is noL defined elsewhere.
Smalllinfinitesimal changes in the discrete design variables and therefore in e
objective function are not defined in Example 8.1. Derivatives with respect to
discrete variables do not eXIst. This conclusion is of m 01' significance as it makes this
book's previous body of work of I imited use in e pursuit of tbe solution to the DP.
To recollectenecessy and sufficient conditions which drove the algorithms were
based on the gradients and their derivatives. T w o areas in the previous chapters escape
with a limited impact of is statement. beyar L P and zero-order methods for
numerical solution to unconstrained nonlinear optimization. Both of them play a
significant role in discrete optimization problems. It is therefore possible to conclude
at solution to Example 8.1 established in Equation (8.3) is incorrect (e side
constraints are in violation).
In D P the solution in Equation (8.3) represents the solution to a relaxation problem.
Problem relaxation can take severalforms. Mostly it is applied to erelaxonore
weakening of the consain or the objective functions. There no explicit
constraints in Example 8.1. Inis instance the relaxation refers to the removal of the
restriction of discreleness ofthe variables. This is identified as continuous relaxation.
The rel ed problem or relaxation has several advantages [3].
. If a constraint r llaxation is infeasible so is the problemlmodel it relaxes.
Constraint relaxation expands the set of feasible solutions. The relaxed optimal
value must improve or equal the best feasible solution 10 e original
problemlmodel.
322 DISCRETE OPTIMIZATION
The optimal value of any relaxed model provides a lower bound on eoptimal
solution if il is a minimization problem. Similarly il establishes an upper bound
for maximization problems.
Many relaxation models producoptimal solutions at are easily rounded to
good feasible solutions of the original problemlmodel.lis appears to
discrete optimizon for engineering design problems.
The values established in Equation (8.3) are a solution to continuolIs relaxation of
Example 8.1. It was easy to obtain is solution while a technique for the solution to
e discrete unined opzation is still unknowne Kuhn-Tucker
conditions are no longer useful.le solution in Equation (8.3) is not acceptable-xi
and xj are not elements of the permissible se1 (8.2). T he value of I = 0 will be betler
best discrete solution-lower bound 0 " the solution 10 original problem.
8.1.2 Dlscrete C!ptlmal Solutlon
A standard approach 10 solving discre1e optimization problem particularly in
engineering is 10 use the continuous relaxation of the mathematical model and
genemte a continuous solution. For Example 8.1 this is given in Equation (8.3). More
than likely e discrete variables will not belong to the pdefined discrete sets. These
variables are then changed (or rounded) to e nearest discrete values. For problems
with constraints feasibility is checked. The best feasible solution is lhen chosen.
Unlike continuous problems there a no necessary and sufficient conditions to
satisfy.
ln e case of Example 8. 1 Equations (8.2) and (8.3) provide the relevant
information to establish the solution. For convenience tJley are reproduced here:
XJ R (8.2a)
X
2
e [0.5 1.5 2.5 3.5] e X
2d
(8.2b)
X
3
[0.22 0.75 1.73 2.24 2.78] E X
3d
(8.2c)
(8.3) x ;= 2; X; = 2; x ;= 2; I = 0
le continuous variable XJ is chosen 10 have a value of 2.0. Figu 8.1 indicates four
sets of disc e values for X 2 and X 3 around the continuous solution. The least value of
the objective function at these points will be considered e solution. Evaluating the
values of the objective nction at those points:
x ;= 2; X; = 1.5; X; = 1.73; 1 = 0.3758 (8.4a)
x ;= 2; x ;= 1.5; X; = 2.24; 1=0.8552
(8.4b)
l '
8.1 CONCEPTS IN DISCRETE PROGRAMMING 323
3
'
1
+
'
1
1
1

t
'

-
e
'
i
r
i
-
-
'
'

'
1
7
i
b
o
a
-
-
t
J
o
-
-
T

A
t

l
l

1
l
r

-
-
-

-
a

w
U

-
o
0.5

_ L _ _

0.5 1.5 2 2.5

Dlscrete vorloble ~
Flgure 8.1 Dlscrete nelghborhood 01 continuous solution.
3.5
X ~ =2; X;= 2 xi = 1.73; ] = 0.9158
(8.4c)
x ~ = 2; x ;= 2.5; xi = 2.24; ] = 0.3752 (8.4d)
From this exercisee 8t value of e objective function is available in Equation
(8.4d) and would be regarded as the adjusted optimum solution to the original M P
problem. lis is also the common practice in engineering design optimization.
T he best solution however is obtained at
x;= 1.7433; X;= 1.5; x;= 1.73; ]=0.1782 (8.5)
T he difference between Equations (8.5) and (8.4a) only in e variable XI suggests
t additional continuous optimization needs to be performed for each set of E
discrete values selected confirming the rlier observations by Fletcher. us
optimization should be easier as the order of the mathematical model w il be duced
by the number of discrete variables since they have beensigned numerical values.
This simple example commendsate very least a three-step procedwforproblems
incude discrete variables:
Step 1: A conlInuous relaxation that identifies several sets of discrete variables for
further explor on.
324 O I S C R E T E OPTIMIZATION
Step 2: For each such set of discrete variable combination a continuous
optimization is performed 10 establish a new optimum va1ue of the continuous
variables and the corresponding objective function. If all of lhe variables are
discrete then only the function and constraInts need to be eva1uated at each of
the sel of variables.
Step 3: A simple compison of the above solutions/va1ues in Step 2 to identify e
optimum solution to e discrete problem.
This unconstrained optimization example in three variables has demonstrated
that discrete optimization requires a lot of work compared to continuous
optimization. This work expands significantly if the number of variables increases
or if the mathematical model is enhanced through the inclusion of constraints.
Another essential f lre in the above exploration is no new mathematica1
conditions were necessary for stablishment of the discrete solution beyond a
simple comparison of the objeclive function. The nature of discrete variables and
discrele functions precludcs any sophisticatcd mathematical conditions
established by derivatives of the functions involved in tbe model. Trapping and
branching based on comparison of values are the mainstay of discrete algorithms.
There techniques are classified as heuristic methods. This encourages unique and
personal implementations of the search for discrete optimization that can h
tailored for a class of problems.
8.2 D I S C R E T E O PTlMIZATION T E C H N I Q U E S
There are three discrete optimization techniques in this section. The first one is
Exhaustive Enumeration. This involves identifying tbe solution to e
mathematical model for all possible combinations of the discrete variables. This
is suggestive of the zero-order numerical techniques of Chapter 6. Those
methods lackcd sophistication as they involved only evaluation of the functions
at a phenomenal number of points. They were able to take advantage of th
plentiful computer resources available today. Imagine operating in the
peer-to-peer computing environmnt afforded by MP3" or Gnutella" and
solving the problem using all of the PCs in the world. The second method is the
Branch and Bound method. This is based on partial enumeration where only part
of the combinations are explored. The remaining are pruned from consideration
because they w i11 not determine the solution. This is currently the most popular
method for discrete optimization for engineering design. The last method is
Dynamic programming an elegant approach to optimization problems but
which did not gain favor because it involved significantly larger amounts of
computation than competitive methods even for problems of reasonable size. It
is restricted to problems thal require a sequential selection of the design
variables. Today such resourcIimitations are disappearing in the world of
powerful PCs and hence lhe method deservcs lo bc revisited.
l
;;:;11

8 2 O I S C R E T E OPTIMIZATION T E C H N I Q U E S 325
Standard Format-Dlscrete Optlm/zatlon: T he author is not a WllI of a
standard format for discrete optimization problem. It subsumcs format of lhe
coponding continuous relaxation problcm. In this book Ihc fol1owing format fol'
the mixed optimization problem is used.
Minimize f(XY ) ]n<; [Y]n
d
Subjcct to: h(X Y) = [0]; [hJI
g(X Y) [0]; [g]m
J;S X j S .i;'; ;= 12.. . n
c
(8.6)
(8.7)
(8.8)
(8.9)
Yi E Y d ; [Yd]p ; i = 1 2. . . n d (8.10)
X represents lhe set of n c continuous variables. Y rep senls the set of n d discrcte
variables. f is e objective function. h is lhe set of 1 equalily constraInts. g is lhe sel
of m inequality constraints. Expression (8.9) rep:sents e side cons ints on each
continuous variable. Expression (8.10) expresses e side constraints on the discrete
variables. Each discrele variable Yi must belong to a preestablished set of Pi discrele
values Y t/I' If n d = 0n il is a continuous optimization problem. If 11c = 0cn it is
a discrete optimization problm. If both are p sentthen it is a Mixed Problem.
Contlnuous Relaxatlon: The continuous relaxon of lhe mixed optimization
problcm (8.6(8.10) is identica1 10 problcm (8(8.10) with Y as a continuous variable
t.lC discrensnl (8.10) is replac by a continuous side cons int for e
discrete variables of the form of (8.9). is not explicitly developed since (8.6)(8.9)
and a modified (8.10) is fairly representative of e continuous relaxed mathematical
model .
Reduced Model: The duced model is important in subsequent discussions. It is
e model solved after a part of the set of discrete variables are set at some alowable
discretva1ues. This removes lhose variables from e problem as ir valus have
been set. The mathematical model is then defined by the remaining discrete variables
w el e original conlinuous variables. If Z(n
z
) are e remaining discrete
variables need 10 be solveden
Minimize l(X Z) [X]" ; [Z]n (8.11)
Suect 10: (XZ) = [0]; [11]/ (8.12)
g(X z) ~ [0]; [i]m (8.13)
J;S Xj S xj'; i = 1 2 . . . n c (8.14)
Zi e z.t; [z.t): (8.15)
326 DISCRETE OPTIMlZATlO N
Toble 8.10 O pmal Volue ofCoououous Voriablex. for Discrete Combloauoo
fXloodx3
X21x) 0.2200 0.7500 1.7300 2.2 2.78
0.5 0.9067 1.0833 1.4100 1.5800 1.76
1.5 1.2O 1.4167 1.7433 1.9133 2.0933
2.5 1.5733 1.7500 2.0767 2.2467 2.4267
3.5 1.9067 2.0833 2.4100 2.5800 2.7600
Note t in the above 4 is not a n e w set . It co ponds to a reduced Y d.. In
Exhaustive Enumeration generally it is empty set . AII of the discrete variables
set to some allowable value.
8.2.1 Exhaustive Enumeration
This is esimplst of e discrete optimization techniquS. It evaluates an optimum
solution for all combinations of the discrete variables. The best solution is obtained
by scanning the list of feasible solutions in e above investigation for the minimum
value. The total number of evaluations is
/1. = IIp/ (8.16)
If eithrnd or Pi (or both) is largen is represents a lot of work. lt also represents
an exponential growth in thcalculations with number of discrete variables. In a
mixed opzation problem is would involve nc continuous optimum solution of
the reduced model. If the mathematical model and its mputer calculations are sy
to implement this is not a serious burden. If mathematical model quires
extensive calculations for exnplenite element or finite difference
calculations-then some concem m a y arise. In such caseslimiting the elements of the
discrte variables to smaller c1usters m a y rep sent a useful approach.
E xample 8 . 1 : b e exhaustive enumertionofExample 8.1 is available in Ex8_1.m.
1
Table 8.1 a shows optimum value of the continuous variable x I for every a110wable
combination of discrete variables X2 and x )Table 8.1 b shows corresponding
value of the objective function for each of the cases. Since is is an unconstrained
problem no discussion of feasibility is involved. There are 20 values and the best
solution for E xnple 8.1 is
x ~ = l.i433; X; = 1.5; x ;= 1.73; r = 0.1782 (8.5)
There is some good news though. First each of the 20 cases is essentially a
s;ngle-lIariable opzation problem. T he number of design variables in original
IFiles 10 be dowllloaded from Ihe web a indicalcd by boldfuce S Siflype.
8.2 OISCRETE OPTIMIZATION TECHNIQUES 327
Toble 8.1b Opumol Value of Objecuve Fuocuoo for Dlscrele Combiootion of Xz aod X3
x/X3
0.2200 0.7500 1.7300 2.2400 2.7800
0.5 1.9107 1.3542 2.7915 4.8060 7.8840
1.5 3.3240 1.3542 0.1782 0.8327 2.4707
2.5 8.0707 4.6875 0.8982 0.1927 0.3907
3.5 16.1507 11.3542 4.9515 2.8860 1.6440
model (3) is reduced by e number of discrete variables (2). Model reduction is
involved in enumeration techniques.
Algorithm: E. 'austlve Enumeration (A8.1)
Step 1. ! = inf X = [0 0 .. 0]
For every a1lowable combination of (Y.. Y2Y"d) = ( Yb)
Solve Optimization Problem (8.11)(8.15) (Solution X )
Ifh(X Y
b
) = [0] and
Ifg(X. Y
b
} [0] and
Ifj(X Y
b
) < !
T h e n f ft.X Yb}
X X.
Y Y b
End If
End If
End If
EndFor
Example 8.2 iIIustrates use of algorithm A8.1 with feasibility requirements.
Exmnple 8.2 The Aerodesign Club has initiated fund-raising activiti by selling
pizza during lunch in the student activity center. Over the y rs ey have collected
ta to identify a mathematical model that will r uce eir cost which directly
tr sla to higher profits. Only two types of pizza w i1I sell-pepperoni (x.) d
cheese (x21. The local pizza chain w i1I deliver a m a x i m u m of26 pizz. Factoring into
account wastage and the time available to make the sale ey arrive at the fol1owing
mathematical model:
Minirnize f(x.. X2) = (XI - 15)2 + (X2 - 15)2
Subject to: 8.(Xh X2): XI + X2 S 26
g2(X.x21: x. + 2.5x2 S 37
(8.17)
(8.18a)
(8. I 8b)
328 DISCRETE OPTIMlZAnON
XI ~ 0; x 2 ~ O. and integer (8.18c)
The graphical solution for continuous relaxalion is available in Figure 8.2 and the
analytical Solulion (not included) identifies that 12.86 pepperoni pizzas and 9.65
cheese pizzas will provide for the minirnum cost of operation.
Ex8_2.m is e m-file at will perform the exhaustive enumeration of the
problem. The discrete solution is
X; = 12; xi= 10; r = 34; g2 is active
(8.19)
This confirms tneighboring discrete values around a discrele solution works well.
Programming exhauslive enumeration is straightforward and particularly easy with
MATLAB. Given processing speed large memory availability on edesktop. sy
programming through software like M ATI..AB. etive enumeration is probably a
very good idea loday. Whal recommends it even further is the solution is a global
timum. FlIrthermore. unlike the earlier chaptrs where thconcepts of derivatives.
matrices. linear independence. and so on were essentia1 to understand and implement
the techniques of optimization lack of sucb knowledge is no disadvange to solving
a discrete optimization problem using exbaustive enumeration. T bere is only one
essentiaI skillIs1ating mathematical model into program code. With E
experience provided in is book. this is definitely not an issue.
40
35
g ~ = O
. .
- ....
6 i e - 7
5
10 15 20 25 30 35 40
pepperonl plzza
Flgure8.2 G phical solutlon Example 8.2.
;
8.2 DISCRETE OPTIMIZATION TECHNIQUES 329
e solution in Equation (8.19) can be obtained with continuous relonofE
problem. In this case the discrete solution has to be recovered by adjusting the
continuous solution to neighboring discrete values.
E xhaust#ve Enumeratlon for Contlnuous Optlmlzatlon Problems: This title
must be quaIified further-in lieu of gradient-basd numericaI techniques. The
availability of serious desktop computing power. the promise of global optimization.
and ease of implementation suggesl a multistage exhaustive enumeration strategy
for continuous optimi tion problems. Exhaustive enumeration after all is a search
strategy. similar 10 the ones used before. T he search for solution is akin 10 e
zero-order melhods employed for unconstrained optimization (Chapter 6).
The higher stages in multistage enumeration involve enumerating on decreased
intervaIs centered around e solution from previous stage. The vaIues of the
design variables used for enumeration will incorporate progressively improved
tolerance. The strategy must take into account the possibility that the initiaI interval
mighl overlook globaI optimal solutions tllat do not appear attractive in a particular
stage. Hence. more than one minimum is tracked in each stage. Each of these
minimum can be considered enllmeration cluster. Each such c1uster is multistaged
until a prescribed tolerance is achieved. In engin ring.m ufacturingtolerance of the
order of 0.0001 is achievable today. Tolerances below is limit probably need to be
justified unless e area is microelectronics. From a programming perspective it jllst
adds an extra loop to the implementa1ion. le final solution obtained represents a
continuous solution within the final lolerance implemented by the technique.
Such an exercise is lefl10 the student . As with all implementation the merit of lhis
approach lies in the tractabili1y of the mathematical model and the need for a global
optimum solution. Since the method is largely heuristic there is no guarantee that the
solution is actuaIly a global optimum (of course assuming the functions involved are
not convex).
8.2.2 Branch a n d B o u n d
Exhaustive or complete enumeration is possible only for a limited set of variables
because of the exponentiaI growth of effortqui d to examine aIl possible solutions.
Using partial (or incomplete or selective) enumeralion to arrive al the solution to the
discrele optimization problem would provide tlt necessary advantage in handling
large problems. Partial enumeration works by making decisions about groups of
solutions rather every singlone. T he 8ranch and 80und (88) algorithm is one
of the successIalgoritl1m s to use selective/partial enumeration. lt does so by using
relaxation models instead of the original comple1e discrele ma1hematicaI model . The
8 B algorithm uses a tree structure 10 represent cqmpletions of partial solutions. The
'eesucture uses nodes and edges/links 10 represenllhe trail of partial solutions. This
is callfatlromulg e ptial solution. The terminology is borrowed from operation
rese h literature [3] and is explained below wi resp to Example 8.1 which is
reproduced here for convenience:
330 DISCRETE OPTIMIZATION
Minimize f(XlX2X3) = (XI - 2)2 + (XI - X2)2 + (XI - X3)2 + (X2 - X))2 (8.1)
XI E R (8.2a)
X2 [0.5 1.5 2.5 3.5] E X
2d
(8.2b)
X3 E [0.22 0.75 l.73 2.24 2.78] X
3d
(8.2c)
Partlal Solutlon: A solution in which some discrete design variables are fixed and
oers arej'ee or undetennined. Variab1es that are not fixed are represented by e
symbo1 :" For example
x = u :0.5 (8.20)
rep sents the optimization prob1em wiX2 = 0.5 and XI and X3 to be determined. The
paal solutions ara1so termed the nodes of the tree. le free variables are usually
solved using a continuous relon of the model .
Compler/ons (0' P aTt Solutlon): Ench partial solution or nodcan givt: rist: lu
additionnl partia1 solutions/nodes directIy under it. This is called brallching or
expansioTl of lhe node. In this new set of nodes another one of frec discrelc
variablcs is fixed and e remainder slill continue to be free. For example under
node represented by the partial solution in Equation (8.20) five additiona1 nodes are
ssible:
X = [ f 0. 5 0.22]; X = [ f 0.5 0.75]; X = [ f 0. 5 1.73];
X = [ f 0.5 2.241; X = [ f 0. 5 2.781
Thesc completions branch out and hcnce a treelikc structure. There is a hierarchical
structuring of the nodes. Each branch represents 1eads to a tier in which some variable
is held at its prescribed discrete value.
Fathoming the Partial Solutlon: Expanding the node all the way till the end is
termed fatlwming a node. During the fathoming operation it is possible 10 idenlify
those partial solutions at a node that do not need to be investigated further. Only the
best solution at e node is branched or completed. T hose partial solu t io nsll Ino odes at
will not b e investigated further are pruned 0 rtenninated i n e lIC.
Actlve Partlal Solution: A node thal has not been tenninated or pruned and has
nol bcen completed.
EdgeslLlnks: Thes'enode-tonodeconnections in a. They identify the value
_ r 1 . . _ . . ~ . . ..n_ L '....... :........J
8.2 D1SCRETE OPTIMIZATION TECHNIQUES 331
Root: ThB B algorithm searches a10ng a IIC. lt starlS al the root where a11 the
23t:31?1 J: :;;:??J22::
preferred to jumping across nodes in esame erarchy.
Incumbent softJMont The incumbent solution is thsbGSt feasible solution M o w n SO
far with a11 discrete variables assuming discrete values. When lhB B search final1y stops
EhalinCUREbBEEtsolutionifomexistsis a ziobal optimum soitHion-h thefollowing
theincumbentsolution is denotedbyr--13mincumbaudESign vmables X...
E xample 8.1- Uslng Branch a n d Bound: Here e B B search is applied
Example 8.1.BBEX8JJn calculates allof thE data needed for genEraIing
T he B B itse1f is developed in Figu s 8.3 and 8.4.
Start: Node O. All ofvariab1esare fC.The solution to the continuous relaxation
ofthe obj tive function (solved as a continuous problem) is identified at the
node.
Nods1-4: This is thfirst tier of the B B tre The discrete variable X2 is assigned
Each of thgpennisBiblediscrete values at the nudcs in this tier-This ES indicated
X=[fffJ
X* = [1.741. 5 1.73]
p =0.178
p* = 0.178lncumbenl solutlon
XoI". = [1.74 1. 5 1.73J
Flaure 8.3 Branch and bound Example 8.1.
332 DlSCRETE OP nMIZATION

X" = [1.57 2.50.22]
P=8.07
X" = [1.752. 5 0.751
P=4.68 xe=I107zs1.731 x'=I2.26zaz.Ml
P = 0.898 P = 0.192
F1gure 8.4 Branch and bund Example 8.1-fathoming Node 3.
on the edges between the nodes in Figure 8.3. The remaining v iables B
frIcontinuous. The optimum solution at the various nodes is displayed on E
tree. The minimum solution is the one at will be fathomed at the next step.
Here Nodes 2 and 3 - e candidates for expanding the branches. Nodes 1 and 4
will not be fathomed (unlcss their optimum value is better an Ihe incumbent
solution)ey will be terminated or pruned e objective function has a
higher valuc.le continuous relaxation ensures a lower bound on the optimum
value of the objective at the node. Node 2 is taken up for fathoming based on
node ordering
Completions of Node 2: This expands Node 2 by setting e discrete variable X 3 10
its permissible set of discrete values. It creates branches Nodes 5-9. lt is also
the second tier of the tree. The solution is indicated at e various nodes. In all
of them the two disc te variables are set 10 indicaled values while Xl is free.
which is a continuous variable by definition. Since all of the discrele variables
bave discrete values this node is also fathomed. Node 7 indicates the best
solution. Since all of e disc te variables ar assigned disc le valuesE
solution at lhis node is the current incumbent solutionThe remaining nodes will
be pruncd. The currenl incumbent solution (f. = 0.1782) is higher
solution al node 3 (f = 0.15). This requires Nod3 10 be fathomed.
Completions ofNode 3: Figure 8.4 iIlustrates the branching at Node 3. The process
is similar 10 completions of Node 2 above. The best solution is at Node 13 with
a value off = 0.1927. This is largerthan the Current incumbent solution. AlI of
e nodes can be lerminaled.
N o further nodes are available for expansion. The current incumbent solution is
also the global optimal solution. This is the solution to e optimization problem. In
8.2 DlSCRETE OP nMIZATlON TECHNIQUES 333
is example 15 partial solutions were n essary. Each partial solution can be
considered an enumeration. In exhaustive enumeration 20 numerations were
required for Example 8.1. There is a nel gain of 5 enumeralions. T h B B search
redu sthe number of total enumerations needd 10 solve e problem.
Algorlthm: Branch a n d B o u n d 8.2): For even a small set of discrete
variables actually drawing th B B tree struclure is most likely a difficult enterprise.
The 11'ee structure merely idenlifies the search process. Example 8.1 did not have
constraints so feasibility was nol examined. With constraints present choosing the
p tial solution to complete based only on minimum value of the objective function is
not a justifiable s tegy. In this case the besl feasible solution is e candidate for
completion. Also an active feasible partial solution has a better value
current incumbent solution is also a candidate for completion. T he following
algorithm capres essence of the Branch and Bound algorithm.
Step O. Establish the root node with all discrete v iables .
lnitialize solution index s to O.
. .
Sel current incumbent soIution to f-- = "" X-- = [] (minirnization
problem).
Stcp 1. If nny active feasible parua1 solutions remain ( is beuer than the current
incumbenl solution) selecl one as X<J) and go to Step 2.
If none exists and there exists an incumbent solution il is optima1.
If none exists and the is no incumbent solution the model is infeasible.
Step 2. Obtain e completion of the partial solution Xusing a continuous
relaxation of the original model.
Step 3. If there e no feasible completions of partial solution X terminate
X<I) increment s s + 1 and retum to Step 1.
Step 4. If the best feasible completion of partial solution X<I) cannot improve the
incumbent solution then terrninate X increment s s + 1 and retum to Step
Step 5. Ife best feasible completion is better the incumbent update the
incumbent solution
r.!<I) x..x
terminate X<s> increment s s + 1 and retum to Step 1.
E xample 8.2 The Branch and Bound algorithm is applied to Example 8.2. The
Sh is applied through the B B . le mathematica1 model for Example 8.2
whereXland X 2 are integers is
Minimize !(x" x v = (Xl - 15)2 + (x2 - 15)2 (8.17)
334 DISCAETE OPTIMIZATION
Subject 10: 81(X" X2): X( + x 2 :!> 26
82(XIX2): X( + 2.5x 2 :!> 37
XI ~ 0; X 2 2! 0
The solution is
(8.18a)
(8.18b)
(8.18c)
X;= 12; X;= 10; f= 34; g2is active (8.19)
i z : ; 2 2 J 3 1 fe ? L i t z5 2 2 e i f z : : i : : z z t s z
following development
X
1
E [10 Il 12 13]
E [9 10 11 ]2)
Uslng Branch a n d Bound: e B B e is shown in Figure 8.5. The data for e
CcuJat through B B E x 2 . m where the continuous l ation is solv
analyticaIly using thc K T conditions.'he tree is brnnched ns follows:
X Co:::[139)
1'* =40
X. = [12.86 9.951
1'* :::33.13
1nfeaslble
1nf slble
Rgure 8.5 Branch and bound Ir for Example 8.2.
Infslble
8.2 DISCAETE OPTIMIZATION TECHNIQUES 335
NodeO: is the root node. All the variables e fr. econtinuous relaxation
of the problem is solved. T he solution is indicated on tree. Currently this is
the only available active partial solution and hence needs to be completed.
Nodes ]-4:epartial solution at Node 0 is completed using variable XI from the
reduced discrete set identified above. This leads to four branches and the
optimal values ar indicated on the . The best values are indicated at e
nodes. For isproblem .y are also feasible. There enow four active partial
solutions.
Node 3: The so]ution at Node 3 is also a feasible solution of the original discrete
optimization problem since the design variables are discrele and belong to e
permissible discrete set. Node 3 isfatJlOmed. is provides the first incumbent
solution.
Thre are three available active feasible partial solutions (Nodes 12 and 4)
for completion. Since the solutions at e nodes are all feasible only the best
active feasible partial solution is picked up for completion-Node 4. Nodes 1
and 2 can be terminated.
Completion Node 4: Node 4 has a feasible solution better e current
incumbentsolution.ltis completed by branching 10 Nodes 5 6 78. Only Node
5 has a feasib]e solution. Tbe besl solution al Node 5 is not better than the current
incumbent solution. Nodes 5 6 7 8 can be terminnted.
There are no more active paal solutions and ehas becn fathomed. T he
current incumbent solution is the optimal solution.
lis completes the B B search applied to Example 8.2. The duction in
enumeration compared to exhaustive enumeration is difficult 10 comment on for
two reasons.
1. The deliberate choice to restrlct ese1 of discrete values used 10 branch the
seemed arbitrary. While is was done for convenience a serious justification
n be advanced based on the continuous solution at Node O. The reduced
discrete set is chosen to include a limi1ed neighborhood/window of the
continuous 1 on.
2. The exhaustive enumeration based on disctevalues was easierto evaluate for
a pure discrete problem. Applying K T conditions to obtain the possib]e p tial
solutions to e continuous relaxation and choosing the acceptable solution is
an involved exercise. Using numerical lechniques Iike SQP 10 solve the
problem which w il1 mosl Iikely be done in practice has its own problems.
Dependence on good starting values wino guarantee of convergence requires
user inlervention 10 identify e particular solution. It would requir a lot of
effort 10 automate the pro 5S so as 10 apply e Branch and Bound algorithm
10 general nonlinear mixed optimization problems. Linear mixed optimization
problems do not possess is limitation at least to the same extent.
336 DISCRETE OPTIMIZATION
In is examplee active partial solutions at the root and in first tier were all
feasible. What would be the strategy if the root nodorother nodes have an infeasible
active partial solution? In case e 8 8 search process must complete all active
partial solutions and not only feasible active partial solutions. That is a modification
at can be easily made to the 8ranch and 80und Algorithm above.
Other Search Techniques: le two search techniques in is section nnely
exhaustive enumertionand branch and bound provide an additional feature of
discovering global optimum solutions.'hey also belong to the c1ass of methods used
to address global optimum.be sy availability of large and faster computing
resources has brought renewed emphasis to globally optimal designs. SimuUlted
annealing and genetic algorithms ar among the leading candidates for global
optimi tion. ley e covered in the next chapter and can also be characterized as
enumerion chniqueslike ones in is section. 80th of them can and are used
for discrete optimization. They 'e not discussed in this chapter. The application of
most of these techniques is still heuristic and often depends on experience based on
numerical experiments among c1asss of problems. Some applications are problem
specific and require user intervention and learning. Standard implementations in ese
methods. especially for nonlinear problems are nol yet the norm.
8.2.3 Dynamic Programmlng
Discrete dynamic programming (DDP) is an exciting technique for handling a special
cass of problems. Examples 8.1 d 8.2 are not direct members of this cass. chard
8ellman [4) was responsible for tin ducing e concept and the algorim. It is an
opzing produre based on Bellmansprinciple of optimality. The principle is based
on a sequen ofdecisions on partial solutions. so at when ccrtain variables have been
determined. the remaining variables establish an optimum completion of the problem [5).
A noer expression of this nciple is obined from Reference 6. A n optimal sequen
of decisions he property that at e crent timc. whatever thc initial state and e
previous decisions. E maingdecisions must be an optimal set with regard toestate
m ng from the first d sion. D D P is about a sequence of decisions (often in time) and
is sometimes term asequential dsionproblem. Problems I ike Examples 8.1 d 8.2
do not directly fitisclsifi tion but they n bsformed to fitisruirement (5).
Very often. D D P problerns can be described as optimal pa determination problem.
Such Ph problems are solved using directed graphs (digraphs). lese problerns may
actual1y have no connection to physi Jpaths or distances. Like the 8 8 treeycnable
better appreciationlapplication of the algorithmic produre and are not sictly quired.
DDPrusdefition of s.tes and stages. and only the former is discussed he. le
equationsed to esblish the principle of optimality are ))edUlctiollal equations.
Example 8.3 will be used to define enewterms.
Example 8.3 A n established university is inlsted in developing a new Co))gefor
Information Technology during the currenl ye. Table 8.2 ilIustrates data
2 DISCRETE OPTIMIZATION TECHNIQUES 337
Table 8.2 Faculty H1ring Pollcy Data for Example 8.3
ltem Fall Winter Spring Summer
Ruired Faculty 5 10 8 2
Recrui entCost 10 10 10 10
Unit Facu1ty Cost 2 2
UnitRening Cost 2 2 2 2
necessary to calculale the cost of recruiting faculty foris enterprise for the first ye
of opern. The college will operate during four quarters (FaU. Winter. Sp g. d
Summer) indicated by ecolumns. Decisions are associated with the beginning of the
quarters. The first row indicates the new faculty required to implement the coursat
will be offered. The second row is e recruitment cost for new faculty each quarter.
which is significant because of e competitive dmand for qualified people in this
. le ird row is e unit cost of new faculty in terms of the standard institut
faculty cost . Course specialion and availability makes this cost different in
different quarters. The last row is unit cosl for retaining extra faculty hired in E
earlier quarter (to minimize cost over the year). What should be the hiring policy for
universily lo minimize cosl?
lis example is c1early il1ustrative of sequential decision making. Decisions are
requd to be made every qu er. For example it is possible 10 cruit all needed faculty
during e first quarter and pay B ning cost. or it may cost less to hire during ch
quarters and hire again in efo. le problem will be
solved using optimaUshortestpath method even oughthe arenorou involvedin
.eproblem.bebest ways to understandeshortestphalgor m is to draw a directed
graph (digraph) betw n the nodeslstates in the problem.
Stes: States present pointslconditions in D P where decisions have to be
considered. T hese are sometimes 'ed to as policy decisions or just polici. T his
has a suentiaJ connection as current decisions cannot be nside d UDe
previous decisions have been made. Clearly in Example 8. 3 present e
beginning of each quarter. le number of states is usually one more thenumber
ofdcisions to incorporate the effect of the last policy/decision. Nodes in the digraph
rep sent states in the D P problem. The digraph represents a connection between each
pair of states at is permissible. This can be made c1earer by stating tif a node is
going to be affected by a decision at any previous nod. then there is a direct
connection (Iinelarc) between enode. The cost of the connection is indicated on E
connection itself. A n arrow can be used to indicate the direction of traverse along is
linelarc segment. These Iins/arcs erefore represent decisiofls. Once the digraph for
the D P problem is available. the optimum solution correspoflds to the shortest path
from Ihe beginnillg 10 the ending state ofthe digraph [3).
338 DISCETEOPTIMIZATION
99
74
Flgure 8.6 Digraph for Example 6.3.
Figur8.6 is digraph for the hiring policy. There are five states with the first
four representing e qu where the decisions must be made. Node 1 Psents
the Fall quarter and 50 on. Along the arcs on e digraph is thC05t of the deci5ion or
policy at w iJI originate at the current node and meet a11 of the requirements at e
lrnode. In order 10 usthe information in the digraph efficlivelyand 10 develop
Ihe mathematical model the quarters are assigned indices from 1 to 4. The following
abbreviations are used: required facullY RF; cruitment COSI RC; unit facully cost
FC; unil relaining cost - holding cosl H C . The policy decisions or cost between the
start node i and the end nodej is C(ij). For example. C(l 4) rep sents e policy
the Fall q rterwhich will meet e staffing reqrements for e Winlerand
Spring quarters a150 thus paying e appropriate retaining pena1ties:
C (l4) = R C(l) + [RF(l) + RF(2) + RF(3)).FC(l)
+ ([RF(2) + RF(3))H C(l) + [RF(3)]HC(2))
Ex8_3.m is the m-file at generates the data for the digraph for Exarnple 8.3 d
identifies e 50lution by solving thefunclional eq lions and backtracking epa
to identify the optima1 squence. Note at for differnt problems on1y the cost
calculation w iJI differ. The functional equalions and backtracking elements of the
D D P method and should be the same.
Functional Equatlon for Dynamlc Programmlng: The functiona1 equation is
the essential part of the D D P a1gorithm or method. It employs a recursive pro du
10 identify the besl solution al each state. Lel Value (i) be thbest value of the decision
al node i. laking inlo account a11 possible arcs/paths thal lead to node i. Let Node (i)
be the node led to the best policy at e current state. T he recursion relations for
IJ states can be sel up as follows:
8.2 DISCRETE OPTIMIZATION TECHNIQUES 339
Value (1) = 0
Value (k) = min (Value (+ C(/k); 1 S 1 < k} k = 2 . . . 11 + 1
Forexampl in Example 8.3
VaLue (4) = min (Value (1) + C(l 4); Value (2) + C(2 4); Value (3) + C(3 4)}
Value (4) = min ([0 +85]; [15 + 62]; [45 + 26]) = 71
Node (4) = 3 (which establishd the minimum value)
T hese ca1culations are embedded in Ex8_3.m. T he output from the program IS:
E xample 8. 3 -D y na m i c programming
C o s t between p a i r s o f nodes/states C (i j )
o 15 45 85 99
o 0 30 62 74
o 0 0 26 34
o 0 0 0 12
0 0 0 0 0
odes - Values (i)
optimal p a t h values a t v a r ~ o
o 15 45 71 79
B e s t previous n od e o reach current n o d e -Node(i)
0 1 2 3 3
The Value (4) is 71 alld Node (4) is 3. The optimum value for the problem is Value
(5) which is 1Ua1 10 79.
Shortest Path: The shortest path is obtained by backtracking from tlle end stat
ing the Node (i)infonnation-which is the row of data above.
At NodB5thEbest nodEfrom which to reach Node 5is Node3
o 2 3 3
AINod3 the best node from which to reach Node 3 is Node 2
o 2 3 3
At Node 2 the best node from which to reach Node 2 is Node 1 (this is e startmg
node)
o 1 2 3 3
This solution is indicated by e thick soIid line in Figure 8.6.
340 OISCRETE OPTIMIZATION
n e acaIh li iri tnIgp olic cya .cco orldngto issolu tio nistohi ir e sttherqu iI1 edf: acuIty
for FaJlqu ter the FaJl qu ter. D o the same for the Winter quarter. For the
Spring quarter however hi for both the Spring and Summer quarters.
Alternate Paths: There is an alternate solution for thc same optimal value. The
output from Ex8_3.m includes is information though it was not included above. In
eoutput in the command window each row contains additional nodes at will yield
same optimal vaJue.leonlyeny is 1 in third row and second colurnn. This suggests
that e thiI1d node Node (3) can o have the value of l-that is Node 1 also
provides the same vaJue to reach Node 3. T his will cause e Node (i) information to
change as

3 3
The shortest pa Cen be interpreted as follows:
At Node 5 the best node from which to reach Node 5 is Node 3

3 3
At Node 3 the best node from which to reach Node 3 is Node 1
o 1 3 3
In Figure 8.6 this is indicated by the thick dashed arcs. The hiring policy n o w would
be to hire for FaU and Winter in Fal1. Then hire for Spring and Summer in Spring.
Dynamlc Pro rammlng a
The fol1owing algorithm is identificd in-Refcrencc 3 for pas that do not cycle. T he
original 8 el1man algorithm incJudes an iterative finement that allows for cyclic
pas. In Example 8.3 ere are no cyclic pas. This algorithm is implemented in
Ex8_3.m.
Step 0; Number nodes in sequential order. Arcs ar defined for j > i. Set the start
node S optimal value as
Value (s) = 0
Step 1: Terminate if all Value (k) have been calculated. If p represents the last
unprocessd node. then
Vallle (P) = min{ Va/ue (i) + C (i. p): if arc (i p) exists 1 S i < p}
Node (p) = the number of Node i that achieved the minimum above
Return to Step 1
8.3 AOOITIONAL EXAMPLES 341
This was a simple example of DDP. It is important to note at solution in final
lue (incJudes on1y the best solution at all states/nodes. Like other techniques
the' c several different categories of problems addressed by DDP. incJuding special
discrete problems like the K napsack problems. It is also possiblto bring Examples
8.1 and 8.2 under the framework of DDP. though to impart a sequentinaturlin the
process of selecting design variables in this case m a y involve a to transformation of
eproblem. Even then it m a y not be efficient to solve it using DDP. In practice D D P
has always becn regarded as computationally complex and prohibitive. It is seldom
used for problems wi a large number of states or if the estimation of cost is
intensive. Another criticism of D D P is at it caes out a lot of redundant
computation for continuous problems. Today. these are less of an issue in the
past. For stochastic and discrete problems D D P may sti11 be an effective tool . 8 el1man
has also indicated that it c be uscd to solve a set of unstable systems of equations.
D D P also compus design sensitivity as part of its solution. T his section was
primy meant to introduce D D P as a useful tool for some categories of problems.
Readers arc encouraged to consult e references for more information on esuect .
8.3 A D D I T I O N A L E X A M P L E S
In this section. two additional examples are presented. The first one is an engineering
cxnple from mechanical design und illustrates h o w the disc te version of a
four-variable problem very effectively reduces to a single-variable problem atcan
be easily adsed by enumemtion. A n unbelievable reduction in complexity
yielding global optimization is obtained by elementary calculations. It definitely
invites scrutiny regaroing Eort in solving continuous nonlinear optimization
problems. T he second one is e application of the bmnch and bound search to a 0-1
intcgcr progmmming problem. Much of the application of discrete optimization is in
e 'ea of decision m ang. and decisions like for or against. yes or no are possiblc
using a 0-1 linear integer programming model.
8.3.1 Example 8.I B e a m Deslgn
his cxample is a structura1 design problem. In facit is another variationlapplication
of the I-bcam structure seen rlier. T he continuous problcm was developed and
solved as part of a student project . In is example e beam is used as a horizontal
member in a fixed E" series gan cmne. T he hoist with a capacity of W = 2000 Ib.
is at the center of the be 1. T he objective isdesign a minimum mass ben that will
not fai1 under bending. shcar and spccified detlection. T he length L of the beam is
specificd as 96 in. Steel was choscn for thc material of e ben. Thc modulus of
elasticity of steel is E = 29 E+06 psi . T he weight density of s Iis P = 0.284Ib/in
3

T he yicJd s ng in on is 'y = 36 E+03 psi. T he yield strengin shear is y=
21 E+03 psi.esp ified allowable detl tiOD is 0.25 in. A factor of safety of 1.4 on
the load is assumed. Figure 8.7 iIIus tes the cenc loading on the beam as well as
the design variables. (A11 calculations are in inches and pounds.)
/
/
.:....:<<'1
x
/
/
/
343 8.3 ADDITIONAL EXAMPLES
DISCRETE OpnMIZAnON
L

342
The Dlscrete Problem: The discrete problem shares the same mathematical
model except the values for the design variabls will belong to a set of discrete
values. It is possible to use ith complete enumeration or the B B search to
solve problem. Before proceeding further in this direction it is useful to
recognize the reason for discrete optimization is to choose an off-the-shelf
I-beam which will keep the cost and production time down. Several mills provide
information on standard rolling stock ey manufacture. Reference [7] is used to
collect data for this example. lf a standard S-beam is needede reference lists
several beams with acal values for the design variables XIX2X3X4Choosing
an I-beam from the list also identifies all of the design variables. This becomes a
single-variable problem the variable being the particular item from the list of
beams. In this case there are no geometrical constraints necessary and neither are
e side constraints. A complete enumeration can be performed on the candidale
beams from stock list and the best beam can easily be identified. Compare this
enumeration to solving the nonlinear mathematical model with scven functional
constraints. W h e n such an opportunity arises exploiting the discretc optimization
problcm is highly rccommendcd. O n the other hand consider a different problem
where a plastic I-beam w i1I be molded in the manufacturing faci1ity. Hcre the
continuous problem needs to be solved for the minimum cost.
Ex8_ 4.m w iIl perform a complete enumeration of this example. It includes a
truncated stock list from which the fmal solution is obtained. The programming is
straightforward. T he output in the Command window is
X : = O 1" =48.239 x; =0.1563; X;=2.; = 5 28;
Optimum B e a m is :W6X9
Mass of the Be am 71.543 (lb)
Depth Width W e b Thickness Flange Thickness
5.900 3.940 0.170 0.215
Constraints: Deflection Normal
(<= 0)
-2422048221.600
Solution found
(inches)
Stress Shear Stress
-86086.261
Note that e solution is significantly higher an the one obtained in the continuous
relaxation once again illus ting the continuous relaxation provides a lower
bound on the solution.
v
-
M
Y

'til
ZX4
(normal slress)

1
1
1
1
(defl tion)
F

T he mathematical model for this problem can be exp ssed as:

Minimize
Subject to:
Design variables: Example 8.4.
f. i .X) = p LA .
gl(X): W U - 0
g2(X): W l x
l
-
g3(X): W - 21X3y S 0
Figure8.7
(shear s 5S)
Some geomctric constraims to relatc the various design variables ar
g4(X): xl - 3X2 ::;; 0
g5(X): 2x
2
- xl ::;; 0
g6(X): x3 - 1.5x4 0
g7(X): 0.5X4 - x3 0
T he side constraints on e design variables are
0.25 ::;X
4
::; 1.25 0.125 ::;x
3
S 0.75
2::;x
2
::;15 3::;x
1
S 2 0
-2468680.570
The following relations defme e cross-sectional a a(Ac) moment of inertia and
first moment of area about the cen id (Qc):
Ac =2x
r 4
+X1X) - 2X
4
1 = _ (x
2
- X
4
)(X
1
- 2x3)3
= - - - -=---'--"'------::
12 12
Zern e Integer Programmlng
Many L P programs/models have special features which are usually exploited tbrough
special algorithms. A n important class of L P problems require e variables be binary
that isy can have one of two values. T hese problems can be set up so values
8.3.2
Qc
Thc solution to the continuous relaxation model is
345 8.3 ADDITIONAL EXAMPLES
:
DISCETE OPTIMIZATlON 344
Capi1Budgeting Example Table8.3
V aJue Year3 Year Yearl Group Item
A
u
q
J
A
U
Z
J
h
u
e
J

&

3
'
a
n
u
n
u
-
E
n
u
'
A

J
U
-
E

3
n
U
4
4
3
2
1
2
5
0

Systems
Microsystems
Software
Modeling
M Is
Photonics
a

T
Z
J
Z
O
3
X " X6 [Oor 1]

-
4
0

x

+

a
x

+
3

3

Z
+
5
b

+
+

8 10 Budget
i = 1 2... m
u

g

--J
X

a

n
T
'
M
M
F

Minimize
Subject to:
Note is is not a minimization problem as requid by the standard format .
A orithms: .ere are severa1 algo m s . 1 have been developed for solving ZIP
models. T hey all use implicitJselective enumeration rough Branch and Bound sh
together with continuous relaxation. However the binary nature of e variables is
exploited when completing the partial solutions or versing the trl. le Balas
Zero-One Additive AlgorithmlMethod is one of commendiques for s
c1ass of problems [6]. T he algorithm from same reference is inc1uded below.
Xj = 0 or 1 j = 1 2 . . . n
Except for the binary sidconstraints on e design variablesis represents e
mathematical model for an L P problem (the standard model [1res equality
cons ints only). Most algorithms assumeM positive.If any cj is negative then
1is sllbstitllted for xJ in eproblem.
Several varieties of optimizationJdecision problems involve binary variables. They
inc1ude the knapsack (a pure ILP with a single main constraint) capital blldgeting (a
multidimensional knapsack problem) assembly line balancing matching set
covering and facility location and so on. Some examples are detailed below.
for e variables are zero or one--defining a Zero-One Integer Programming
Problem (ZIP). The general form of a ZIP is
f(X): :2
}=1
Example 0 'a Knapsack Problem: Minimize e nllmber of pennies (X1)
nickels (x21 dimes (X3) and quarters (X4) to provide correct change for b cents:
X1 +x2 +x3 +X4
X1 + 5X2 + IOx3 + 25x4 = b
'e mtegers
Minimize
Sllbject:
X " X2X3X4
In e above problem the unknown variables are not binary variables but e integers
noneeless.
A orl m : Balas Zero - On e Additive (A8.4)
The enumeration is acked by two vectors U aitd X. Both of length n. lf a partial
solution consists of1.. sassigned in order the values of X corresponding
toeassigned variablsareOor 1. 'he remaining v iable'ee and have the vallle
-1. T he componenof U are
I i fk = J a nditsco m pIlemn Ithasno tbcn con ere
II llk= :k = 1 or 0 and its pJem h een consid d
o I ifk>s
Step 1. If b :2: 0 optimal solon is X = [0]. OtherwisU = [0] X = [ J].f. = 0 0
X. = X (incumbent solution)
Step 2. Calclllate
Example 0 'Capital Budgeting: le College of Engineering is considering
developing a unique Ph.D. prograrn in Microsystems in the next thrl years. This
expansion calls for significant investment in resources and personnel. Six independent
focus areas have been identified for consideration a11 of which c not be developed.
Table 8.3 pI1CSents the yearly cost associated with creating the various grOllpS over the
e y rs. le last line in e table represents the available blldget (in millions of
dollars) per year allotted to this development.le last column is e vallle (in
$100000) the group is expected to generate in the next 5 years.
The capital budgeting problem llses six binary variables (Xh . X6) one for each
item in e table. The ZIP problem can be fonulated as
J is the set of assigned variables
y= bj - : 2 a
20X1 + 3X2 + 30X3 + 5X4 + IOxs + 5X6
3X1 + 2x2 +x3 + 2x4 + 5xs S 10
Maximize
Subject to:
346 O I S C R E T E O P T I MA T I O N
y = min Yi' i = 1 2 . . m
I=L
C
fj
Ify O d 1<1' then I =J and X =X G o to Step 6
0 erwise continue
Step 3. Create a subset T of-ee variables defined as
T=ij:f+ <1aij < 0 for i such aty/<O)
1fT = o (empty set) then G o to Step 6 EIse continue
Step 4. Infeasibility test:
If i such that Yi < 0 and
Yi - L min(O aij) < 0
j G T
G o to Step 6 EIse continue
Step 5. B aIas branching test: For each free variablecreate the set
= (i: Yj - aij < 0)
If aII sets M j are empty G o to Step 6. Otherwise C aIculate for ee
variable
= -aij)
ieM}
where = 0 ifthe set is empty. Add to the current partiaI solution the
V iableXjat maximizes G o to Step 2
Step 6. Modi U by setting the sign of the righ ost positive component . A lI
elements to the right are O. Retum to Step 2
If there are no positive elements in U e enumeration is complete. The
;_ _ . d I
optimaI solution is in the incumbent vector X-. The objective is in.r.Jf.r is
there is no feasible solution.
Balas.m is e m-file implements the B aIas algorithm. It was translated from a
PascaI program available in Reference 6. It is a program and prompts the user for the
problem information-n m A b c. le test problem given in Reference 6 is included
below.
Exampk8.5
Minimize f(X): lOx. + 14x2 + 2lx3 + 42x
4
A
l ~
~ ~
8.3 A OOlTlO N A L E X A M P L E S 347

T
i
-
-
v
M
W
/

P
Subjectto: -8x 1 - llX2 - 9X3 - 18x4 S -12
-2x.-2x
2
7x3-14x4S-14
-9x. - 6x2 - 3X3 - 6x4 s-lO
X.. X2' X3. X4 E [0 1)

M
m
r
d

l
i

5
U
H

h
Theoutput omBalas.m:
Solution exists
~
- - - - - - -
The variab1es are:
1 0 0 1
The objective function 52.00
constraints - Va1ue a n d Right h a n d side
-26 -12
-16 -14
-15 -10
T his corresponds to the solution identified in Reference 6.
Complete Enumeratlon: T he Balas aIgorithm performs p ~ a ~ enumeration. The
complete or exhaustive enumeration (n.) for n binary variables is given by the relation
ne=2n
TypicallyerIteEprise decision making involves several hundred variables and linear
models. O n the 0erh dengin ring problems characterized by a nonlinear model
may have about 20 binary variables mquiring ne=lO48576enumBEaIions-This is-no
bi'deal with desktop resources today.KEep in mind complte enumeratlon
avoids any relaxation solution (purely discrete problems only).
3
i
-
i
j
i
i

C aI lSimplex.m: is a function m-file that c be used to solve L P programs. It
c m beusdtoobtain solutions to L P mlaxation models during the B B SEarch.It is
based on the aIgorithm available in Reference 6. It is translated from a Pascal program.
le mathematicaI model is
Minimize c
T
x
Subject to: [A]] = [b]
x ~ o
Slack variables are incJuded in e coefficient matrix [A]. Greater or equal to
constrnts c be set up with negative slack variables as the program will
348 OISCRETE OPTIMIZATION
automatically execute a dual-phase simplex method. Another altemative is to e
L P programs provided by MATLAB. T he function stalement for using Call
Slmplex.m is
function [xsol f] CallSimplex(AB C M N)
T o use the function. A. B. C. M. N have to be defined. M is numbcr of constraints
and N is the number of design v iables.
R E F E R E N C E S
1. Fletcher. R.Praclical Melhods olOp'imizalion Wiley New York 1987.
2. PapanbrosP. Y. and Wilde J. D. PlCiples 01 O p l i ~ I Design-Modeling and
ConUlalion. Cambridge University Press N e w York 1988.
3. Rardin R. L. Oplimilalion in Operalions Researcl J Prcnlice-Hall Englewood CIfs
NJ1998.
4. Bellman R. Dynamic ProgranungPrinceton UniversiLy Press Princeton NJ 1957.
5. White D. J. Dynamic Programming Holden-Day San Francisco 1969.
6. Syslo M. M. Deo N. and Kowalik J. S. Dlscrele Oplimization Algorllhms
ntice-HallEnglewood Cliffs NJ. 1983.
7. Ryerson Slock Sleel Lisl. Joseph T. Ryerson Inc. 1995.
P R O B L E M S
8.1 Translate Example 1.1 into a discrete optimization problem and solve by
exhaustive enumeration.
8.2 Translate Example 1.2 10 a discrete optimization problem and solve by
exhaustive enumeration.
8.3 Translate Example 1.3 into a discrete optirnization problem and solve by
exhaustive enumeration.
8.4 Implement a multistage exhaustive enumeration for thc continuous
optimi tion of Example 1.1.
8.5 Transla Example 1.1 into a discrete optirnization problem and solve by branch
d bound method.
8.6 Translate Example 1.2 into a discrete optimi tion problem and solve by branch
and bound method.
8.7 Translate Example 1.3 into a discrete optimization problem and solve by branch
and bound method.
8.8 Identify appropriate stk availability for the flagpole problem in Chapter 4 and
solve the corresponding discrete optimization problem.
8.9 Solve Example 1.3 using stock information available in Ex8 _ 4.m.
PROBLEMS 349
8.10 Y o u are the owner of an automobile dealership for Brand X cars. Set up a D P
for monthly inventory control based on cars expected 10 be sold. high ordering
costs. unit costs for the car. and inventory costs for excess cars. Solve it using
the code in Ex8_3.m.
8.11 Solve thc following using Balas algorithm-using completc cnumeration
Minirnize f(X) = XI + X 2 + X 3
Subject 3 x I + 4 X 2 + 5x3 8
2x
1
+ 5x2 + 3 X 3 ~ 3
X. X2. X 3 [ 0 1]
8.12 Progr1 an exhaustive enumeration techniquc for zero-one IP using MATLAB.
Solve Problem 8.10.
8.13 Is it possible to solve the capital budgcting exnple? If so what is the solution
using complete enumeration?
8.14 Solve capital budgeting example using Balas program.
8.15 Create an example which involves making binary choiccs regarding your
participation in course and extracurricular activitics 10 maximize your grades.
Solve it .
9
G L O B A L OPTIMIZATION
This chapter introduces two of the most popular techniques for finding the best value
for the 0ective function also referred to as the gIobal optimum solution. They are
Simulated Annealing (SA) and the Genetic Algorithm (GA). The names are not
accidenta1 since the ideas behind them are derivedo mnatural processes. S A is based
on cooling of metals to obin defined crysta1line structures based on minimum
potentia1 energy. G A is based on the combination and recombination ofe genes in
the biologica1 system leading to improved D N A sequences or species selection. In
actual application they ar more like a continuation of Chapter 8 (Discrete
Optimization) because these methods can be used for continuous and discrete
categories of the optimization problems. T hey share an important characteristic with
other optimization techniques in ey are primily search techniques-they
identify e optimum by searching the design space for the solution. Unlike e
gradient-based methods these techniques are largely heuristic genera11y involve large
ounts of computonand involve some statistical rea50ning. Globa1 optimization
is being pursued seriously today at cting a significant amount of new research to e
area of applied optimization after a long hiatus. A significant drawback to many of
thesetechniquesis ey require empirical tuning based on the class of problems
being investigated. T here is also no sy way to detecmine techniquel
problem-sensitive parmeters to implement automatic optimization technique.
The global optimum cannot reall y be characterizedbeing different from the local
optimum. T he latter is referenced when only a part of the design space (as op sed to
e comple space) is viewed in the neighborhood of a particular design point . The
standard Kuhn-Tucker conditions established and used in earlier chapters (for
continuous problems) will identify e loca1 optimum only. This is a1so more
emphatic in numerica1 investigations of continuous problems where the algorithms are
expected to move toward the opm u m values close to where th:y are started (initial
350
9.1 PAOBLEM DEFINITION 351
While it makes sense to look for the globa1 optimumi5 no method to
LEn6ff v Jchar
ultt
c in uo1u 1sc. '01 mv. Fex;Em roo blems guaranaE lIso lu Iti is di iffi fic
c
toa Ett ha tSl a d representson lyaminis c u
r : Pi on p b lem s not n ces sly c onv T herefore i n t helit t er
both ofthese methods are expct ed solutions c10se optimum
solution rather the globa1 solution emselves-'I11isis notto suggestthat investigation
oftheglob oluuOnis a w a of e if the possibility of onenotblished
;222221;ti::;fr;;LP222JZ;21
0ptiIIlizaGon techniques are simple to implement while mquutng exhaustiVE
calculations.This can potenudly soak up all ofe idle timon the persona1 desktop
computing resource le ng it mediate aactiveness.
23e techniques and their ascussIon axe VEry basIC following E eme espoused
in apter 8. e der is encouraged to it journal publications monitor

m a aU 1IEzzE odf hniq us. ln is c hapte r st s o m e i s s sues s r e g i ng g 10

u on 'ep nt f011 1
9.1 P R O B L E M DEFINITION
'hfollowingdcvclopmentor-global optimizadon is bas d o n unconstrainedcontinuous
opzation. Solving cons inEdpmblemsdGemotappeartohaveevolvedtoacommon
tecd is not illus .ted in this book. Note at cons ined problems can be
converted to an unconsined problem using a penalty formulation (Ilap 7)le
problem can be defined as
M inimize f[X); [X)n
Subjectto:
(9.1)
(9.2)
s problem were (only one um)e opon could ly be
tablishedby any of the aifOpriattechniquesused in the bk since the loca1 minimum
is also e global minimum. To vestiga the global minn it is therefore expected
uldbem ylo ninirr 15several of nhavingd rentva1r
objective. least among them is theneglobal solution. Loca1 minimums in ese
of continuous problems areosesafy e K T conditionsd will be numerically
determi jfestarting s w ineneibooodof pminimum.This
definition may be applied to discreeopzationproblems that uscontinuousrelaxation
o bin e soMUon-For discmte problems using comple enumerationthe global
ootimimuon Should hdiscoverEd automatically.FIT1eEEfGEe-the discussion applies to
discretep lems thatu only incomplete enumeration.
9.1.1 Global Minimum
The global rninimum for the exists if e function is continuous on a
nonempty feasible set S at is closed and bounded [1]
352 GLOBAL opnMIZATION
le set S defines a region of e design space is feasible. A ny point in S is a
candidate design point X. In the n-dimensional Cartesian spa the point X
coponds to a vector of lcngth IIXII. The boundaries of the region are part of the st
S (c1ose .le length of the v tor X should be a finite number (bounded). 1
definition for the global minimum X. can be stated
f(X.) f(X); X e S (9.3)
The conditions for the existence of the global minimum as given above are attributed
to Weierstrass. Whileeglobal minimum is guar t d if the conditions are satisficd
ey do not negate e existence of the global solution if conditions are not satisfied.
Considering the broad range d type of optimization problems the conditions ar
fairly limited in uselness. The conditions for i a l mumare esame
as constraint (9.3) except the set S is limited to a small region around X..
Convexlty: If an optimization problem can be shown b e convex then e local
minimum will also be a global minimum. A convex optimization problem requir thc
objcctive function b e convex. If constraints are present then al1 of inequality
constraints must be convex while the uality constraints must be linear.
ConvexSet: lIs usually relates to design vector (or design points) of e
problem. Consider two design points X. and X
2
As before the set S defines a design
region and by implication bo X. and X
2
belong to S. T he line joining X. and X
2
contains a whole series of new design points. If al1 of these points also bclong to S
en S is a convex set. A cIosed set S r luis that the points X
1
d X
2
can also be on
the boundary. I f however. the line joining X. and X
2
contains some points that do not
belongtoS. en S is not a convex set . For a two-dimensional design problem. convex
d nonconvex sets arc illustrated in Figure 9. 1.
Convex Funct/ons: A convex function is defined only for a convex set. Once
again let S be a convex set from which two design points X. and X
2
arc selccted.
Letfbe the function whose convexity is to be cstablished. Consider any point X"
on a line joining X. d X
2
. The functionfis convex ifj{XJ is less va1ueof
e corresponding point on linejoiningftX.) andftXu. ln n dimensions is is
not easy to sebut a one-dimensional representation is prented in Figure 9.2. le
condition for convexity in e figure is
f(x ) S r (9.4)
A mathematical condition of convexity is that thHessian m ax (matrix of second
derivativcs) of the functionfmusl bc positive semidcfinitc at al 1 points in the set S.
Note thal this corresponds to e K T conditions for unconstrained minimizution

d
9.1 PROBLEM DEFINITION 353

s NonconvexSe.
X2J
I X1

Xl
gure 9.1 C onvex and nonconvex se!.
3
4

k
u
o
h
y

i
J
problems. From a practical viewpoint. the eigenvalues of function must be greater
an or equal to zero at al 1 points in S.
lt is important to realize a global optimum may sti1l exist even if the convexily
conditions are not m et. ln real optimization problems. it may be difficult 10 estublish
convexity. In practice. convexity checks are rarely performed. A n assumption is made
a global optimum exists for the problem. is a subjective elementforced
by experience. insight. and intuition. This is also often justified by e possibility
a better solution than e current one can be found.
x
X ] X
a
X2
Flgure 9.2 Convex functln.
354 GLOBAL opnMIZAnON
9.1.2 Nature of the Solutlon
It is assumed e global optimum solution w iIl be obtained numeria11y. A good
understanding of the nature of the solution is possible with referenca continuo
design example. Exnsion to di re problems is possible by considering that any
continuous solution co'esponds to a specific combination of the set of disc te values
for the design variables. Figure 9.3 is an i1Iusn of a figure with scveral minima
and maxima with a song global minimum at x = 4 X2 = 4. le figeavailable
rough flg9_3.m1 is the plot of e function(aIso Example 9.1)
sin (0.1 +(X.-4)l+(X2- 4) (9.5)
f(x.Xz) = - 2 0
.. - - ~ 0.1 +1 - 4 ) z + ( x
2
- 4)Z
Figure 9.4 iIlustrates e contour plot T he contour plot can be betrappciated in
colorusing colorbarfor interpretation. Running the m-fiIe will create both figures.
From relation (9.5) it is cIear the ripples w iIl subsidize as XI and x2 move much farthcr
away from where the global minimum occurs beaus of th influcnce of the
denominator. The numerator is constraincd to Iic between -1 and + 1 multiplied by thc
Several mlnlmnd maxlma wlth global solullon
5

5
0

q
H

-15
-20
10
.t::
x
z
-10 -10
Figure 9.3 Surface plot of Exampl9.1.
Ipiles 10 be downloaded m the web si!e ar indicnllld by boldface snns serif typc.

91 PROBLEM OEFINION 355

Several mlnima and maxlma with global solution- contour plot

1
4

1
5
2
2
F
5
J
l
~ ~ ~ ~ 0 2 4 6
x values
Flgure 9.4 Conlour plot of Example 9.1.
scaling factor of 20. In the limit the ripples should start to become a plane. T his can
be confirmcd by executing flg9_3.m with different ranges for variables.
From Figure 9.3 it is easy to spot globaI solution by observation. Surrounding
e globaI minimum are rings of 10 1 extrma. Numerically it is a chaIlengc 10
negotiate a1l of the bumps and dips. A simple way to charactc.e numerical
techniQues for a minimization problem is that ey attempt to find a ditch (or valley)
to faIl into. If this isas an unconstraincd minimum problem the K1conditions
willlead the solution to be ppcd y oneofe locaI minimum rings depending
on the location ofc starting point for thc iterations. lf the iterations ' C started cIose
to X ( = 4 x2 = 4en it is more I ikely c global optimum w il1 be discovered.
Staring atxl = -5 X2 = -5 using standard numerical t hniques will probably not lead
to globaI minimum.
This is a simple illustration but notGthem areManitBlocal minimums and
m ums d aEingleglobalmmm.In ign problems wimorc 2
V iablesit is not readily apparent how many locaI minimums exist and ifeglobaI
minimum is rcally that distinct even if it is postulat to cxist. W hat about the
complcte enumeration technique? T hcre is no way thc globaI minimum can escape thc
356 G L O B A L O PT1MIZATION
sifting due to is technique. There is a1so no technique at is simpler complete
enumeration. le problem is Ihe time to finish searching Ihe infinite points of Ihe
design space to discover Ihe solution.
T he solution to global optimum must Iherefore lie between standard numerical
techniques and complete enumeration. The former will not find Ihe solution unless it
is fortunate. T he 1atter carmot find Ihe solution in time to be useful. Global
opzation chniques tend to favor the latter approach because ofe simplicity of
imp1ementation. To be efullhe solution must be discovered in a reasonable time.
9.1.3 Elements of a Numerlcal Technlque
le prob1ems faced by a numerical technique searching for Ihe global minimum have
b n touched on in e previous section. In most standard numerical techniques E
loca1 rninimum serves as a strong m a et or a beacon for Ihe solution. In most cases
design changes are not accepted if Ihere is no improvement in the solution. If standard
numcrical techniques have to be adapted to look for global rninimum Ihen Ihey must
encourage solutions atin eshort run may increase Ihe objective value so ey
can find anolher valley at whose bottom is e global minimum. T hey must achieve
Ihis wilhout sacrificing signific tpotency of Ihe local rninimum. because a local
minimum solution is better than no solution at a11. Figure 9.5 illustrates Ihese ideas
using a single variable.
In Figu 9.5e iterations sta at point A. In a standard nneri 1technique. a
Sh dir tion is found which leads to point B. At B. if nventional numerical
techniques were used a search direction leading to point C will be accepted while a
search direction leading to point D will be strongly discouraged. But if point D was
somehow reachableen it cou1d move on to E and en F and continue to Ihe g1oba1
f(x)
x
F1gure 9.6 Strategy for global optimum search.

3
2

9.2 N U M E A I C A L T E C H N I Q U E S A N D ADDITIONAL E X A M P L E S 357

minimum. The main idea in finding the global minimum is to encourage solutions at
will escape the al ction of Ihe local minimum. This means al point E it is quite
possible e search direction may 1ead to point G. 11 is en r luired at a 1arge
number of iterations be permitted so at Ihe va11ey ho1ding Ihe g1oba1 minimum w i1l
be revisited. From is brief i1lustration Ihe characteristics for a technique ed to find
a global minimum can be summarized as:
Short-term increases in the objective function must be encouraged.
Traditional gradient-based techniques will not be particu1ar1y useful bcause
ey are based on search dirctions at de Ihe objective function at Ihe
current itcration. T hey w i11 not pcrmit incr ofIhe objective function value.
If descent-based search directions are not part of Ihe technique any
direction decre es e objective shouId be accepted to promote attraction
by Ihe 10cal rninimum.
As location of Ihe global minimum is not known search must be permitted
over a 1argc region wilh no bias and for a 1arge number of iterations.
Since acceptable directions can decrease or increase obj ve function
value a hcuristic determination of Sh direction (or design change) must
beptofthe chnique.
These features resemb1e Ihe zero-order technique for unconstrained
opzation d Ihercfore will rcquire a large number of iterations to be
effective. le large number of iterations w i1l force Ihe technique 10 be simp1e.
T he solution can only be concluded based on Ihe number of iterations as
necessary conditions (availab1e for continuous problems only) cannot
distinguish between loca1 and global optimum.
Wilhout a doubt Ihe best technique is complete or exbaustive enumeration. It is
Iherefore necessary to assume that complete enumeration is Il Ot all optioll in further
discussion.
T he prob1em description illusnanddisc sion have dealt wilh unconsained
rninimization. Consaints can be included by addressing Ihe above characteristics to
apply to feasible solutions only. It is 1eft to Ihe reader to contmplate e ability to
he stica1ly generate a large number of feasible design changes for consideration.
9.2 N U M E R I C A L TECHNIQUES A N D ADDITIONAL E X A M P L E S
T his section introduces two popular melhods for global optirnization. Bolh are derived
from naturaI physical processat help identify optimum states. le first one
Simulated A nnealing (SA). refers a physical process at establishes differcnt
crysta1line structures each of which can be associated wi different minimum
potential energy. This is attempted by reheating and cooling of materials/meta1s
while controlling rale of cooling. le second Ihe Genetic Algorilhm (GA) works
through mutalion of candidate solutions untillhe best solution can be eslablished a
358 G L O B A L OPTIMlZATION
process at can be related to natur1 solution or biological evolution. lese
techniques are iIIustrated with respect to unconstrained problems in this section.
9.2.1 Slmulated A n n e aJln g (SA)
Thc first application using S A for optimization was presentcd by Kirkpatrick et al.
[21.The examples covered problems from electronic systemsin puticular
component placement en by the need for minimum wiring connections. The
algorithm itself was based on the use of statistical mechanics demonstratcd by
Metropolis et al. [3] to establish thermal cquilibrium in a col1cction of atoms. The
term itself is more reprcsentative of the process of cooling matcrials particularly
metals after raising their temperature to achieve a definitc crystalline statc. A s
mentioned earlier the extension to optimization is mainly hcuristic. T hc simple
idea behind S A can bEillustrated through the Inodiaed basic algorithm of S A due
to Bochavesky ct al. (4).
Algorlthm: Baslc Slmulated Anneallng (A9.1)
Stcp 1. Choosc starting design "0. Calculate fo = J("o)
(Necd stopping critcria)
Step 2. Choos a random point on thc surface of a unit n-dIlIII:Rsional hypcrsphere
(it is a sphere in n-dimensions) to establish a search direction S
Step 3. Using a stepsize calculate
ft =f(X
o
+) ; A f =
Step 4. If A f S 0 then p = 1
Else p = e-fJt.f
Step 5. A random number r (0 S r < 1) is generated.
If r S p then the step is accepted and the design vector is upted
seno ch ge is made to edesign
G o to Step2
If implemented algorithm has to executed for a reasonably large number of
iterations for the solution toto a global minimum. The simplicity of the algorithm
is very appealing. A ca refi 110 bscrva6o n will
t
pa ra a
berlelat totheBoltzman n p ro ba.bi lity distrib uti on. In s tatistical
mechanics it is exp ssed as -klT; T is considered c annealing temperatu andke
Boltzmann constant. It dictates the conditional probabat a worse solution can
beac pted. For the algorithm to be effective it is commended that p be in the range
0.5 S p S 0.9. In formal S A terminology reprsents eann ing temperature. T he
foIlowing ideas in the algorithm arc apparent :

V
9.2 NUMERICAL TECHNIQUES A N O AOOtnONAL EXAMPLES 359
The algorim is heuristic.
The algorithm is suggestivc of a biased random walk.
Descent of the function is directly accommodaled.
Dirl tions at increase e value of the function are sometimpermitted. This
is important to escape local optimum. This is the potential to discover thglobal
optimum through e algorithm.
T his basic algorithm can defmitely be improved. During the initial iterations it is
necessary to encourage solutions to escapelocal optimum. After a reasonably large
number of itcrations the opportunity to cscapc local optimum (window of
opportunity) must be continuously rcduced so that thc solutioll docs not wandcr all
over the place and canccl the improvements to the solution at have already taken
p1ace. This can be imp1emen by scheduling changes e pameter
iterations increase. In S A tenninologyis is referred to e annealing schedule.
lesc modifications do not in any way decrease the problem!user dependence of the
algorithm. This problem dependencis usually unattractive for automatic or standard
implementation of the S A mcthod. Rcsearchers are engagcd in construction of
algorithms at do not depend on uscr-specified algorithm contro1 paramete [5J.
The S A algorithm for continuous variables implcmented in the book includes many
of basic ideas from the original algorithm. There are many additional changes
no he. First negative sign in the probility r m is folded into itsclf. A
si le procedure is included for obtaining the value It is based on the initial valuc
of "0 and fo. is ca1cula yicld a value of p = 0.7 for a Afual to 5 0 % of
inie algorithm favors a negative value of by d i. Sincc the process for
establishing an initial i1Ius ted above can detcrminc a po ve value the algorithm
corrects this occurrencc. It is latcr illustrated through numerical experiments that a
smal1 magnitude of thc pararncter allows the solutionwander all over the p1acc
while a 1arge value provides a s'Ong 10 cal ataction. Instead of the S A step being
incorporated when objective increases it is triggered when objective is 1ikely
to increasc bascd on slope. Ine a1gorithm below the co01 parameter is kcpt
constant at its initial value. The calculation of is borrowed from the traditional
one-dimensional stepsize calculation using the Golden Section method (Chapter 5) if
the objective is like1y to decree. If the slope of thc objective is positive or zero a
fixed is chosen. This value is a maximum of one of the previous
Algorlthm: Simulated Anneallng 9.
Step 1. Choose "0.J("o)p = 0.7 = -ln(p)/0.5 x J("o)
< ls = 1 If s > Os = l x s
360 GLOBAL OPTIMIZATION
Step 2. Choose a random point on the surface of a unit n-dimensional hypersphe
to establish a search direction S
Step 3. Calculate Vj{Xo);
If Vj{X
o
) < 0 then compute ID .
A X = S; = X o + A X ; X
oi+ 1
If Vj{X
o
) 0
4f=f(+ f(); a = m ( 1)
p =
A random number r; (0 S r < 1) is gencrated.
If r S pen e step is accepdand e design vector is updated
A X = a S ; = X o +d.X; ii+ 1
Else continue ii + 1
0 0 to Step2
The algorithm is availablc in SimulatedAnneallng.m. It is called through
Call_SA.m. T he algorithm is applied to Example 9.1 which was used for iIlustration
in Scction 9.1. T o executc the algorithm the following m-files (introduced in Chapter
6) are ncccssary: UpperBound_nVar GoldSection_nVar.m. Following the
practice of drawing contours for tW(variable problcrns two files are necessary for
describing .e example the second primily to quickly draw a contour plot using
m aix calculations. In this case the files are Example9_1.m and
Example9_1 plot.m. The latter file name must be set in SimulatedAnnealing.m.
T he calling statement and variables are cxplained in SlmulatedAnneallng.m. Thy
can be read by typing
>> h e l p Simulated A n ne a l i n g
in the command window (the current directory must contain the files)
T he ader is strongly encouraged to run the example 00 his computer. It would be
a singular coincidence if any of the sults presented here are duplicated. Running
numerical experiments is simply a mattcr of calIing the S A algorithm and taking a look
at the rults (especially e plot). T he design changes are illustrated using colors:
red-represen g dasing slopes-and blue-for positive slopes t lead to
design changes. T he n pment in the local minimum and the es pearc1early
iIlustrated on the contour plots. T he contour plots e filled color plots to identify areas
on the figure where the function has a local minimum or maximum. Darker colors
present minimums and lighter colors the maximum. T he examples were run usually
for 1000 iterations. It is conceivable they could have becn stopped earlier. O n a
200H z PII laptop it required less a minute for each run. A large number of
iterations is useful to indicate if the solution escapes the global minimum once having
obtained it . While it was not cvidcnt in is cxnpleit is likely problem dcpendenl
since in is cxample Ihe global minimum was significantly larger an thc local
J
7
P
9
1
2


9.2 NUMERICAL TECHNIQUES AND ADDITIONAL EXAMPLES 361

"

minimurns. Several features of the S A algorithm are iIlustrated using some numerical
expenments.
Example 9.1 This is the function used in Seclion 9.1. The global minimum is at X =
[44] with an 0tive valuej" = -19.6683. In e figur the contours suggest a
strong global minimum encircled by altemate local maximums and minimums. For
most of the runs e starling value was far from the global minimum.
sin (0.1 +(XI - 4):L + (-'1. - 4)
f(x. x
2
) = -20
l' - 0.1 +(x
1
- 4):L + (x
2
- 4)l
(9.5)
Minimize
-pd

3
F

-
3
2
q
p

H
K
Z

5
2
5

1
2
2
2
2
Five numerical experiments are presented below. It should be emph ized that ese
runs cannot be duplicated. Overall the algorithm worked very well. For exactly
same starting values and number of iterations (calling Call_SA.m without any
changes-about40 s) e global minimum was found 9 5 % of the time.
Run 1: Starting design = [-5 -5]; s =2.7567; Number of iterations = 1000
Useful number of itcrations (when dcsign changes occurred) = 383
Final design = [3.9992 3.9997]; Final Objective =19.6683
Figure 9.6 i1lustrates the trace of the design chang. It escapes the starting relative
minimum fairly easily to get .pped by a different local minimum. It is able to escape
to the global minimum where it is strongly attracd to it. be .ces in the local
minimum appear to prefer to stay in the local minimum suggesting the solutions are
not being encouraged 10 escape.
Run 2: Starting design = [44]; 0.0362; Number of iterations = 1000
Useful number of iterations (when design changs occurred) = 991
Final design = [100.72 127.75]; Final Objective = 0.128
Figure 9.7 iIIu ates thc trace ofe search for the global minimum. Most of the points
are not in the plot as can be seen from the solution. This solution was started at the
global minimum. T he search process app rs to wander all over the place and finally
off map. There is no pattem in thlimited points secn on the figure except e
suggestion at the search is encouraged tocape. T his can be ticd to the low
magnitude of s.
Run 3: Starting design = [44]; s = -1; Number of iterations = 10
Useful number of iterations (when design changes occurred) = 553
Final design = [3.8832 4.3709]; Final Objective =19.173
Figure 9.8 ilIustra the trace of the design search. In this run the value of s was sel
to -1. Once again the starting value of the design was the global minimum itself. From
the fige it is c1ear the search is not allowed to escape e global minimum that
easily. The choice of the control parameter was arbiyand several exnples have
to be studied 10 see if is value is problem independent . T he value of I is typically a
363 9.2 NUMERICAL TECHNIQUES AND ADOmONAL EXAMPLES

2
-

U
A
I
/
5
GLOBAL OPTIMIZA T10N
10
a
B
4
2

u
g
a
g
N
M

362
o 2
lu
-2
x
'
-4 -6 -8
10 B s 4 o 2
values
-2
X
1
4 -6
Simulated annealing owp.
Discussion on the Numerical Experiments: 'he following observations are
based on the experience of applying S A (A9.2) to Example 9.1. They correspond to
e reported experience with S A in egeneral literatu.
F1gure9.7
Slmulated anneaJing Example 9.1-converged solution.
good choice for optimization a1gorim spularly for a weU-sca1ed problem. W h at
is clear from the runs so far is that a negative va1ue of s of the order of 1 implements
a loca1 at ctor.
Flgure9.6
Annealing schedule (changing s) is important consideration in SA. Starting
with a low va1ue e control p m ers must be increascd-not too fast or it
will get pped a loca1 minimum.
Annealing schedul is developed through tria1 and error insight is not
obvious and based on probable behavior of the sh.
Length of time (or number of iterations) that e problem needs to be searchcd
is difficult to dtermine. While is can be based on monitoring given the
stochastic naturc of the search itcannot be asserted thateglobal minimum has
been estab1ished. T he S A algorithm itself can be expected to converge while this
property is too slow for practical use [6]. A larger number of iterations is more
likely to yield the globa1 solution.
S A search does not subscribe to iterative improvement. It i5 an example of a
stochastic evolutionary process. T he implementation uscd here Pseots a
memoryless se.
Run 4: Starting design =5-5]; s = -2.7567; Number of iterations = 1000
Useful number of iteratioos (when design changes occurred) = 558
Final design = [-3.3397 6.2019]; Fina1 Objective = -2.5674
T his is a repeat of Run 1: Fi9.9 traces the search for the globa1 minimum for this
run. It appears at 1000 iterations is not sufficieot to determioe the global minimum.
While the solution has escaped the starting loca1 minimum it has not yet escaped e
next local minimum. Since the va11eys (loca1 minimums) in this example are circular
note at the solution moves around in the va11ey but has not managed tocape.
Run 5: Starting design = [-5 -5]; s = -2.7567; Number of iterations = 10000
Useful number of iterations (when design changes occurred) = 3813
Final design = [4.0037 3.9996]; Final Objectiv= 19.6682
N o figure is presented. The solution converges to the global minimum.
"
3
t
365 9.2 NUMERICAL TECHNIQUES AND ADDITIONAL EXAMPLES
g
a
E
'
N
H

2
4
0
4

b
p

q
d
N
J
S

t
A
q
d
f
d
GLOBAL OPTlMIZATION 364
g
a
E
'
M
10 B e 4 -2 0 2
-val S
-4 -6 -8 10 8 4 -2 0 2
X -values
-4 -8 -8
Dlscrete Problems: In discrcte problems the S A algorithm will use a discrete
design vector detuned stochastically o m the avai1able set Each element of e
design vector is randomly chosen from e permissible set of values. The remaining
elements of thS A algorithm are directly applicable. Ife objecve decrccs then
the design change is accepted. lf it increascs then a conditional probability of
acccptance is determined allowing the solution to escape thc minimum. This is lcft to
ader. T he significant effort is the identification ofe random design vector.
Example 9.2 This is a function of two variables that MATLAB uses for iIlustrating the
graphics capability ofsoftware. Typing peaks at the Command prompt will draw
the surface plot. Typing pe peaks.m at the Command prompt will provide e
uations that were uscd to obtain the curve. This function used as e objectivc is
available in Example9_2.m and Example9_2plot.m . S A is used ~ o identify the
global minimum. Figure 9.10 illustrates the contour plot a10ng wichanges in e
design. T he function has two local minimums and thr local maximums. T he
following reprcsents a converged run.
Slmulaled annealing: nononvergence-lnsulficienlllerallons. Flgure9.9
Slmulaled anneallng: Increaslng - decreasdoppunity to escape. Flgure9.8
Rest ting from differcnt dsign must stiU be implemented to overcome strong
local influences.
Restarting from the same initial de is a1so important for establishing
convcrgence propees as is is a stochastic search process.
Constrslned Problems: T his chaptcr does not deal with constrainls or discrctc
problcms. lerc arc two ways of handling constraints dircctly
1. lf the new design point is not feasible it is not accepted and no dcsign change
tak place. T his is quite 1ctivc for random sh dircctions.
2. le function at drivcs the a1gorithm during each iterntion is the one with the
largcst violation. If the current point is feasible then the objective function will
drive search. T his make sense e process is largely stochastic and is
. expected to wander around e design space.
Starting ign = [-2 2]; 2.5; Numbcr of iterations = 1000
Useful number of iterations (when desigo .changcs occurrcd) = 338
Consncd problcms c be expected to take a significantly Iarger numbcr of
S rches. T he is currently no pr'ure to identify convergencc prope ofthcSA
with sp tto constrained optimization.
366 GLOBAL OPTlMIZATlON
-4 -3 -2 -1 0 2 3 4 5
X
1
values
Flgure 9.10 Sfmulated annealing: Example 9.2.
Final design = [0.02279 -1.6259]; Final Objective = -6.5511
This is the global minimum of the problem.
T here was a significant change made to e S A algorithm based on initial stswi
eex ution. At the starting design the design was largely positive (8.95). Changing
it to a negative number did not permit thsolution to escape the locol rninimum or to
swing oround the nearby local maximum. Numerical experimenls suggest e
constant f 3be maintained between
2.5 sf 3S-1 (9.6)
Otherwise no change was made to e algorithm.
9.2.2 Genetic Algorithm (GA)
le Genetic Algorithm is an important part of 0 new area of applied research termed
Evolutionary Computation. Thcse are actually search procses and naturally useful
9.2 NUMERICAL TECHNIQUES AND AOOITIONAL EXAMPLES 367
for discovering optimum solutions. Holland [7] was the first to usc the technique but
iuse as an optimization tool began in eamest in e latc 19808 developed momentum
in the mid-1990s and continuto attract serious interest toy. The term evolutionary
is suggestive of e natural process associated wibiological evolution-primarily
the Oarwinian rulofthe selection of the fittest. T his takes place because e constant
mutation and recombination of the chromosomes in e population yield a better gene
StructUI. A11 of these terms appe in the discussions related to evolutionary
computation. This basic pro ss is also folded into e various numerical techniques.
The discussions can become clear if term chromosomes can be associated with
design variables. Over the y Evolutionary Computation has included Genetic
Programming-which develops prograrns rnirnick e evo1utionary proc;
Genetic Algorithms optirnization of combinatoriaVdiscrete prob1ems; Evolutio ry
Programming-optimizing continuous functions without recombinon;
Evolutionary Strategies-optirnizing continuous functions with recombination [8].
T his c1assification is quite loose. For example G A s have been used and continue to
be used effectively to solve continuous optimization prob1ems. Evohonary
algorithms have been successfully applied to optimization prob1cms in several areas:
enginecring design pararneter fitting knapsack problems transportation prob1ems
image processing traveling salesman scheduling. and so on. As can be infe from
e list. a significant number are o m thc arca of discrctc programming.
This section and is book include only G A s though al1 of the evolutionary
algorithms share many common features. G A s generate and use stochastic
information in their implementation and therefore can be considered global
optimization techniques. They are exceptional1y useful for handling ill behaved
discontinuous. and nondifferentiable problems. Unlike the S A convergence to the
globa1 optimum in G A s can only be weak1y established. G A s generate and use
population of solutions-design vectors ( X ) - un1ike S A where a search
direction (8) was generated. T his distinction is essential to understand the
difficulty of handling continuous problems in OAs. For example. using random
numbers to gnerate a value for Xlo the values of 3.143.141 or 3.1415 will al1 be
distinct and have no relation to each other.
Generlc Genetlc Algorlthm (GGA) (A9.3)
Step 1: Set up an initial population P(O)-an initia1 set of solution or chromosomes
Evaluate ini1 solution for fitness-di rentiate. collate and rate
solutions
Generation index I = 0
Step 2: Use genelic: operalors to generate the set of children (crossover mutation)
368 G L 0 8 A L OPTIMIZATION
A d d a new set of randomly generated population (immigrallts)
Reevaluate population-fitness
Pcrform competitive selection -which mcmbers w il1 be part of next
generation
Selectpopation P(t + l)-same number of members
If not converged tt + 1
GoToStep2
The various terms are explained below.
Chromosomes: While it is appropriate to regard each chromosome as a desi
vector there is issue of representation of the design vector for handling by genetic
oprators. It is possible to work with design vector directly (as in e included
example) or use some kind of mapping t l 1 (real encoding) or binary (binary
encoding). Earlier work in G A used binary encoding. In is book the design vector
is used direct1y for a11 transactions. Real cncoding is commended. A piece of the
chromosome is called an allele.
Fitness a n d Se/ection: The fitness function can be related to thc objective
nction.be population can be rankedlsorted according to e objective function
value. A selection from this population is necessy for (1) identifying parents and (2)
identiingifey w i11 be promodtoenext generation. There are various selection
schemes. A fraction of the best individua1s can be chosen for reproduction and
promotion the remaining population being made up of new immigrants. This is ca11d
tournament selection. A modification of is procedure is to assign a probability of
selection for ordered set of the population for boreproduction and promotion.
T his probability can be based on roulette wheel elitist selection linear or geometric
rankillg [10]. The implementation in this book uses ranking without assigning a
probability of sel tion. 1t uses the objective function value to rank solutions.
Inlt IPopulation: The parametr here is the number of initia1 design vectors
(chromosomes) for e starting generation N
p
' number is usually kept the same
in successive generations. Usually a norma1 random generation of the design vectors
is recommended. In some instances doping adding some good solutions to e
population is performed. It is difficult to establish h o w many members must be
considered to represent the population in every generation.
Evaluation: The various design v tors have to be eva1uated to s if ey can be
used for creag children at can become part of the next generation. Fitness
attributes can servwell for this puose. For the case of unconstrained optimum the
objective functionJ(X ) is a good measure. Since global minimum is unknown a
progressive measure may be used.
l ~ m ; ; i i ; ~
9.2 N U M E R I C A L T E C H N I Q U E S A N D ADDITIONAl E X A M P L E S 369
Genetic Operators: lese provi ways of defining new populations from
existing ones. lIsis also where the evolution takes place. T w o types of operation are
recognized. The first is crossover or recombination and the second mutation.
Crossover Operators: There are several types of crossover strategies. A simple
crossover is ilItrated in Figure 9.1 1. Here a piece of e chromosome-an
a11ele-is exchanged between two p ents P" P
2
to produce children C " C
2
The
lengofthe piece is usually determined randomly. Mathematically if the parents e
X = [x" X 2 . . X n] and Y (defined similarly) and r is the randomly selected truncation
index. en the children U (= C 1) and V (= C
2
) by simple crossover are defined as [11]
x

.
r
t

1
1
r
1
4
l
l
L
-

-
u
v

ifi < r
oerwis
if i < r
otherwise
(9.7)
Arithmetic crossover is obtained by a Iinear combination of two parent chromosomes
yield two children [9]. If Il.
1
and ^'z are randomly generated numbers then the
children can be defined
y
y

h
h

+
+

x
x

h
L
.

=
q
q

If
(9.8)
1; i ' 0it is a convex Iination
P t
Parents
P 2 C t C 2
Childrcn
gure9
370 GLOBAL OPTIMIZATION
If there are no restrictions on 's it is an qne combination. For real's it is a linear
combination. A n important crossover operator is a direction-based crossover
operator ifJtY) is less thanJtX) and r is a unit random number:
c = r(Y - X ) + Y
Mutation: Mutation refers to e replacement of a single element of a design vector
by a randomly genemted value. The elment is also randomly chosen. For design
vector X for a randomly selected element k
c=JX
j
i*k
IX
k
i=k
There are othr types of mutation including nonuniform mutation and directional
mutation. The reader is referred to Reference 9 for a relevant description.
I mm.'fants: This is eaddition of a set 0 1 ' l'andomly generated population in each
generation before the selection is made for enext generation. T he search process in
G A is broadly classified ploration and exploitation. Exploration implies
searching across the entire design space. Exploitation is focusing more in the area
promising a solution. Typically crossover and mutation are ways to narrow the search
to an area of design spa based on e characteristics of the current population-
suggesting a local minimum. To keep e search open for the global mInImum other
areas must be presented for exploration.lis Is achIcved by bringing in an unbIased
population and having them compete for promotion to enext generation along wi
thecnt set . In many variations these are allowed to breed wIth e current good
solutions before evaluation.
The ideas presentcd here are basic to all of evolutionary computation d specific
to G A. They are also simple but complete. For each category listed above there are
more sophisticated models at are being researched and developed and e reader is
encouraged to seek information from the current literature. T hc specific G A algorithrn
implemented in book incorporates all of e elements above in a simple
implemenon.
Genetlc Algorlthm (A9.4) The algorithm actua11y implemented is given below. In
line with many other algorithms avaIlable In thIs text It Is provIded as a way to
understand and appreciate the working of the G A. It is a1so meant to provide a starting
point for the user to implement his own variation of the G A since these techniques are
sensitive to e specific problem being sdied. Another important feature mIssing
o mIsimplementation is an encoding scheme. T he design vector is handled directly
to provIde a direct fcedback.
. . .

R
H
U
-
-

l
J
u
-
-
U
F
I
F
t
i
-
-
j
i
n
-
-
J
r
9.2 NUMERICAL TECHNIQUES A N D ADDITIONAL EXAMPLES 371
SP 1. Set up an initial population. Ten sets of design vectors are randomly
created. From this set an Initia1 generation of two vectors is selected. This is
pool available for propagation. This is also the mInImum to perform
crossovers. The fitness function Is eobjectIve functIon.
Generation Index t = 1
Maximum genemtion Index = tmax
Step 2. Genetic Operations
T w o Simple crossovers
T w o Arithmetic crossovers (convex)
One Directional crossover
Mutation of all seven design vectors so far
Fourimrg
Selection of the best two design vectors (from 18 vectors) for promotion
to e next generation
If 1 = 1m Stop
Increment generation index: It + 1
G o to Step2
Example 9.3 The example forissection is a Bezier curve fitting problem. Given a set
of datafind the fifth-order Bezier curveatweda. leobjective function is
letsquar errorbetwneoriginal data and the data generated by the Beziercurve.
sproblem has been adclressed earlier rough a continuous deterrninistic technique in
an earlier chapter.ereare ehtdesi variables assiated with the problem. lereare
several reasons for the choice of the problem. Fithe global solution if it exists must
be close zerod condis provIdes a way describe eight design variables in
two-dimnsional space (plot). le best value for the design vector every generonis
plotted to see how the curve fit improves. Since encoding Is not being used in the
algorithm a reasonably long desIgn vector is necessaryobserve thcrossovers. le
original data are obtained from the folIowing function:
Jtx) = 1 + O.25x + 2e-
x
cos 3x (9.9)
Figure 9.12 illustrates e seP of the example including the eight design variables
which are the x and y compo of the intemal vertices tdetermine e fitted
curve completely.'be function is hard coded in C aI lGA.m. There are several reasons
why is Is not e best example for illustration of G A. FIrst it Is example in
continuous variables. Second local minimums are not very distinguishable and there
are innumerable m y of them. The global minimum is not substantially different
from the local minimum value to w rantexplomtion of the globa1 solution. Third the
problem Is not ill behaved. T he most important reason depends not on the algorithm
buton ecomputatIon of the objective function-ca1culatIng the least squared error.
372
2.5
2
g
~ 1.5
h
0.5

GLOBAL OPTIMIZA TlON

02

x
z
0.4
gure 9.12
0.6
Bezier curves
Orfgln Dala
- - Rlleddala
o Vertlces
0
(Xs'X
o
)
... . / 0"
} / . }
0.8 1
x-data
1 2 1.4 1.6 1.8
Genetlc Algorithm. description of Example 9.3.
2
It uses a Newton-Raphson technique at is prone to fai1urc if thc curve folds over
severelyor c vertices are very c10se togeer. One way to overcome this is to cause
the x-Iocation of the vrtices to monotonically increase (which is implemcntcd in e
population gcncration function). This upsets many of the crossover operations SIOCC
arenowcoally random in dofply random.
le following files are nccessary for ssp ific exnple relating to the handling
of Bezicr curves: c.mFactorlal.m Comblnatlon.m Bez_SCLError.m and
are available in the code dircctory for the chapter.
T he G A in AIgorithrn (A9.4) is implemented using the following m-flIes.
C aI lG A . m
GenetlcAlgorlthm.m
populator.m
SlmpleCrossover.m
ArimetlcCros:Iver.m
DictionalCrossover.m
Mutatlon.m
DrawCurve.m
Setting up parameters and data for the a1gorithm
Dicts the algorithm
Gencrates random design vcctor
Pcrforms simple crossover between two parents
Performs nvex crossovcr betw n two parents
Performs one dircctional crossover
Performs mutation of edesign vectors
Draws thc curve for each design vcctor

9.2 NUMEICALTECHNIQUES AND ADOITIONAL EXAMPLES 373

Figures 9.13 to 9.17 capturc thc app1ication o f G A to Example 9.3 through 200
generations. Thc reader should be able to run the cxample from the code dircctory
of Chapter 9. Because of the heuristic nature of the gcncration of solutions the
solution trail will be different from at shown below. Running the examplc
several times by invoking CallGA from the Command window should illustrate
the G A always manages to closcly approximate thc curve after 200
generations. The maximum gcneration va1ue is not predictable and cannot bc
estab1ished with certainty. It relies on numerica1 experimentation. T his number
appears casily to be over 1000 in many investigations. The reader is urged to
expcrimcnt with the example to cxplore this issue. In running C aJlG A . m the bcst
values of the design vector and thc corresponding objcctive function valuc are
capturcd in the variable Xbest-an array of 201 rows and 9 columns (assuming
200 generations). The first 8 eolumns carry the best design for the generation and
the last column is the corresponding objective function value.
Fig e 9.13 is e best design from the starting generation. It has objective
function va1uc of 32.13. lis is ecor squared over 100 data points of the origina1
data and e curve generated by c design variables. Thc design variables arc e
verticiIIustratcd in the figurc. The basic curve of Figu 9.t3 is obtained using
2.5
2
1.5
a
. .

> 0
0.56

-0.5
0
:
0.5
Flgure9.13

1.5 2 2.5
x data
Best fltatgneratlon t" 1.
374 G L O B A L O P T I M I Z A n O N
9.2 N U M E R I C A L T E C H N I Q U E S A N D ADDITIONAL E X A M P L E S 375
g 0.5
=
0-"
".... .. 4"""' ......... . . . . . ' + .
-0.5
-1

ZO L
0.4 0.6 O. 1.2 1.4 1.6 1.8 2
x data
Flgure 9.14 Best f it : generation t= 200.
35
30
2s
a
u
P
3

10

50 200 250
x data
gure 9.15 Objeclive function for every generation.
> > DrawCurve(Xbest(1 1:8))
is pr'ure. 1t is possible to scparately accumulate only designs are feasible
while continuing to implement the standard procedure. Sccond use a strategy to ive
the search process through constraint satisfaction and decreasing e objcctivc
simultaneously or separately. Thc algorithms indicatcd in this chapter can easily
incorporatc n dedch ges.
The a1gorithmic procedure as implemented do not real1y care ife problem
is continuous or discrete. 1t just works on a current set of values for the design
variables. Discrete va1ues w il1 primarily require minor changes in thc way the
population is generated. A1so thc implcmcntation of the crossover and mutation
operators w il1 need to be changed. These changes in no w a y detract from the m or
idea bchind the GA.
D wCurve.m function. T o draw the initia1 curve (e figure is edited subsequently
with plotedit) in the C o m m a n d window:
Figure 9.14 is the best design at e fina1 generation (generation number = 2).
le objcctive function is 0.0729. 1t is clear from the figure G A is attempting
to find the solution. T his can be established by Iking at Figure 9.15 which is e
iterationlgeneration history of the G A for this example. T he objcctive function is
genera11y dccreasing through e generations. Figur9.16 is a sca1ed representation to
display the behavior of the G A at latriterations. From e data it is al80 apparent thnt
is cxamplc needs to run for morc 200 generations. The fina1 figure Figure 9.17
represents movcment and the clustering of the vertices through the generations. It
is suggestivc of loca1 refining is associated with the GA.
Suggested Extensions: Most of the engin ring design problems involvc
consints. Currcntly the most popular way to handle constraints is to reformulate thc
problem using pcna1ty functions (Chapter 7).'here are scvcral othr altematives to
Closlng Comments: In summary for a heuristic techniquce G A is very
impssive. lC samc can be said for SA. Including the fact at practical
engineering problems must require partly disete repsentatione tcchniques in
Chapte 8 d 9 are very relevant to e practi of design and development . With
the enormous computationa1 resoue in e desktops and laptops an effectivc
programming language Iike M A T L A B and a wide varicty of algorithms at arc
available it is possiblc now to design an individual approach to many optimization
376 GLOBAL OPTIMIZATION
0.5
0.45
0.4 scaled for lowest values
0.35

u
t
-
t

n
u

2
Z

-0.5
-1
-1.5
-20 .4
0.6 0.8
0.15
0.1
0.05

REFERENCES 377
Locatlon of deslgn varlables over 200 generations
60 80 100 120 140 160 180 200
x data
2.2 2.4
20 40 1.2 1.4 1.6 1.8
x value of vertex
2
Flgure 9.16 Obtive functlon scaled 10 show contlnual decrease lor laler generatlon.
gure 9.17 Trace 01 Ihe deslgn varlables Ihrough generatfon.
REFERENCES
problems. T his is important to practicing engineers who can bring inluition and insight
for the particular problem ey have to solve in a tight time frame. In the engineering
curriculuraditional design optimizalion courses lend to deal only wi
detestic continuous algorithms. T hat in itself was m o the time available
for instruction. Disc te optimization or global optimization are neeessarily skills
l m e d by personal endeavor.lC overcmphasis on deterministic programming needs
to be revisited in the light of current developin e arca of optimization. Tay
computer programming in engin ring depar ents and its reinforccment is usually
adjuncl 10 other developments in e curriculum. N e w programming platforms Iike
MATLAB offer a way to resolvc is disadvanlage. N e w algorithms also reflecl a
simplicity that can be sily slated and applied.
is book has been about programming in MATLAB leaming about traditional and
current techniques in design optimization and transloting these optimization
iques into appli .tions. lt is also about exploration and developing an insighto
the process of design optimization. T he code is provided is only meant to be a
starting point .
1. Arora J. S.lntroduction 10 Optimal Design McGraw-Hi I I New York 1989.
2.K'kpackS. Ge)att G. C.lInd Vecchi M. P. Optimization by Simulated Annea1ing
Science Vol. 220 No. 4598 May 1983.
3. M e polisN. Rosenbluth A. Rosenbluth M. Telle A. and Teller E. Equlltions of
Sate Ca1culations by Fast Computing Machines Joumal of Chemical Physics Vol . 21
pp. 1087-1092 1953.
4. Bochavesky 1. 0. Johnson M. E" and Myron L. S. Generalized Sulad A nnea1ing
for Function Optimization Teclmometrics Vol . 28 No. 3 1986.
5. Jones A. E. W. and Forbes G. W. An Adaptive Simulated Annea1ing Algorithm for
Global Optimization over Continuous Variables Jour : 1ofGlobal Oplimil.Cltion Vol. 6
pp. 1-371995.
6. Cohn H. and Fielding M. Simulated Annealing: Searching for an Optima1 Tempe ure
Schedule SIAM Joumal on Optimization VoL 9 No. 3 pp. 779-782 1999.
7. Holland J. H. Outline of a Logical Theory of Adaptive Systems Joumal of A C M Vol.
3pp.297-3141962.
378 GLOBAL OPTIMIZATION
8. Dumitrescu D. Lazzerini B. Jain L. C. and Dumitrcscu A. Evolutionary
Computation Intemationa1 Series on Computationa1 Intelligence L. C. Jain (edC R C
Press Boca Raton F L 2000.
9. Gen M. and Cheng R. Genetic Algorithms and Engineering Optimization Wiley Series
in Engineering Design and Automation H. R.Parsaci (scrics cd.) Wilcy-Intcrscicnce
New York 2000.
10. Goldbc D. Genetic Algorithms UI Search Optimizarion and Machine Lea mUlg
Addison-Wesley R ding M A 1989.
11. Houck C. R. Joines J. A. and K ay M. G. A Genetic Algorithm for Function
Optimization per accompanying the G A O T (Genetic Algorithm Optimization
TIbox) N o Carolina State Univeity.
P R O B L E M S
N o fonnal problems are suggested. le reader is encouraged to develop his o w n
variants of two algorithms including fonnal ones available in e sugges
literature. H e is also encouraged to consider extensions to constrained and discrete
problems. H e is recommended to use his versions to explo various examples found
in the literature. In comparison with e detenninistic techniques (until Chapter 8)
thes heuristic methods are truly fascinating and il1ustrate considerable robustness
and what is cven more challenging 10 our pcrception they can solve any problem with
no reservations whatsoever.hey are incredibly basic in construction simple to
program and easy to deploy. They only require a lot of time to search the design spa.
"^
F J
10
OPTIMIZATION TOOLBOX
FROM MATLAB
l e M A11.AB f ily of producls includes the Optimization Toolbox which is a librarv
o f mfil. n be used to ve p lems in a of optimizon. 1t con-
many of the methods that have been explored in is book and several at are not
available in the Toolbox. For osconsidering the practice of design optimization
seriously the Optimization Toolbox is a worthy investment . Ma ny ofthe techniques
and discussions in the book were kept simple to il1ustrate conptsd em-files
were used to provide M A11.AB programming experience. Most of the m-files available
on e wwill need to be d loped further for scrious prac application.
A nother important reason for having access to e Optimization Toolbox is it
allows one 10 take advantage of M A 11.AB' S open source policyat is to have cess
10 the source files. You may be able to modify and extend those filesmeet your
special optimization quirments. While some of the programs from the Toolbox
have b n used or referenced elsewhere in is bookis chapter collects them in a
single place for sy reference. A s can be expected the material ofis chapter will
be dated.'he chapter is based on M A11.AB Version 5.3. A s of is writing Version
6.0f6.1was shipping.In generalthe MATtAB productsespecially the Optimization
Toolbox have reached a level of maturity such t newer versions s ho
invalidate the infonnation in issec o n. W hat should be exp ted however is t
the material in is chapter w i11 most likely be upgraded enhanced and extended with
each new rele e o f M A.
7 Q
380 OPTIMIZATION TOOLBOX FROM MATI.AIl
10.1 THE O PTlMIZATlO N T O O L B O X
Most of the basic information in is chapter is from user's guide for the
Optimization Too1box [1]. Ifyou are buying a specia1ly bundled C O that includes the
Optimization Toolbox then you may not have Reference 1 in book fonn. You may
have Reference 1 as a P O F file on thc C O or you may have access to it. For exnmple
on e Windows N T platform for dcfault dircctory insta11ation the P O F version of
Reference 1 is avai1ablc at
C:MA T L A B R 11 helppdCdocoptimoptim_tb.pdf
Refcrence 1 providcs examplcs of usc discusses sp ia1 cases a1gorims edd
many 0r significant topics.is chapter does not attempt to document them.
Limid duplicaon is ed to bring attention to the most useful f ltures.be
emphis is on i1lustrating the use ofe Toolbox for various examples indu d
the book. In fact a lnrge number of the techniques in thToolbox are not used in e
book and are not covered in is section. The experience provided in this chapter
should be sufficient to understand and use the new ones cectivcly.
To use c programs or m.filcs in the Toolbox threimportant actions must bc
initiated. One formulnte e problem in the format expccted by MATLAB with pcct
to c particular function thnt will bc calIed. Most1yis will involv using
appropte procedures to sct the right.hand side of the constrnints to zero or constant
va1ues. The Toolbox a1so makes a point of distinguishing betwecn linear and
nonlinear constraints. Scondset/change e paramete for e opnizntion
a1gorithms. T hi use the appropriate optimizer and ensure a va1id solution has
bcen obtained. AII ofse actions natura1ly depend on the actua1 mathcmatical model
of eproblcm.
0.1.1 Programs
T hc different types of optimization problems at can bc solved wi thc version of
the Toolbox installed on computer can be discovered in the Command window
Help window or in the browser using Help Oesk. In e MATLAB Command window
type
> > h e l p optim
The output is reproducd below. They are a11 of the various m.files in the Toolbox and
their optimization function is a1so idcntified.
Optimization Toolbox.
Version 2.0 (Rl1) 09Oct1998
What's new.
Readme - N ew features bug fixes and changes i n this
version.
10.1 THE OPTIMIZATION TOOlBOX 381
Non1inear minimization of functions.
f a i a b a d - - S c a l a r bounded nonlinear Eunction
minimization.
m i : 2 2 : :M u l i m e n constrained nonlinear
minsearch-Multidimensionalunconstrained nonlinear
minimizationb y Nelder-Mead direct search method .
f m i a u n c - M u l t i d i m e n s i o n a l unconstrained nonlinear
minimization.
f g a m i n f - M I l l tidimensional cons trained minimi zation
semi-infinite constraints.
Nonlinear minimization of multi-objecive functions.
f g o a 1 a t ? a i n - M u l t i d i m e iona1 goa1 attainment
optimization.
minimax Mu1tidimensiona1 minimax optimization.
Linear 1east squares (of matrix prob1ems) .
1 a q l i n - L i n e a ? l e a s t s q u a r e s with linear constraints.
::: t : : - L L n least ares with non ativity
Nonlinear least squares (of functions).
? s q c u w e f i t - N o n l i n e a r curvefitting v i a least squares
(wth b o lds) .
lsqnonlin - Nonlinear least squares w it h upper and lower
bounds.
Non1inear zero finding (equation solving).
~ z e r o - Scalar nonlinear zero finding.
fsolve - Non1inear system of equations s01ve function
Solve.
Minimization of matrix problems.
1inprog - Linear programming
quadprog - Quadratic programming.
Controlling defaults and options.
optimset Create o r a1ter optimization OPTONS
structure.
optimget - Get optimization parameters from OPTIONS
structure.
Demonstrations of large-sca1e methods.
c i r c u s t e a t - Q u a d r a t i c programming t o f i n d Shape o f a
circus ent.
m o l e c u l e - M O l e c u l e conformat ion SOILltion using .
unconstrained nonlinear minimization.
382 OPTlMIZATION TOOLBOX FROM MAn
optdeblur m a g e deblurring using bounded
linear
least-squares.
Demonstrations o f medium-scale methods.
optdemo - Demonstration menu.
tutdemo - Tutorial walk-through.
bandemo - Minimization o f banana function.
goaldemo - Goal attainment.
dfildemo - Finite-precision filter d e s i g n (requires
Signal Processing Toolbox) .
datdemo - Fitting data t o a curve.
The tutdemo is recommendcd for a demonstration ofuresof Tool
O f the above items optimset linprog quadprog fminunc a n d
fmincon are further explored in is chapter.
A clearer picture of the capability ofe functions can be obtained from the tabular
graphi rcpres a 1of thc P lem scts in the first chapler of Reference 1. For
examplee four tcchniques referenced in is chapter are iIlustrated in Table 10.1.
10.1.2 Uslng Programs
Eacb of thc functions in the Toolbox implements an optimizatios tcchllique can
be ed in sev al ways.'hese are documented in Reference 1 and are also available
online through one of the mechanisms of obining help in MATLABFor example
typMghelpmneonin thCommandwindowidentifies thefollowingdiffemtways
ofusing functionlprogram (the following is edited to show only types of usage).
FMXNCON Finds t h e constrained m i n i m u m o f a function o f
several variables.
F MN C O N solves problems o f t h e form:
m i n F(X) subject to:
A * X <= B A e q * X B e q (linear constraints)
Table 10.1 Graphical D cription or Some Toolbox Functions
Typ
Unconsaincd minimizauon
Linear programming
Quadrauc programming
Consned minimizauon
Notation
min.f(x)
min.f
T
x such that
[A][x] [b]; [A] [ x]=[b];
m T l +
[ A J[x] [ b]; [ A .J[;x] = [ . b.ql x S u
m nin.f(x: );
[AJ[.[X] ] S [b); [A.qUx]=[b];
funcuon
fminunc
fminsearch
linprog
quadprog
fmincon
10.1 THE OPTIMIZATION TOOLBOX 383
C(X) <= 0 Ceq(X) 0 (nonlinear constraints)
L B <= X <= U B
X=FMINCON(FUNX O A B)
X=FMINCON(FUNX O A B A e qBeq)
X=FMINCON(FUNX O A B A e qB e qL B U B
X=FMINCON(FUNX O A B A e qB e qL BU BN O CON)
X=FMINCON(FUNX O A B A e qB e qL BU B NONLCONO P TONS)
X=FMINCON(FUNX O A B A e qB e qL BU B NONLCONOPTIONS
Pl P2 ... )
[XFVAL]=FMINCON(FUNX O .. .)
[XF V A LEXITFLAG]=FMINCON(FUN X O ...)
[XF V A LEXITFLAGOUTPUT]=FMINCON(FUNX O ...)
[XF V A LE XT F L A GO U T P U TLAMBDA]=FMINCON(FUNX O ...)
(XFVALE XTFLAGOUTPUTLAMBDAGRAD] =FMINCON (FUNX O ...)
[X FVA L E XT F L A GO U T P U T L A D AG R A D HESSIAN]=FMINCON
F U NX O ...) .
T here 12 ways of thenincon function. 1C dierens depend on e
information in the mathcmatical model at is bcing usoo and the postoptimization
information that should be displayed. In using tbe differcntnctioncallsy missing data
at do not exist for the particu1ar examples must be inditOO using a null vector (0).
bere are e k evord the above list of function calls at need special
altention in cons inOO nonlinear optimization.
EXITFLAG: For fmincon is indicas if thc optimization was successful .
If EXITFLAG is:
>0 t he n FMINCON converged t o a solution X.
o then the maximum n u m b e r o f function evaluations w a s
reached.
< 0 then FMINCON d i d n o t converge t o a solution.
OUTPUT: is specia1 MATLAB structure element. Structure is a programming
cons ct usOO in many higher-Ievellanguages. It is simi1ar to array in MATLAB. A n
array will contain only simi elements ukc all numbcrs or all strings. A s cture
on the other hand can contain disparate elements such as severa1 numbers and strings.
The elements of e structure can be accessed by using the name of the 5 C
element followed by a period and C PI efinOO keyword for the particular elcment .
For cxample two of the elements of the OUTPUT are:
OUTPUT. i terations: the number of function cvaluations
OUTPUT.algorithm: thenameofe algorithm usOO to solve the problcm
L A M B D A I T his is al50 another structure in MATLAB and contains Lagrange
m tipuers. 1e valuesofes multipliers are useful for veriing the satisfaction of
e necessary conditions of optimization. T he values can be recover by e same
pr' C indicated for thc O U T P U T structurc.
-
384 OPTIMIZAnON TOOLBOX FROM MAn
LAMBDA .lower: value of the multipliers at the lower bound
LAMBDA. ineqnonlin : value of the multipliers tied to the inequality
consn(s'sin ok)
T hesc items werc associated with the functionnincon. Other functions may havc
differentladditional typeof information availabl.
10.1.3 Setting Optlmization Parameters
T here ar several parametcrs that are used to control the application of a particular
optimi tion technique. T hcse parameters are available through thc OPTIONS
struclure. A Il ofthem are provided with default values. Thcre will be occasions when
it is ncessary to change the default values. One example may be at your constraint
tolerances are required to be lower th the default Or EXITFLAG recommends a
larger number of function evaluations to solve E oblem. T hese pmeters are set
using a name/value pr. G encrally e name is usually a string whilc the value is
typically a numbr. The list of parameters is available in Reference 1 and can be
accessed most easily in the Command window by typing
> > help optimset
A partiallist of the parameters in the OPTIONS structure copied from the Command
window. follows.
OPTIMSET PARAMETERS
Diagnostics print diagnostic information about the
function to b e minimized o r solved [ o n I {off}l
DiffMaxChange - Maximum change in variables for finite
difference gradients [ positive scalar I .{le-l} 1
DiffM inChange - Minimum change in variables for finite
difference gradients [ positive scalar I {leS} 1
Display - Level of display [ off I iter I {final} ]
GradConstr - Gradients for the nonlinear constraints
defined b y user [ o n I {off} 1
GradObj - Gradient(s) for the objective function(s)
defined b y user [ o n I (off}l
Hessian - Hessian for the objective function defined b y
user [ o n I {off} ]
HessUpdate - Quasi-Newton updating scheme [ {bfgs} I dfp
I gillmurray I steepdesc 1
LineSearchT y pe Line search algorithm choice
cubicpoly I {quadcubic} 1
MaxFu n E vals - Maximum number of function evaluations
allowed [ positive integer 1
Maxlter Maximum number o f iterations allowed
positive integer ]
10.2 EXAMPLES 385
TolCon Termination tolerance o n the constraint
violation [ positive scalar J
TolFun. Termination tolerance o n the function value
positive scalar J
TolX - Termination tolerance o n X [ positive scalar 1
T he parameter names are identified in bold fon1. T hese parameters are the ones
w i11 con1a standard optirnization application. The default values for paramer
are given within the curly braces. For example:
> > o pions o pimset( Diagnostics' 'on' 'Display'
'iter' 'Maxlter' '500' 'HessUpdate' 'dfp' 'TolCon' 'l.
Oe-OS')
U the above statemcnt is used prior to actually calling an optimization techniquee
default values for those pneters w i11 be changed in the OPTIONS structure. T he
default vaJues for a particular technique can be found by typing the Command window
> > options opimset( fmincon' )
lere several othcr ways to use the optimset program itself. For details consult
Reference 1.
10.2 E X A M P L E S
1 examples i11usated in is section are continuous optimization problems
discussed in e book. ley are from Iinear programming. quadratic programming.
unconstrained minimization. and constrained minimizatioo. Only the appropriate
mathematical model is iotroduced prior to i11us .ting the eof the Toolbox function.
10.2.1 Linear Programmlng
T he L P example uSed here includes all three types of constraints. The toolbox
function is linprog.
Example 10.1
Maximizc z = 4xI + 6x
2
+ 7 X 3 + 8x
4
Subjectto: XI + X 2 + x) + X 4 = 950
X 4 ~ 400
2x
1
+ 3 X 2 + 4 X 3 + 7 X 4 S 4600
3xI +4x: 5 X 3 + 6x4 S 5000
Xlo X2. X3. X 4 ;?; 0
386 OPT1M1ZAT10N TOOLBOX FAOM MATI.AIl
le linprog program solves the following problem:
Minimize fTx; subject to [A](x] S [b]
The objective function for the examplc needs to b changed to a minimization
problem. T he greater or equal to constraint needs to be changed to a less or
cqual to onc. T he linprog program a1lows equality constraint d explicit bounds on
variables. le equality consaint can be set up with A andbThere are not many
pers at need to be changed except switching of the 'LargeScalc' problem to
off.' For only four design variablcs Icaving it on should be permissible. The form of
linprog that will be used is
[XF V A L EXITFLAG] LINPROG(f A b A e qb e qL B UB)
This form is not explicit1y documented in Reference 1 or in the help documentation.
Ex10_1.m
1
is e m-ftle that is uscd to set up and run the problem. It is not displayed
here but running the file in Command window produces thfollowing output
Optimization terminated successfully.
X
0.0000
400.0000
150.0000
400.0000
F V A L
-6.6500e+003
E X I T F L A G =<
1
The solution to problem is
x; = 0; X; =400; x; = 150; x; = 47 . " =6650
10.2.2 Quadratic Programmlng
Quadratic programming refers to a problcm with a quadratic objective function and
Iinear constraints. While such a problm can be solved using nonlinear optimization
tchniques of Chapter 7. special quatic programming a1gorithms based on the
techques of Iinear programming are normally ed as ey are simple and faster. If
the objectivc function is a quadratic function in the variables then it can be cxpressed
inquaform. For examplc
f(x
1
) = x
1
+ 2 x
2
+ + X 2 + ~
can be cxpressed
I Filcs 10 be downlonded from thc web sile nrc indicalcd by boldface ns serif type.
10.2 EXAMPLES 387
)= [1 2]+1 x2] [ i !][::J
M
f(.)= [c]T [x] .X]T [x]
Example 10.2 Solve the quaatic programming problem 3
Min nim n nize i ze ) = [ 1 -2 3]T T
Subject x 1+2x2x 3 2.5 L 1
2x
1
- X 2 + 3 X 3 = 1
X1. X2' x) ~ 0
-1 11
2 ll[x]
1 41
The program to be used from the Toolbox is quadprog. T he standard format for this
program 18
Minimize f(x) = c
T
X + ~ x
T

Suect to: A x S b
A x=b.
L B S x S U B
T he solution can be found in Ex10_2.m. T he default optimization p:ters are
used so at the options structure does not havc to be changed. T he acaI calling
statement used is
[x fval E XTFLAG] quadprog(Hc A b A e qb e qL B U B )
The solution is formatted and displayed in thc Command window. 1t is copied from
the Command window:
Optimization terminated successfully.
Final Values
o p t i m u m D e s i g n Variables
- - - - - -
0.5818 1.1182 0.3182
O pt i m u m function v a l u e
-1. 9218

388 OPTIMIZATION TOOLBOX FAOM MATWI

Lagrange Multipliers for inequality constraint
0.0218
nequality constraint
2.5000
Lagrange Multipliers for equality constraint
0.0164
Equality constraint
1.0000
10.2.3 Unconstralned Optimization
Example 10.3 is used 10 ilIustrale usc of fminunce program to be ed for
unconstrained minimization. 11 is a simple curving examplc (a 101 of
unconstrained minimization problems in engineering are curve fitting used for
developing correlation from empirical data). T he Toolbox has programs specifically
for curve fitting. It may be a useful idea for the reader to compare the solutions using
thosprograms is provides an opportunity to use programs from eToolbox not
illus ted in this chapter.
Extunple 10.3 A set of data over 100 points represeng a cosine function is ted.
A quaatic polynomial is used 10 fit the data. A n unconstrained minimization
problem based on the sum of squ of the error belween original data and the
polynomial over all e data points is solved for the coefficients of the polynomial.
is an exnple of the classicalleast sq rede.rproblem.
The example is solved in m-fiIe Ex1 0_3.m. Running thc example also displays
a comparison of the fit on a figure which is not included here. T w o solutions are
obtained in the example starting from the same initial guess. The first one is solved
using 'er-stplied gradie. T he second is the default finite difference dients.
T he results ar copied from the Command window:
Final Values Oser specified Gradients
Optimum Design Variables
-0.4309 -0.0365 1.0033
Optimum function value
2.4055e-004
Gradients o f the function
1.0e-007*
0.1324 -0.2198 -0.3275

10.2 EXAMPLES 389

F i a a l v a l u e - - D e f a u l t Finite Difference Gradients
Optimum Design Variables

- -
0.4309 -0.0365 1.0033
Optimum function value

.
2.4055e-004
Gradients of the function

1.0e-004 *
-0.0121 -0.0270 0.5715
It is generally belieof gradients wouJd Jead 10 a f iter and
better ution. For t inunc to gradients computed by changcs havc
to bc made to e options s clu well as the file w h the objective function is
c i(IJ10_3m). In ons structe 'OradObj' parameter should bc
sct to 'on'inISIS Emponmt Otherwise M A T U B d o m not cxpECt thE gradients to bE
provid by the user. T he option for this exnple is sel by
T Z ; ; ? ? sTolFun' ;?;J:??;;zp;;;;ff;play'
bis causes epro m to look for diene second elem ofthe retum
value.The functIon Statement usdfor obj103.m is
function [f grad] objlO_3(x)
B oe occtive function d e gradie (grad) are calculated in the same
function m-fiIe.
10.2.4 Constrained Optimizatlon
Example 10.4 T he example is the same as Example 8.4 and Psents lhe
constrained ex nple for this scction.ne mathematical model is
Minimize f(X) = pLAc
Subject to: g.(X): W L
J
- 12EI S 0.25 (dcfl tion)
g2(X): W Lx. - 8/1y S 0 (normal stress)
g3(X): W Q c - 21xJ'ty S 0 (shear slress)
Some gcomctric constraints to relutc the various design variubles are
390 OPTIM(ZATION TOOLBOX FROM MAn.AB
g4(X): x -3X2 0
gs(X): 2x
2
- x ( S 0
g6(X): X3 - 1.5x4 S 0
g7(X): 0.5x4 - x3 S 0
The side constraints on the design variables are
3 Sx( S 2 0 2 S x
2
S 15 0.125 S X
3
SO.75 0.25 S X
4
S 1.25
The foIlowing relations define the cross-sectional area (A
c
)' moment of inertia (and
first moment of area about the centroid (Qc):
Ac = 2x2x4 + x(x Z
Ix t ( - x J ( x-
2x
i

12 12
Qc=0.5.l+0.5x _ X J 2
The paramers for problem ar w= 1.42000L = 96 E = 29 E+06 P = 0.284
'y=36E+03:y = 21 E+03. There are three nonlinear inequality constraints and four
linear inequality constraints. le code for running the prograrn is available in
Ex10_4.m.
T he program to be used is frnincon. The following OPTION parneters are
changed.
options optimset( LargeScale' off' Display'
iter' TolFun' 1. Oe08) ;
le calling statement used for fmincon with the objective function available in
objective.m and the nonlinear constraints in nonlin_cons.m is
[x fval exitflag output lambda]
fmincon( objective' x O A B L B U B nonlin_cons'
options);
The Iinear inequality constraints are set up in the matrix A and vector B. There are
no Iinear equality constraints and they are indicated by the nuIl matriccs. The
lower and upper bounds are applied through the vectors L B and UB. The
customized part of e output displayed in the Command window is included
below. Accessing various Structure values will provide details about the
performance of algorim.
Final Values
Optimum Design Variables
-
- - - - -
6.8671 2.2890 0.1802 0.2500
Optimum function value
.
62.4905
Final Nonlinear Constraints
- - - - -
1.0e+009
-2.7304 -0.0025 -0.0001
REFERENCE 391
Lagrange Multipliers for Nonlinear constrains

. .
0 0 0
Lagrange Multipliers for Linear constraints

- - -
2.5879 0 0 0
The reader is encouraged to run the example with Diagnostics' on explore
opttons structure and e output sucture.
R E F E R E N C E
1.lmmJ-R.BmEI M. A. and Grnce A. Optimization Toolbox-for Use witlr
M A T L A BUser's Guide Version 2 MathWorks Inc 1999.
liiIi
Abstract mathematica1 model 5 9-10
Acceptable solutions 17
Active constraint set 293
Acrospace design 1
Algorithms21
A L M mcthod see Augmcnlcd Lagmnge
Multiplicr method
Ana1ytica1 conditions 175-176
Annealing 8chcdulc 359
Anificia1 varbles98
Augmented Lagmnge Multiplicr (ALM)
m thod276-281
algorim 277-278
application 278-281
unconstrned problem
unsformion 10 276-277
Augmented matrix 103-104
Bez.ier parameccurvcs fitting of
258-262
BFOS methodsee Broydon-Fletcher-
Ooldfarb-Shanno m ed
Biscction tcchnique 209-210
Bmnch and Bound
gorim 333
applica n333-335
tcchniquc 329 336
Broydon-F1etchcr-G oldfarb-Shanno
(BFOS) method
n1gorithm. 250
application. 250-25 1
Cartian spaces. 106
I N D E X
C (Ianguage) 24
Classofproblcms.21
Computationa1 rcsources. 2
gate Omdicnt mcthod. 244-246
255
algorithm 244-245
application 245-246
Constmined optimization problems.
265-315389-391
Augmentcd Lagmnge Multiplier
(ALM) mcthod for solving
276-281
beam design 310-312
control optima 1 313-315
dcfinition of problcm. 266
ect methods for solving 281-302
Extcrior Pcnnlty FunctionPF)
method for solving 271-276
flagpole problem 307-310
formulation ofproblem 266-267
Gencrnlized Reduced Gmdient (GRG)
m e d for solving 297-3
generic a1gorithm for 269-270
indircct methods for solving 270-281
ry conditions for 267-269
Suentia1 Gradient R tomtion
a1gorithm (SGRA) 3307
SucntiaJ Linear Progmmming
(SLP) gorim 284-289
Suential Quadmtic Programming
(SQP) a1gorim 289-296
Consint functions 6 7
Consnts15
393
394 INOEX
Consn (conrinued)
uality.7-8
inequality. 8
in LP prob1ems. 98
in NLPprob1ems. 157-159.172-175.
179-181
side.8
Continuous re1axalion. 321-322
Conlinuous vnriablcs. 20
Con 1.0pnal313-315
Convcrging 10 thc solution. 21
Convcx functions. 352-353
Corners.20
Davidon-Flclcher-PoweU (DFP) method.
246-249
Dcgrees of freedom (00.104
Dcscent funclion. 292
D ign.1.2
Dcsign functions. 5-6
DcsignoeClivcs.5
Dcsign pammelcrs. 5
Design space. 9
Design variab1cs. 4-5
Deslgn veclor. 5
D fP method. see Davidon-Fletcher-Powell
m ed
Digraphs. 336
Discrete optimizion prob1ems (DP).
318-348
Bmnch nnd Bound lechnique for solving.
329-336
continuous re1axationapproach for
solving.321-322
discrete dynamic programming ns
I hnique for solving. 336-341
Exhnustive Enumeration techniqufor
solving. 326-329
Indcsign.341-343
slnndd approach to solving. 322-324
Zero-Onc Integcr Programming (ZIP).
343 348
Dite programming. 20
Discrete varinblcs. 20
D O F (degrees of frecdom). 104
DP. see Discretc optimation prob1cms;
Dynamic progrnmming
Dual problcm. 138-148
Dynamic programming (DP). 336-341
algorim340
functional U on for. 338-339
E B V (Enlering Basic Variablc). 118
Editor (MATLAB). 33-37
EngineeIg.2
Entering Basic Variab1e (EBV) 118
EPF mcthodsee Extcrior Penalty Function
method
Equality nstraints.7-8
Euclidcnn spaccs. dimesionll1. 1O
Exhaustive EnumcraLion techniquc. 326-329
algorithm. 327
applicalion.327-329
Exterior Penalty Function (EPF) method
271-276
algorithm.272-274
appliclllIon.274-276
Fcasib1e domain. 23
Feasib1e solutions. 17
First-order coditions (FOC). 176-178
First-orderllincar variation. 169
Flagpo1e prob1em. 307-310
FOC. see First-order conditions
Formu1ation of prob1emsee Prob1em
fonn8!tion
FORTRAN.24.227
Function(s):
constraint 6. 7
objectivc6
ofone variuble. 162-164. 169-170
oftwov ab1cs.56-63.164-171
GA. see G enctic A1gorithm
Guuss-Jordan c1imination. 107
Generalized Reduecd Gmdicnt (GRG)
m ed297-302
a1gorithm. 299-300
upplicalion. 3302
stepsize. ca1cu1alion of. 299
G eneric Genetic A1gorim(GGA).
367-370
Gcnctic A1gorilhm (GA). 350. 366-377
gcneric fonn. 367-370
suggcsled extensions 10. 374. 375
G10bal optimization. 350-377
definition ofproblcm. 351-357
Genetic A1gorilhm (GA) for ein.
366-377
minimumglobal. 351-353
numcrical techniqucs for. 356-358
Simuluted Annealing (SA). 358-366
solution. global. 354-356
Golden section method. 214-217. 219-220.
223-225
Gmphica1 optimiza 16-18.45-92
definition of problem. 45-48
nnd format for gmphical display. 47-48
function of two variables. 56-63
nnd graphica1 user interface. 81-91
wiHnndlc Gniphics 79-81
at transfer examp1e. 73-79
solution. lphi 1.48-56
structural engineering design examp1e.
64-73
Graphical solutions:
linear progrnmming (LP) prob1ems.
07-15
nonlinear progrnmming (NLP) prob1ems.
171-175
one-dimensional problcms. 205-206
Gmphical User Interface (GUI). 24. 48. 49
79.81-91
G R G method see Gencralized Reduced
Gmdient method
GUI. see Gmphicll1 User Interfacc
Handle Gmphics. 79-81
HnndJcs.49
Heatansfer. 73-79
Inuality cons ints.8
Infinite solulions 17
lnitial dign solutions. 21
Jnleger progr lming.20
IlIteger variab1cs. 20
llcrative tecbniqucs. 21
Kuhn-Tucker conditions. 192-193.
196-200267-269
Lagrnnge mu1tipliers. 180-192
LAMBDA progmm (OpZ 8tionTI1box).
383-384
INOEX 395
Leaving Basic Variab1e (LBV). 118
Lineardndence01
Linear progmmming (LP) prob1cms. 23-24.
93-153
consnts in 98
delerminants . 02-104
dual prob1em associadwi. 38-148
cquality consbintsexnmple involvlng.
130-134
cx .plcs of. 11111 124-138.
385-386
four-variable prob1cms 134-138
graphical Solulions of. 07-115
infmite solution of. 114
MATLAB solution using 120-124
modcling issucs in98-107
negative d ign variablcs in. 98
N L P problems vs.. 20 93-94
nosolon for. 114-115
primproblem. 138
sensitivity nnalysis wi.148-151
Simplcx method forsolving 115-124
stllndard format for. 94-98
transponalion prob1em exnp1e.
124-130
unbounded solution of 114
unique solution of. 114
tvtors i06-107
Mninfmme computing syslems. 24
Muldruw.47
Mathem8!tical mode. 5
Mathem8!ticnl mode1s/modeling. 3. 10-16
MathWorks Inc. 2s
MATLAB.24-44
code snippet. crealing a. 37-40
edilor in. 33-37
l-lime use of. 27. 29-33
bigh-Jeve1 grphics functions of. 48-50
insla11ation of. 26-28
operurs in. 30-31
Optimization T 1box of. 123-124.
379-391
progrnm.cating a 40-44
Simp1ex Method using. 120-124
Symbolic Math T 1box of. 159-161
usefu1ness of 26
Matric:
augmenl.103-104
396 INDEX
MBIIIs (continue.
rank of 104-106
M uimum problems 6
M-fi127
Minimization problems 6
Modified Newton med252-253
Mu1tidisciplinary designs 6
Mu1tiobjcctive digns 6
n-dimensional Euclidenn spa: 5 106
Newton-Raphson technique 206-209 252
Non1incar programming ( N LP) prob1cms
21-23 154-2'
W1a1yti1 conditions in 175-176
consaints in 157-159 172-175
179-191
examp1es of. 194-2
firsl-ordcrnditions in 176-178
formu1ation ofproblem 155-157
graphica1 solutions of 171-175
Kuhn-Tuckcr conditions for 192-193
196-2
Lagrangc multiplier mcthod use of
180-192
LP prob1ems VS. 20 93-94
one variable functions of 162-164
169-170
m n d0erconditions in 178-179
symbolic computotion 159-161
Tay10r scries for use in solving 169-171
twoormo varillblcsfunctions of
164-171
Obj tivc functions 6
0 cct orientcd programg24
One-dimcnsional problems 203-225
biscction tccbnique for use wi209-210
constrned204-205
dcfinition of problem 204
golden scction mcthod for use with
214-217219-220223-225
g M 1 solution of 205-206
importance of 217-219
Newton-Rapbson tccque for use with
206-209
polynomia1 approximBtion with 211-214
two-point boundary value problem
220-223
Operators 30-31
timal solu17
Optimizationl-4
Optimization Toolbox 123-124 379-391
constraincd optimil.ation using 389-391
examples using 385-386
1inear progmmming examplc using
385-386
optimization parameters setting 384-385
quaic progr uning cxamp1e using
386-388
unconstrined optimization using
388-389
using progmms in 382-384
OUTPUT program (Optimization Too1box)
383
Paramed ign 5
Partial derivativ 1
Pascnl24
Pntlem Scarch metbod 234-238
Po1ynol1upproximation211-214 217
Powell's mctbod 238-240
a1gorim 238-239
application 239-240
and conjugacy 240
Primal problem 138
Problcm defi onandfirmu1ation4-10_
See also Matbematical modelsJ
modeling
coSned optimization problems
266-267
nstrnt functions in 7
design functions in 5-6
di parneters in 5
and design spa9
design vnriab1es in 4-5
ity nstri 7-8
global optimil.at in351-357
gruphicu1 optimization 45-48
inequality constraints in 8
nonlinear problems 21..23
I in rprogramming (NLP) prob1ems
155-159
objccuve function in 6
one-dimcnsional prob1em 204-205
sidensaints in 8
standard format for 9-10
unconstmincd optimization problems
227-230
Prob1em rclaxation 321-322
Problem solutions 16-24
Programming discrcte 20
grammingintcg 20
Programming languag24
Programming pamdigms 24
Pseudornndom numbers 230
Quadratic convergence 238
Quaatic po1ynomials 238
Quaatic progrng386-388
Rundom Walk method 230-233 254-255
Redundancy101
Rosenbrock problem 253-255
RotaIg disk thrce-dimensional 110w near
255-258
S A see Simulated A nnenling
Script m-fiIes 27
Sh a1gorim 218
Search dircction 22
Search meods21
Second-order conditions (SOC) 178-179
ScIId-urderlllnear variation 169
Sensitivity anysis148-151
Sequential dccision problems 336
Sequentinl Gradient Restoration a1goritbm
(SGRA) 302-307
nlgoritbm 305
applin305-37
gradient phnse 303-304
restoration phase 304-305
Sequential Linear Progrng (SLP)
284-289
gorim 285-286
application 286-289
Sequential Quaatlc Progmmming (SQP)
289-296
a1gorithm 294
application 295-296
and BFGS method 293-294
multipliers calculation of 293
stepsize ca1culation of 292-293
Sequential Unconstrnincd Minimization
Tcchniques (SUM'270-271
S G R A see Sequentlul Omdicnt Restoration
aJgoritbm
Side constraints 8
Simplcx method 115-124
application of 117 -120
INDEX 397
Entering Basic Variable (EBV) 118
features of 115-117
Leaving Basic Variub1c (LBV) 118
M ATI-ABsolution using 120-124
t wo-phase 125-130
Simulllted A nnealing (SA) 350 358-366
a1temativc a1gorim 359-365
basic algorithm of 358-359
nstraints hand1ing 364
in dlscrcte problcms 365-366
Slack variab1es 95
SLP see Sequential Lincar 'Ogramming
S O C see Sccondordcr conditions
Softw 24
Solutions 16-24
acceptoblc 17
infinite 17
initial design 21
optim 17
SQP see Sequential Quadrntic Programming
Stnndard formal 9-10
Stecpest Dcentmed241-244
S U M T J-ee Scquenti Unconstrained
Minimization Tcchniques
Symbo1ic Math T olbox (MATLAB)
159-161
Taylor scric 169-171
Tolance209
TPBVP sec T wo-point boundary yalue
problem
Transn problcm example 124-130
Twcpoint boundary va1ue problcm
'PBVP)220-223
Unconstrained oplimizlltion problems
227-262 388-389
Bczicr paramctriccurves fittlng
258-262
Broydon-F1etcher-Go1dfarl:Shanno
method for solving 249-251
Cougate Gradient mctbod for solving
244-246
Davidon-F1etchcr-Powcll melhod for
solving24249
definition of prob1em 227-230
generic a1goritbm for 229-230
396 INDEX
Malri (continued)
rankofl04-106
M u m problcms. 6
M-fil27
Minimization problcms 6
ModifiNewton method 252-253
Mullidisciplinnry designs 6
Multiobjeclive designs 6
n-dimensional Euclidean spaces 106
N e wlOn-Rapbsotechquc206-2252
Nonlinear programming (NLP)
21-23 154-200
analytical condItions in 175-176
constrllInts in 157-159 172-175
179-191
cxampl194-200
1-0er condilions in 176-178
mulation of problern 155-157
graphicalsolutions of 171-175
KuhnTuckerconditions for 192-193
192
Lagmnge mullIplier melhod use of
180-192
L P problems vs. 20 93-94
one varinble. funclIons of. 162-1 64.
169-170
second-order condIlions in 178-179
syrnbolic computaon159-161
T l1ylorseri for use in solving. 169-171
two or moro varIables funclions of
164-171
Objective functions 6
0 loricn programming24
One-dImensIonal problcms 203-225
biscction technique for use wIth 209-210
COIned204-205
definition of problcm 204
gold scction method for use with.
214-217219-220223-225
grnphical solution of 205 -206
importance of 217-219
N c wlOn-Raphson lechnique for use with
206-209
polynomial approximation Wilh 211-214
two-poInt boundnry valuc problem
220-223
Opemlo 30-31
Optimal solutions 17
ionl-4
Optimization T lbox 123-124379-391
conslmIncd optImiznlion using 389-391
examples using 385-386
linear prograrmning exnple usIng
385-386
optimizalion pm ersseuing 384 385
qua.tic programming cxample using
386-388
unined optimization usE
388-389
using programs In 382-384
0 rpUT progmm (Optimization T lbox).
383
Parameters design 5
Panial derivatives 164
Pascal24
PattcmS rch mcthod 234-238
Polynomial opproximation 211-214. 217
Powell's m eod238-240
a1gorim 238-239
app1n.239-240
and conjugacy 240
Primal problem 138
Problem definition and formu1ation 4-10.
See also Malhematical mode1s1
modeling
consd oplimizatioD problems
266-267
nsuaint functions in 7
design functions in 5-6
design paramete in5
and design space 9
design vnriablcs in 4-5
equality consuaints in 7-8
global ptation351-357
graphical oplimization 45-48
inequa1ity nsuaints in 8
nonlinear problems 2123
nonlinear programming (NLP) problems
155-159
0tive functioD in 6
one-dimensional problem 204-205
side constmints in 8
standord formnt for 9-10
uncons incd optimization problems
227-230
Problemrel ation321-322
Problem solutions 1624
Progmmming disc te20
Programming integcr 20
PrognunoUnglanguages24
PrognunoUng paradigms 24
Pseudorandom numbers 230
Quadratic convergen238
Quadratic polynomials 238
Quadmtic programming 386-388
Random Wa1k method 230-233 254-255
R undancy101
R05cnbrock problem 253-255
Rotating disk thrc-dimensional flow near
5-258
S A .rtt Simulated Annealing
Script m-filcs 27
Search a1gorithm 218
Shdi tion22
S rch methods 21
Second-ordcr conditions (SOC) 178-179
Second-orr/linear varintion 169
Sensitivilyannlysis 148-151
Suential diSIOD problems 336
Sequcntial Gradient Restoration a1gorithm
(SGRA) 302-307
a1gorim 305
application 305-307
gradient phe303-304
restoratiOD phase. 304-305
Sequential Linear Programming (SLP)
284-289
algorithm 285-286
app1ication 286-289
Squential Quadratic Programming (SQP)
289-296
algorim . 294
applicntion 295-296
a n d B F G S m eod293-294
mu1tipliersIionof293
stepsize calculation of 292-293
Sequential Unconsuaincd IImi7.otion
Techniques (SUMT>270-271
S G R A soe ScqucntiaJ Gradienl Res[oration
n1grithm
Sidc ns nts8
Simplex method 115-124
plication of 117-120
INDEX 397
Entering Basic Variable (EBV) 118
features of 115-117
Leaving 8asic Variable (LBV) 118
M ATlAB solulion using 120-124
two-phasc 125-130
Simuled Annealing (SA) 350 358-366
a1temative algorithm 359-365
basic a1gorithm of 358-359
n5Uin hldJing364
in discrete problems 365-366
Slack variables 95
S L P see Sequenlial Linear Programming
S O C sl!e Sccond-order conditions
Soware24
Solutions 16-24
aeptab1eI7
infinite 17
inilial design 21
optimal17
S Q P see Sequential Quudrntic Prograrnming
Standard fonnnt 9-10
StstOnt method 241-244
S U M TseeSuential Unconstmined
MlnImization Techniques
S y mlicMathT lbox ( M AT l AB)
159-161
Taylor scries 169-171
To1erunce 209
T P B V PsT wopoint boundy va1ue
problem
Transponion prob1em cxp1e124-130
T w cpoint boundnry value problem
'PBVP)220-223
U nn5Uined optimization problems
227-262388-389
8ezicr pnrametric cu fitting
258-262
8roydon-Fletcher-GoldfartShanno
method for 50lving 249-251
Conjugate Gradient melhod for 801ving
244-246
Oavidon-F1clcher-PowclI melhod for
solving24249
deflDItion of problem 227-230
generic a1gorithm for 229-230
E
398 INDEX
Uncorullrained oplimiz.ion problems
(cone
g dient-based lethods for solving
241-251
modified N eW l On method for solving
252-253
necesry IlDd sufficient conditions for
228-229
nongradient melhods for solving
230240
Pattem Sh method for solving
234-238
PowcU's melhod for solving 238-240
random w a1k a1gorilhm for 230-233
Rosenbrock problem 253-255
rotating disk thrce-dimensional sow
r255-S
second-order melhods for solving
2513
Sleepcst Dcscent mclhod for solving
241-244
Unique optirna1 solutions. 17
Unitv ors106-107
Varillble Metric Methods (VM M ) 246
Varinbles:
nrtificiH l 98
continuous 20
desi4 -5
discrele 20
integ 20
slack95
Vecr(s):
design 5
unit lO107
V M M (Variable Metric Melhods) 246
Zero-OnelnP r ogramrning (ZIP)
343-348
Zero-order methods 230
ZIP see Zero-One lnleger Programming