See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/230872616

Nonlinear Programming

Book · January 2003

CITATIONS READS

2,130 26,242

1 author:

Dimitri P. Bertsekas
Arizona State University
389 PUBLICATIONS 77,108 CITATIONS

SEE PROFILE

All content following this page was uploaded by Dimitri P. Bertsekas on 21 December 2016.

The user has requested enhancement of the downloaded file.

Nonlinear Programming
SECOND EDITION

Dimitri P. Bertsekas
Massachusetts Institute of Technology

W W W site for book Information and Orders

http://world.std.com/~athenasc/index.html

Athena Scientific, Belmont, Massachusetts

Contents

1. Unconstrained Optimization p. 1
1.1. Optimality Conditions p. 4
1.1.1. Variational Ideas p. 4
1.1.2. Main Optimality Conditions p. 13
1.2. Gradient Methods - Convergence p. 22
1.2.1. Descent Directions and Stepsize Rules p. 22
1.2.2. Convergence Results p. 43
1.3. Gradient Methods - Rate of Convergence p. 62
1.3.1. The Local Analysis Approach p. 64
1.3.2. The Role of the Condition Number p. 65
1.3.3. Convergence Rate Results p. 75
1.4. Newton's Method and Variations p. 88
1.5. Least Squares Problems p. 102
1.5.1. The Gauss-Newton Method p. 107
1.5.2. Incremental Gradient Methods* p. 108
1.5.3. Incremental Forms of the Gauss-Newton Method* . . . . p. 119
1.6. Conjugate Direction Methods p. 130
1.7. Quasi-Newton Methods p. 148
1.8. Nonderivative Methods p. 158
1.8.1. Coordinate Descent p. 160
1.8.2. Direct Search Methods p. 162
1.9. Discrete-Time Optimal Control Problems* p. 166
1.10. Some Practical Guidelines p. 183
1.11. Notes and Sources p. 187

2. Optimization Over a Convex Set p. 191

2.1. Optimality Conditions p. 192
2.2. Feasible Directions and the Conditional Gradient Method . . p. 209
2.2.1. Descent Directions and Stepsize Rules p. 210
2.2.2. The Conditional Gradient Method p. 215
2.3. Gradient Projection Methods p. 223
2.3.1. Feasible Directions and Stepsize Rules Based on Projection p. 223

v
vi Contents

2.3.2. Convergence Analysis* p. 234

2.4. Two-Metric Projection Methods p. 244
2.5. Manifold Suboptimization Methods p. 250
2.6. Affine Scaling for Linear Programming p. 259
2.7. Block Coordinate Descent Methods* p. 267
2.8. Notes and Sources p. 272

3. Lagrange Multiplier Theory p. 275

3.1. Necessary Conditions for Equality Constraints p. 277
3.1.1. The Penalty Approach p. 281
3.1.2. The Elimination Approach p. 283
3.1.3. The Lagrangian Function p. 287
3.2. Sufficient Conditions and Sensitivity Analysis p. 295
3.2.1. The Augmented Lagrangian Approach p. 297
3.2.2. The Feasible Direction Approach p. 300
3.2.3. Sensitivity* p. 301
3.3. Inequality Constraints p. 307
3.3.1. Karush-Kuhn-Tucker Optimality Conditions p. 309
3.3.2. Conversion to the Equality Case* p. 312
3.3.3. Second Order Sumciency Conditions and Sensitivity* . . . p. 314
3.3.4. Sufficiency Conditions and Lagrangian Minimization* . . p. 315
3.3.5. Fritz John Optimality Conditions* p. 317
3.3.6. Refinements* p. 330
3.4. Linear Constraints and Duality* p. 357
3.4.1. Convex Cost Functions and Linear Constraints p. 357
3.4.2. Duality Theory: A Simple Form for Linear Constraints . . p. 359
3.5. Notes and Sources p. 367

4. Lagrange Multiplier Algorithms p. 369

4.1. Barrier and Interior Point Methods p. 370
4.1.1. Linear Programming and the Logarithmic Barrier* . . . . p. 373
4.2. Penalty and Augmented Lagrangian Methods p. 388
4.2.1. The Quadratic Penalty Function Method p. 390
4.2.2. Multiplier Methods - Main Ideas p. 398
4.2.3. Convergence Analysis of Multiplier Methods* p. 407
4.2.4. Duality and Second Order Multiplier Methods* p. 410
4.2.5. The Exponential Method of Multipliers* p. 413
4.3. Exact Penalties - Sequential Quadratic Programming* . . . p. 421
4.3.1. Nondifferentiable Exact Penalty Functions p. 422
4.3.2. Differentiable Exact Penalty Functions p. 439
4.4. Lagrangian and Primal-Dual Interior Point Methods* . . . . p. 446
4.4.1. First-Order Methods p. 446
4.4.2. Newton-Like Methods for Equality Constraints p. 450
4.4.3. Global Convergence p. 460
Contents vii

4.4.4. Primal-Dual Interior Point Methods p. 463

4.4.5. Comparison of Various Methods p. 471
4.5. Notes and Sources p. 473

5. Duality and Convex Programming p. 477

5.1. The Dual Problem p. 479
5.1.1. Lagrange Multipliers p. 480
5.1.2. The Weak Duality Theorem p. 485
5.1.3. Characterization of Primal and Dual Optimal Solutions . . p. 490
5.1.4. The Case of an Infeasible or Unbounded Primal Problem . p. 491
5.1.5. Treatment of Equality Constraints p. 493
5.1.6. Separable Problems and Their Geometry p. 494
5.1.7. Additional Issues About Duality p. 498
5.2. Convex Cost - Linear Constraints* p. 503
5.2.1. Proofs of Duality Theorems p. 505
5.3. Convex Cost - Convex Constraints p. 511
5.4. Conjugate Functions and Fenchel Duality* p. 521
5.4.1. Monotropic Programming Duality p. 525
5.4.2. Network Optimization p. 529
5.4.3. Games and the Minimax Theorem p. 531
5.4.4. The Primal Function p. 534
5.4.5. A Dual View of Penalty Methods p. 536
5.4.6. The Proximal and Entropy Minimization Algorithms . . . p. 542
5.5. Discrete Optimization and Duality p. 558
5.5.1. Examples of Discrete Optimization Problems p. 559
5.5.2. Branch-and-Bound p. 567
5.5.3. Lagrangian Relaxation p. 576
5.6. Notes and Sources p. 587

6. Dual Methods p. 591

6.1. Dual Derivatives and Subgradients* p. 594
6.2. Dual Ascent Methods for Differentiable Dual Problems* . . . p. 600
6.2.1. Coordinate Ascent for Quadratic Programming p. 600
6.2.2. Decomposition and Primal Strict Convexity p. 603
6.2.3. Partitioning and Dual Strict Concavity p. 604
6.3. Nondifferentiable Optimization Methods* p. 609
6.3.1. Subgradient Methods p. 610
6.3.2. Approximate and Incremental Subgradient Methods . . . p. 614
6.3.3. Cutting Plane Methods p. 618
6.3.4. Ascent and Approximate Ascent Methods p. 625
6.4. Decomposition Methods* p. 638
6.4.1. Lagrangian Relaxation of the Coupling Constraints . . . . p. 639
6.4.2. Decomposition by Right-Hand Side Allocation p. 642
6.5. Notes and Sources p. 645
viii Contents

Appendix A: Mathematical Background p. 647

A.l. Vectors and Matrices p. 648
A.2. Norms, Sequences, Limits, and Continuity p. 649
A.3. Square Matrices and Eigenvalues p. 656
A.4. Symmetrie and Positive Definite Matrices p. 659
A.5. Derivatives p. 664
A.6. Contraction Mappings >. . . . p. 669
Appendix B: Convex Analysis p. 671
B.l. Convex Sets and Functions p. 671
B.2. Separating Hyperplanes p. 689
B.3. Cones and Polyhedral Convexity p. 694
B.4. Extreme Points p. 701
B.5. Differentiability Issues p. 707
Appendix C: Line Search Methods p. 723
C.l. Cubic Interpolation p. 723
C.2. Quadratic Interpolation p. 724
C.3. The Golden Section Method p. 726
Appendix D: Implementation of Newton's Method . . . p. 729
D.I. Cholesky Factorization p. 729
D.2. Application to a Modined Newton Method p. 731
References p. 735
Index p. 773

View publication stats