Quoc Tran-Dinh

Linear Programming and Extensions

From Theory, Methods to Applications

January 2, 2018

STOR-UNC-Chapel Hill
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

The author reuses some parts of [16] in this lecture note. No part of this lecture note may be reproduced,
distributed, or transmitted in any form or by any means, including photocopying, recording, or other
electronic or mechanical methods, without the prior written permission of the authors, except in the
case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by
copyright law.

v/204
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 What is mathematical optimization? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Motivating examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.3 Optimization in operations research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.4 Terminologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Linear programming and nonlinear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.1 Linear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.2 Nonlinear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1 Graphical methods for solving LPs with two variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Types of linear programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 The forms of linear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.1 Forms of LPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2 Converting from general form to standard form . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.3 The construction of feasible solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Examples of linear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.1 Production planning problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.2 Transportation problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.3 Inventory problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.4 Assignment problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

vii
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

2.5 Geometric structures of the feasible set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.1 Affine hulls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5.2 Convex combination, convex hull, and convex sets . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5.3 Convex polyhedra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.6 Basic solutions and basic feasible solutions in standard form . . . . . . . . . . . . . . . . . . . . . . . 39
2.6.1 Basic solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.6.2 Basic feasible solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.6.3 The projected feasible set of a standard LP problem . . . . . . . . . . . . . . . . . . . . . . . . 42
2.6.4 Finding a basic feasible solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.5 The existence of optimal solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3 Simplex Methods for Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.1 The simplex method in matrix form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1.1 The basic idea of the simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.1.2 The complete simplex algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2 The simplex method with simplex tableaux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.1 Simplex tabeaux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.2 Simplex iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.2.4 Features of simplex tableaux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.5 Unique optimal solution vs. multiple optimal solutions . . . . . . . . . . . . . . . . . . . . . . 61
3.2.6 Unbounded linear programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.3 The two-phase simplex algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.3.1 Phase 1: Auxiliary LP problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.3.2 Phase 2: Performing the simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4 Duality Theory and Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.1 The primal and dual pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.1 A motivating example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.2 The primal-dual pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.3 Derivation of the dual problem from Lagrangian function . . . . . . . . . . . . . . . . . . . . 83

viii/204
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

4.2 Weak and strong duality, and complementarity slackness . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.2.1 Weak and strong duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.2.2 The complementary slackness theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.3 The dual simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.1 Basic feasible solutions of the dual and pseudo solutions . . . . . . . . . . . . . . . . . . . . 89
4.3.2 The dual simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.4 Theorem of the alternative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.5 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.5.2 Change the objective coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.5.3 Change the right hand side coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.6 Robust counterparts in linear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.6.1 Robust counterpart of an LP problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.6.2 The structure of uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

5 Introduction to Nonlinear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.1.1 Mathematical formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.1.2 Applications and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.1.3 The focus of this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.1.4 Convex functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.2 Convex quadratic programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2.1 Optimality condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.2 Unconstrained quadratic programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.3 Quadratic programming with equality constraints . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.2.4 Solution methods for general convex QP problems . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.2.5 Three representative applications of quadratic programming . . . . . . . . . . . . . . . . . 130
5.3 Second-order cone and semidefinite programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.3.1 Second-order cone programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.3.2 Semidefinite programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
5.3.3 Mixed conic programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.4 Interior-point methods for cone programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

ix/204
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

5.3.5 Software for linear cone programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

5.4 Convex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.4.1 Mathematical formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.4.2 Concrete applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.4.3 Unconstrained convex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.4.4 Constrained convex optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.4.5 Solution methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.4.6 A brief review on solution methods and software . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

A Overview of linear algebra and multivariable calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

A.1 Vectors and vector operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
A.1.1 Definitions and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
A.1.2 Basic operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
A.2 Matrices and matrix operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
A.2.1 Definitions and special cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
A.2.2 Basic matrix operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
A.2.3 Properties of matrix operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
A.2.4 Block matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
A.2.5 Invertible matrices, dependence and independence . . . . . . . . . . . . . . . . . . . . . . . . . 183
A.3 Systems of linear equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
A.3.1 Definitions and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
A.3.2 Elementary row operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
A.3.3 Gauss-Jordan’s method for solving systems of linear equations . . . . . . . . . . . . . . . 188
A.3.4 Gauss-Jordan’s method for computing the inverse of a square matrix . . . . . . . . . . 194
A.3.5 A relationship between linearly independent vectors and linear systems . . . . . . . . 195
A.4 Multivariable calculus, linear algebra, and convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
A.4.1 Multivariable functions: A review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
A.4.2 Symmetric and positive semidefinite matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
A.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

x/204
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

References
1. Farid Alizadeh, Jean-Pierre A Haeberly, and Michael L Overton. Primal-dual interior-point methods for semidefinite
programming: convergence rates, stability and numerical results. SIAM Journal on Optimization, 8(3):746–768, 1998.
2. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding agorithm for linear inverse problems. SIAM J.
Imaging Sci., 2(1):183–202, 2009.
3. A. Ben-Tal, L. El Ghaoui, and A. Nemirovski. Robust optimization. Princeton University Press, 2009.
4. A. Ben-Tal and A. Nemirovski. Lectures on modern convex optimization: Analysis, algorithms, and engineering
applications, volume 3 of MPS/SIAM Series on Optimization. SIAM, 2001.
5. Dimitri P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods. Athena Scientific, 1996.
6. Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena Scientific Belmont,
MA, 1997.
7. A. Björck. Numerical Methods for Least Squares Problems. SIAM, 1996.
8. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the
alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1):1–122, 2011.
9. G. B. Dantzig. Linear Programming and Extensions. Princeton University Press, 1963.
10. M. Fukuda, M. Kojima, and M. Shida. Lagrangian dual interior-point methods for semidefinite programs. SIAM J.
Optim., 12:1007–1031, 2002.
11. G.H. Golub and C.F. van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, 3rd edition, 1996.
12. M. Grant. Disciplined Convex Programming. PhD thesis, Stanford University, 2004.
13. M. Grant, S. Boyd, and Y. Ye. Disciplined convex programming. In L. Liberti and N. Maculan, editors, Global Op-
timization: From Theory to Implementation, Nonconvex Optimization and its Applications, pages 155–210. Springer,
2006.
14. C.T. Kelley. Iterative Methods for Linear and Nonlinear Equations. SIAM, 1995.
15. J. Löefberg. YALMIP : A Toolbox for Modeling and Optimization in MATLAB. In Proceedings of the CACSD
Conference, Taipei, Taiwan, 2004.
16. Shu Lu and Quoc Tran-Dinh. Introduction to Optimization. Lecture note, 2018.
17. JS Marron, Michael J Todd, and Jeongyoun Ahn. Distance-weighted discrimination. Journal of the American Statis-
tical Association, 102(480):1267–1271, 2007.
18. Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence O(1/k2 ). Doklady
AN SSSR, 269(translated as Soviet Math. Dokl.):543–547, 1983.
19. Y. Nesterov. Introductory lectures on convex optimization: A basic course, volume 87 of Applied Optimization. Kluwer
Academic Publishers, 2004.
20. Y. Nesterov. Smooth minimization of non-smooth functions. Math. Program., 103(1):127–152, 2005.
21. Y. Nesterov. Gradient methods for minimizing composite objective function. Math. Program., 140(1):125–161, 2013.
22. Y. Nesterov and A. Nemirovski. Interior-point Polynomial Algorithms in Convex Programming. Society for Industrial
Mathematics, 1994.
23. J. Nocedal and S.J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineer-
ing. Springer, 2 edition, 2006.
24. N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends in Optimization, 1(3):123–231, 2013.

203/204
STOR @ UNC Linear Programming and Extensions Quoc Tran-Dinh

25. J. Renegar. A mathematical view of interior-point methods in convex optimization, volume 2. SIAM, 2001.
26. R. T. Rockafellar. Convex Analysis, volume 28 of Princeton Mathematics Series. Princeton University Press, 1970.
27. Gilbert Strang, Gilbert Strang, Gilbert Strang, and Gilbert Strang. Introduction to linear algebra, volume 3. Wellesley-
Cambridge Press Wellesley, MA, 1993.
28. R.H. Tütünkü, K.C. Toh, and M.J. Todd. Solving semidefinite-quadratic-linear programs using SDPT3. Math. Pro-
gram., 95:189–217, 2003.
29. L. Vandenberghe and S. Boyd. Semidefinite programming. SIAM Review, 38(1):49–95, 1996.
30. R.J. Vanderbei. Linear Programming: Foundations and Extensions. Springer, 2015.
31. D. Verscheure, B. Demeulenaere, J. Swevers, J. De Schutter, and M. Diehl. Time-Optimal Path Tracking for Robots:
a Convex Optimization Approach. IEEE Trans. Autom. Control, 54:2318–2327, 2009.
32. H. Wolkowicz, R. Saigal, and L. Vandenberghe. Handbook of Semidefinite Programming: Theory, Algorithms, and
Applications. Kluwer Academic Publishers, 2000.
33. S.J. Wright. Primal-Dual Interior-Point Methods. SIAM Publications, Philadelphia, 1997.

204/204