Abstract
The details of a solver for minimizing a strictly convex quadratic objective function subject to general linear constraints are presented. The method uses a gradient projection algorithm enhanced with subspace acceleration to solve the bound-constrained dual optimization problem. Such gradient projection methods are well-known, but are typically employed to solve the primal problem when only simple bound-constraints are present. The main contributions of this work are threefold. First, we address the challenges associated with solving the dual problem, which is usually a convex problem even when the primal problem is strictly convex. In particular, for the dual problem, one must efficiently compute directions of infinite descent when they exist, which is precisely when the primal formulation is infeasible. Second, we show how the linear algebra may be arranged to take computational advantage of sparsity that is often present in the second-derivative matrix, mostly by showing how sparse updates may be performed for algorithmic quantities. We consider the case that the second-derivative matrix is explicitly available and sparse, and the case when it is available implicitly via a limited memory BFGS representation. Third, we present the details of our Fortran 2003 software package DQP, which is part of the GALAHAD suite of optimization routines. Numerical tests are performed on quadratic programming problems from the combined CUTEst and Maros and Meszaros test sets.
Similar content being viewed by others
Notes
Note that the sign of the inner product \(\langle b, v \rangle \) is arbitrary, since, for \(-v\), \(M^T (-v) = 0\) and \(\langle b, -v \rangle < 0\). We shall refer to a negative Fredholm alternative as that for which the signs of the components of v are flipped.
Available from http://galahad.rl.ac.uk/galahad-www/ along with a Matlab interface.
References
Arioli, M., Laratta, A., Menchi, O.: Numerical computation of the projection of a point onto a polyhedron. J. Optim. Theory Appl. 43(4), 495–525 (1984)
Axehill, D. Hansson, A.: A dual gradient projection quadratic programming algorithm tailored for model predictive control. In: Proceedings of the 47th IEEE Conference on Decision and Control, pp. 3057–3064, Cancun (2008)
Bartlett, R.A., Biegler, L.T.: QPSchur: a dual, active-set, Schur-complement method for large-scale and structured convex quadratic programming. Optim. Eng. 7(1), 5–32 (2006)
Benzi, M., Golub, G.H., Liesen, J.: Numerical solution of saddle point problems. Acta Numer. 14, 1–137 (2005)
Betts, J.T., Frank, P.D.: A sparse nonlinear optimization algorithm. J. Optim. Theory Appl. 82, 519–541 (1994)
Bierlaire, M., Toint, P.L., Tuyttens, D.: On iterative algorithms for linear least squares problems with bound constraints. Linear Algebra Appl. 143, 111–143 (1991)
Bisschop, J., Meeraus, A.: Matrix augmentation and partitioning in the updating of the basis inverse. Math. Program. 13(3), 241–254 (1977)
Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)
Boggs, P.T., Tolle, J.W.: Sequential quadratic programming. Acta Numer. 4, 1–51 (1995)
Boland, N.L.: A dual-active-set algorithm for positive semi-definite quadratic programming. Math. Program. Ser. A 78(1), 1–27 (1997)
Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63(2), 129–156 (1994)
Calamai, P.H., Moré, J.J.: Projected gradient methods for linearly constrained problems. Math. Program. 39(1), 93–116 (1987)
Conn, A.R., Gould, N.I.M.: On the location of directions of infinite descent for nonlinear programming algorithms. SIAM J. Numer. Anal. 21(6), 302–325 (1984)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Global convergence of a class of trust region algorithms for optimization with simple bounds. SIAM J. Numer. Anal. 25(2), 433–460 (1988). See also same journal 26, 764–767 (1989)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Testing a class of methods for solving minimization problems with simple bounds on the variables. Math. Comput. 50, 399–430 (1988)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. SIAM, Philadelphia (2000)
Curtis, F.E., Han, Z.: Globally Convergent Primal-Dual Active-Set Methods with Inexact Subproblem Solves. Technical Report 14T-010, COR@L Laboratory, Department of ISE, Lehigh University, 2014. In second review for SIAM Journal on Optimization
Curtis, F.E., Han, Z., Robinson, D.P.: A globally convergent primal-dual active-set framework for large-scale convex quadratic optimization. Comput. Optim. Appl. (2014). doi:10.1007/s10589-014-9681-9
Curtis, F.E., Johnson, T.C., Robinson, D.P., Wachter, A.: An inexact sequential quadratic optimization algorithm for nonlinear optimization. SIAM J. Optim. 24(3), 1041–1074 (2014)
Dominguez, J., González-Lima, M.D.: A primal-dual interior-point algorithm for quadratic programming. Numer. Algorithms 42(1), 1–30 (2006)
Dorn, W.: Duality in quadratic programming. Q. Appl. Math. 18, 155–162 (1960)
Dostál, Z.: Optimal Quadratic Programming Algorithms: With Applications to Variational Inequalities. Springer Optimization and Its Applications, vol. 23. Springer, New York (2009)
Dostál, Z., Schöberl, J.: Minimizing quadratic functions subject to bound constraints with the rate of convergence and finite termination. Comput. Optim. Appl. 30(1), 23–43 (2005)
Duff, I.S.: MA57—a code for the solution of sparse symmetric definite and indefinite systems. ACM Trans. Math. Softw. 30(2), 118–144 (2004)
Duff, I.S., Reid, J.K.: The multifrontal solution of indefinite sparse symmetric linear equations. ACM Trans. Math. Softw. 9(3), 302–325 (1983)
Fletcher, R.: An \(\ell _1\) penalty method for nonlinear constraints. In: Boggs, P.T., Byrd, R.H., Schnabel, R.B. (eds.) Numerical Optimization 1984, pp. 26–40. SIAM, Philadelphia (1985)
Forsgren, A., Gill, P.E., Wong, E.: Primal and dual active-set methods for convex quadratic programming. Math. Program. 159(1), 469–508 (2016)
Friedlander, M.P., Gould, N.I.M., Leyffer, S., Munson, T.: A filter active-set trust-region method. Technical Report Preprint ANL/MCS-P1456-0907, Argonne National Laboratory, Illinois (2007)
Friedlander, M.P., Leyffer, S.: Global and finite termination of a two-phase augmented Lagrangian filter method for general quadratic programs. SIAM J. Sci. Comput. 30(4), 1706–1729 (2008)
Friedlander, M.P., Orban, D.: A primal-dual regularized interior-point method for convex quadratic programs. Math. Program. Comput. 4(1), 71–107 (2012)
George, A., Liu, J.W.H.: Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood Cliffs (1981)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: A Schur-complement method for sparse quadratic programming. In: Cox, M.G., Hammarling, S.J. (eds.) Reliable Scientific Computation, pp. 113–138. Oxford University Press, Oxford (1990)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: Inertia-controlling methods for general quadratic programming. SIAM Rev. 33(1), 1–36 (1991)
Gill, P.E., Murray, W., Saunders, M.A.: User’s guide for QPOPT 1.0: a Fortran package for quadratic programming. Report SOL 95-4, Department of Operations Research, Stanford University, Stanford (1995)
Gill, P.E., Murray, W., Saunders, M .A., Wright, M .H.: A Schur-complement method for sparse quadratic programming. In: Cox, M .G., Hammarling, S .J. (eds.) Reliable Numerical Computation, pp. 113–138. Oxford University Press, Oxford (1990)
Goldfarb, D., Idnani, A.: A numerically stable dual method for solving strictly convex quadratic programs. Math. Program. 27(1), 1–33 (1983)
Gould, N.I.M., Orban, D., Robinson, D.P.: Trajectory-following methods for large-scale degenerate convex quadratic programming. Math. Program. Comput. 5(2), 113–142 (2013)
Gould, N.I.M., Orban, D., Toint, P.L.: GALAHAD—a library of thread-safe fortran 90 packages for large-scale nonlinear optimization. ACM Trans. Math. Softw. 29(4), 353–372 (2003)
Gould, N.I.M., Orban, D., Toint, P.L.: CUTEst: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput. Optim. Appl. 60(3), 545–557 (2015)
Gould, N.I.M., Toint, P.L.: A quadratic programming bibliography. Numerical Analysis Group Internal Report 2000-1, Rutherford Appleton Laboratory, Chilton, Oxfordshire, England, 2000. See http://www.numerical.rl.ac.uk/qp/qp.html
Gould, N.I.M., Toint, P.L.: SQP methods for large-scale nonlinear programming. In: Powell, M.J.D., Scholtes, S. (eds.) System Modelling and Optimization. Methods, Theory and Applications, pp. 149–178. Kluwer Academic Publishers, Dordrecht (2000)
Gould, N.I.M., Toint, P.L.: An iterative working-set method for large-scale non-convex quadratic programming. Appl. Numer. Math. 43(1–2), 109–128 (2002)
Gould, N.I.M., Toint, P.L.: Numerical methods for large-scale non-convex quadratic programming. In: Siddiqi, A.H., Kočvara, M. (eds.) Trends in Industrial and Applied Mathematics, pp. 149–179. Kluwer Academic Publishers, Dordrecht (2002)
Gould, N.I.M., Loh, Y., Robinson, D.P.: A filter method with unified step computation for nonlinear optimization. SIAM J. Optim. 24(1), 175–209 (2014)
Gould, N.I.M., Loh, Y., Robinson, D.P.: A filter SQP method: local convergence and numerical results. SIAM J. Optim. 25(3), 1885–1911 (2015)
Gould, N.I.M., Robinson, D.P.: A second derivative SQP method: global convergence. SIAM J. Optim. 20(4), 2023–2048 (2010)
Gould, N.I.M., Robinson, D.P.: A second derivative SQP method: local convergence and practical issues. SIAM J. Optim. 20(4), 2049–2079 (2010)
Gould, N.I.M., Robinson, D.P.: A second derivative SQP method with a ‘trust-region-free’ predictor step. IMA J. Numer. Anal. 32(2), 580–601 (2012)
Gu, Z., Rothberg, E., Bixby, R.: Gurobi Optimizer, version 5.5. 0. Software program (2013)
Gupta, A.: WSMP: Watson Sparse Matrix Package Part I—Direct Solution of Symmetric Sparse System. Research Report RC 21886, IBM T. J. Watson Research Center, Yorktown Heights (2010)
Hager, W.W., Hearn, D.W.: Application of the dual active set algorithm to quadratic network optimization. Comput. Optim. Appl. 1(4), 349–373 (1993)
Hintermüller, M., Kunisch, K.: The primal-dual active set strategy as a semismooth Newton method. SIAM J. Optim. 13(3), 865–888 (2002). (electronic) (2003)
Hogg, J.D., Reid, J.K., Scott, J.A.: Design of a multicore sparse Cholesky factorization using DAGs. SIAM J. Sci. Comput. 32(6), 36273649 (2010)
Hogg, J.D., Scott, J.A.: An indefinite sparse direct solver for large problems on multicore machines. Technical Report RAL-TR-2010-011, Rutherford Appleton Laboratory, Chilton (2010)
Hogg, J.D., Scott, J.A.: HSL_MA97: a bit-compatible multifrontal code for sparse symmetric systems. Technical Report RAL-TR-2011-024, Rutherford Appleton Laboratory, Chilton (2011)
ILOG CPLEX. High-performance software for mathematical programming and optimization (2005)
Lancaster, P., Tismenetsky, M.: The Theory of Matrices: With Applications, 2nd edn. Academic Press, London (1985)
Lescrenier, M.: Convergence of trust region algorithms for optimization with bounds when strict complementarity does not hold. SIAM J. Numer. Anal. 28(2), 476–495 (1991)
Maros, I., Meszaros, C.: A repository of convex quadratic programming problems. Optim. Methods Softw. 11–12, 671–681 (1999)
Mészáros, C.: The BPMPD interior point solver for convex quadratic problems. Optim. Methods Softw. 11(1–4), 431–449 (1999)
Morales, J.L., Nocedal, J., Wu, Y.: A sequential quadratic programming algorithm with an additional equality constrained phase. IMA J. Numer. Anal. 32, 553–579 (2012)
Moré, J.J., Toraldo, G.: On the solution of large quadratic programming problems with bound constraints. SIAM J. Optim. 1(1), 93–113 (1991)
Moré, J.J., Thuente, D.J.: Line search algorithms with guaranteed sufficient decrease. ACM Trans. Math. Softw. 20(3), 286–307 (1994)
Moyh-ud Din, H., Robinson, D.P.: A solver for nonconvex bound-constrained quadratic optimization. SIAM J. Optim. 25(4), 2385–2407 (2015)
Nocedal, J., Wright, S.J.: Numerical Optimization. Series in Operations Research, 2nd edn. Springer, Heidelberg (2006)
Polyak, R.A., Costa, J., Neyshabouri, S.: Dual fast projected gradient method for quadratic programming. Optim. Lett. (2012). doi:10.1007/s11590-012-0476-6
Powell, M.J.D.: ZQPCVX a FORTRAN subroutine for convex quadratic programming. University, Department of Applied Mathematics and Theoretical Physics (1983)
Reid, J.K., Scott, J.A.: An out-of-core sparse Cholesky solver. ACM Trans. Math. Softw. 36(2), 9 (2009)
Robinson, D.P., Feng, L., Nocedal, J., Pang, J.-S.: Subspace accelerated matrix splitting algorithms for asymmetric and symmetric linear complementarity problems. SIAM J. Optim. 23(3), 1371–1397 (2013)
Schenk, O., Gärtner, K.: On fast factorization pivoting methods for symmetric indefinite systems. Electron. Trans. Numer. Anal. 23, 158–179 (2006)
Schmid, C., Biegler, L.T.: Quadratic programming methods for reduced hessian sqp. Comput. Chem. Eng. 18(9), 817–832 (1994)
Spellucci, P.: Solving general convex QP problems via an exact quadratic augmented Lagrangian with bound constraints. Techn. Hochsch, Fachbereich Mathematik (1993)
Stefanov, S.M.: Polynomial algorithms for projecting a point onto a region defined by a linear constraint and box constraints in \(\mathbb{R}^n\). J. Appl. Math. 2004(5), 409–431 (2004)
Vanderbei, R.J.: LOQO: an interior point code for quadratic programming. Optim. Methods Softw. 11(1–4), 451–484 (1999)
Williams, J.W.J.: Algorithm 232. Heapsort. Commun. ACM 7, 347–348 (1964)
Yuan, G., Lu, S., Wei, Z.: A modified limited SQP method for constrained optimization. Appl. Math. 1(1), 8–17 (2010)
Zhu, C., Rockafellar, R.T.: Primal-dual projected gradient algorithms for extended linear-quadratic programming. SIAM J. Optim. 3(4), 751–783 (1993)
Acknowledgements
The authors are very grateful to Iain Duff for providing extensions to MA57 to cope with both sparse forward solution and the Fredholm alternative, and to Jonathan Hogg and Jennifer Scott for doing the same for MA77 and MA97. We are also grateful to Iain, Jonathan, Jennifer, Mario Arioli and Tyrone Rees for helpful discussions on Fredholm issues. N.I.M. Gould’s research supported by EPSRC Grants EP/I013067/1 and EP/M025179/1.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Gould, N.I.M., Robinson, D.P. A dual gradient-projection method for large-scale strictly convex quadratic problems. Comput Optim Appl 67, 1–38 (2017). https://doi.org/10.1007/s10589-016-9886-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-016-9886-1