Abstract
We analyze the proximal Newton method for minimizing a sum of a self-concordant function and a convex function with an inexpensive proximal operator. We present new results on the global and local convergence of the method when inexact search directions are used. The method is illustrated with an application to L1-regularized covariance selection, in which prior constraints on the sparsity pattern of the inverse covariance matrix are imposed. In the numerical experiments the proximal Newton steps are computed by an accelerated proximal gradient method, and multifrontal algorithms for positive definite matrices with chordal sparsity patterns are used to evaluate gradients and matrix-vector products with the Hessian of the smooth component of the objective.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Andersen MS, Vandenberghe, L (2015) CHOMPACK: a python package for chordal matrix computations, Version 2.2.1. cvxopt.github.io/chompack
Andersen MS, Dahl J, Vandenberghe L (2013) Logarithmic barriers for sparse matrix cones. Optim Methods Softw 28(3):396–423
Andersen M, Dahl J, Vandenberghe L (2015) CVXOPT: a python package for convex optimization. www.cvxopt.org
Becker SR, Candès EJ, Grant MC (2011) Templates for convex cone problems with applications to sparse signal recovery. Math Program Comput 3(3):165–218
Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202
Bertsekas DP (2009) Convex optimization theory. Athena Scientific, Belmont
Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge
Byrd RH, Nocedal J, Oztoprak F (2016) An inexact successive quadratic approximation method for L-1 regularized optimization. Math Program 157(2):375–396
Davis TA, Hu Y (2011) The University of Florida sparse matrix collection. ACM Trans Math Softw 38:1–25
Dembo RS, Eisenstat SC, Steihaug T (1982) Inexact Newton methods. SIAM J Numer Anal 19(2):400–408
Dempster AP (1972) Covariance selection. Biometrics 28:157–175
Eisenstat SC, Walker HF (1996) Choosing the forcing terms in an inexact Newton method. SIAM J Sci Comput 17(1):16–32
Hastie T, Tibshirani R, Wainwright M (2015) Statistical learning with sparsity. The lasso and generalizations. CRC Press, Boca Raton
Hiriart-Urruty J-B, Lemaréchal C (1993) Convex analysis and minimization algorithms I, volume 305 of Grundlehren der mathematischen Wissenschaften. Springer, New York
Hsieh C-J, Sustik MA, Dhillon IS, Ravikumar P (2011) Sparse inverse covariance matrix estimation using quadratic approximation. Adv Neural Inf Process (NIPS) 24:2330–2338
Kyrillidis A, Karimi-Mahabadi R, Tran-Dinh Q, Cevher V (2014) Scalable sparse covariance estimation via self-concordance. In: Proceedings of the 28th AAAI conference on artificial intelligence, pp 1946–1952
Lee JD, Sun Y, Saunders MA (2014) Proximal Newton-type methods for minimizing composite functions. SIAM J Optim 24(3):1420–1443
Moreau JJ (1965) Proximité et dualité dans un espace hilbertien. Bull Math Soc Fr 93:273–299
Nesterov Y (2004) Introductory lectures on convex optimization. Kluwer Academic Publishers, Dordrecht
Nesterov Y (2012) Towards non-symmetric conic optimization. Optim Methods Softw 27(4–5):893–917
Nesterov Y, Nemirovskii A (1994) Interior-point polynomial methods in convex programming, volume 13 of studies in applied mathematics. SIAM, Philadelphia
Olsen PA, Oztoprak F, Nocedal J, Rennie SJ (2012) Newton-like methods for sparse inverse covariance estimation. Adv Neural Inf Process (NIPS) 25:764–772
Renegar J (2001) A mathematical view of interior-point methods in convex optimization. SIAM, Philadelphia
Scheinberg K, Goldfarb D, Bai X (2014) Fast first-order methods for composite convex optimization with backtracking. Found Comput Math 14:389–417
Scheinberg K, Ma S (2012) Optimization methods for sparse inverse covariance selection. In: Sra S, Nowozin S, Wright SJ (eds) Optimization for machine learning. MIT Press, Cambridge, pp 455–477
Scheinberg K, Tang X (2013) Complexity of inexact proximal Newton methods. Technical Report 13T-02-R1, COR@L. Lehigh University, 2013
Tran-Dinh Q, Kyrillidis A, Cevher V (2014) An inexact proximal path-following algorithm for constrained convex optimization. SIAM J Optim 24(4):1718–1745
Tran-Dinh Q, Kyrillidis A, Cevher V (2015) Composite self-concordant minimization. J Mach Learn Res 16:371–416
Vandenberghe L, Andersen MS (2014) Chordal graphs and semidefinite optimization. Found Trends Optim 1(4):241–433
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by National Science Foundation Grants 1128817 and 1509789.
Rights and permissions
About this article
Cite this article
Li, J., Andersen, M.S. & Vandenberghe, L. Inexact proximal Newton methods for self-concordant functions. Math Meth Oper Res 85, 19–41 (2017). https://doi.org/10.1007/s00186-016-0566-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00186-016-0566-9