Abstract
Sparse covariance selection problems can be formulated as log-determinant (log-det) semidefinite programming (SDP) problems with large numbers of linear constraints. Standard primal–dual interior-point methods that are based on solving the Schur complement equation would encounter severe computational bottlenecks if they are applied to solve these SDPs. In this paper, we consider a customized inexact primal–dual path-following interior-point algorithm for solving large scale log-det SDP problems arising from sparse covariance selection problems. Our inexact algorithm solves the large and ill-conditioned linear system of equations in each iteration by a preconditioned iterative solver. By exploiting the structures in sparse covariance selection problems, we are able to design highly effective preconditioners to efficiently solve the large and ill-conditioned linear systems. Numerical experiments on both synthetic and real covariance selection problems show that our algorithm is highly efficient and outperforms other existing algorithms.
Similar content being viewed by others
References
Banerjee O., El Ghaoui L., d’Aspremont A.: Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. J. Mach. Learn. Res. 9, 485–516 (2008)
Bilmes, J.A.: Natural statistical models for automatic speech recognition. PhD thesis, University of California, Berkeley (1999)
Burer S., Monteiro R.D.C., Zhang Y.: A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs. Math. Program. 95, 359–379 (2003)
Chen, S.S., Gopinath, R.A.: Model selection in acoustic modeling. In: Proc. EUROSPEECH’99, pp. 1087–1090, Budapest, Hungary (1999)
Dahl J., Vandenberghe L., Roychowdhury V.: Covariance selection for nonchordal graphs via chordal embedding. Optim. Methods Softw. 23, 501–520 (2008)
d’Aspremont, A.: Identifying small mean reverting portfolios. Quant. Finance (2010, to appear)
d’Aspremont A., Banerjee O., El Ghaoui L.: First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30, 56–66 (2008)
Dempster A.P.: Covariance selection. Biometrics 28, 157–175 (1972)
Dobra A.: Variable selection and dependency networks for genomewide data. Biostatistics 10, 621–639 (2009)
Edwards D.: Introduction to graphical modelling, 2nd edn. Springer, New York (2000)
Fan J., Feng Y., Wu Y.: Network exploration via the adaptive LASSO and SCAD penalties. Ann. Appl. Stat. 3, 521–541 (2009)
Freund, R., Nachtigal, N.: A new Krylov-subspace method for symmetric indefinite linear system. In: Proceedings of the 14th IMACS World Congress on Computational and Applied Mathematics, Atlanta, USA, pp. 1253–1256 (1994)
Friedman J., Hastie T., Tibshirani R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9, 432–441 (2008)
Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)
Hedenfalk I., Duggan D., Chen Y., Radmacher M., Bittner M., Simon R., Meltzer P., Gusterson B., Esteller M., Raffeld M., Yakhini Z., Ben-Dor A., Dougherty E., Kononen J., Bubendorf L., Fehrle W., Pittaluga S., Gruvberger S., Loman N., Johannsson O., Olsson H., Wilfond B., Sauter G., Kallioniemi O.-P., Borg A., Trent J.: Gene-expression profiles in hereditary breast cancer. N. Engl. J. Med. 344, 539–548 (2001)
Jarre F., Rendl F.: An augmented primal-dual method for linear conic programs. SIAM J. Optim. 19, 808–823 (2008)
Krishnamurthy, V., d’Aspremont, A.: A pathwise algorithm for covariance selection. Preprint (2009)
Lan, G., Lu, Z., Monterio, R.D.: Primal-dual first-order methods with \({\mathcal{O}(1/\epsilon)}\) iteration-complexity for cone programming. Math. Program. (2010, to appear)
Lauritzen, S.L.: Graphical models. In: Oxford Statistical Science Series, vol. 17. The Clarendon Press/Oxford University Press/Oxford Science Publications, New York (1996)
Lu Z.: Smooth optimization approach for sparse covariance selection. SIAM J. Optim. 19, 1807–1827 (2008)
Lu Z.: Adaptive first-order methods for general sparse inverse covariance selection. SIAM J. Matrix Anal. Appl. 31, 2000–2016 (2010)
Meinshausen N., Bühlmann P.: High-dimensional graphs and variable selection with the lasso. Ann. Stat. 34, 1436–1462 (2006)
Nesterov Y.: Smooth minimization of non-smooth functions. Math. Program. 103, 127–152 (2005)
Nesterov Y., Todd M.J.: Primal-dual interior-point methods for self-scaled cones. SIAM J. Optim. 8, 324–364 (1998)
Pittman J., Huang E., Dressman H., Horng C.-F., Cheng S.H., Tsou M.-H., Chen C.-M., Bild A., Iversen E.S., Huang A.T., Nevins J.R., West M.: Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes. Proc. Natl. Acad. Sci. USA 101(22), 8431–8436 (2004)
Rockafellar R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)
Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2003)
Sachs K., Perez O., Pe’er D., Lauffenburger D.A., Nolan G.P.: Causal protein-signaling networks derived from multiparameter single-cell data. Science 308, 523–529 (2005)
Scheinberg, K., Rish, I.: Learning sparse Gaussian Markov networks using a greedy coordinate ascent approach. In: Balcázar, J., Bonchi, F., Gionis, A., Sebag, M. (eds.) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science 6323. pp. 196–212 (2010)
Storey J.D., Tibshirani R.: Statistical significance for genome-wide studies. Proc. Natl. Acad. Sci. USA 100(16), 9440–9445 (2003)
Sturm J.F.: Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optim. Methods Softw. 11/12, 625–653 (1999)
Toh K.-C.: Solving large scale semidefinite programs via an iterative solver on the augmented systems. SIAM J. Optim. 14, 670–698 (2003)
Toh K.-C.: An inexact primal-dual path following algorithm for convex quadratic SDP. Math. Program. 112, 221–254 (2008)
Toh K.-C., Todd M.J., Tütüncü R.H.: SDPT3—a MATLAB software package for semidefinite programming, version 1.3. Optim. Methods Softw. 11/12, 545–581 (1999)
Tsuchiya T., Xia Y.: An extension of the standard polynomial-time primal-dual path-following algorithm to the weighted determinant maximization problem with semidefinite constraints. Pac. J. Optim. 3, 165–182 (2007)
Ueno U., Tsuchiya T.: Covariance regularization in inverse space. Q. J. R. Meteorol. Soc. 135, 1133–1156 (2009)
Vandenberghe L., Boyd S., Wu S.-P.: Determinant maximization with linear matrix inequality constraints. SIAM J. Matrix Anal. Appl. 19, 499–533 (1998)
Wang C., Sun D., Toh K.-C.: Solving log-determinant optimization problems by a newton-cg proximal point algorithm. SIAM J. Optim. 20, 2994–3013 (2010)
Whittaker, J.: Graphical models in applied multivariate statistics. In: Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, Chichester (1990)
Wille, A., Zimmermann, P., Vranová, E., Fürholz, A., Laule, O., Bleuler, S., Hennig, L., Prelić, A., von Rohr, P., Thiele, L., Zitzler, E., Gruissem, W., Bühlmann, P.: Sparse graphical gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana. Genome Biol. 5, R92 (2004)
Wong F., Carter C.K., Kohn R.: Efficient estimation of covariance selection models. Biometrika 90, 809–830 (2003)
Wu W.B., Pourahmadi M.: Nonparameteric estimation of large covariance matrices of longitudinal data. Biometrika 90, 831–844 (2003)
Yeung K.Y., Bumgarner R.E., Raftery A.E.: Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data. Bioinformatics 21, 2394–2402 (2005)
Yuan M., Lin Y.: Model selection and estimation in the Gaussian graphical model. Biometrika 94, 19–35 (2007)
Yuan, X.: Alternating direction methods for sparse covariance selection. Preprint (2009)
Zhang Y.: On extending some primal-dual interior-point algorithms from linear programming to semidefinite programming. SIAM J. Optim. 8, 365–386 (1998)
Zhao X.Y., Sun D., Toh K.-C.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20, 1737–1765 (2010)
Zhou G., Toh K.-C.: Polynomiality of an inexact infeasible interior point algorithm for semidefinite programming. Math. Program. 99, 261–282 (2004)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Li, L., Toh, KC. An inexact interior point method for L 1-regularized sparse covariance selection. Math. Prog. Comp. 2, 291–315 (2010). https://doi.org/10.1007/s12532-010-0020-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12532-010-0020-6
Keywords
- Log-determinant semidefinite programming
- Sparse inverse covariance selection
- Inexact interior point method
- Inexact search direction
- Iterative solver