research-article

An improved GLMNET for l1-regularized logistic regression

Authors:

Guo-Xun Yuan,

Chia-Hua Ho,

Chih-Jen LinAuthors Info & Claims

KDD '11: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 33 - 41

https://doi.org/10.1145/2020408.2020421

Published: 21 August 2011 Publication History

Get Access

Abstract

GLMNET proposed by Friedman et al. is an algorithm for generalized linear models with elastic net. It has been widely applied to solve L1-regularized logistic regression. However, recent experiments indicated that the existing GLMNET implementation may not be stable for large-scale problems. In this paper, we propose an improved GLMNET to address some theoretical and implementation issues. In particular, as a Newton-type method, GLMNET achieves fast local convergence, but may fail to quickly obtain a useful solution. By a careful design to adjust the effort for each iteration, our method is efficient regardless of loosely or strictly solving the optimization problem. Experiments demonstrate that the improved GLMNET is more efficient than a state-of-the-art coordinate descent method.

References

[1]

J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," Journal of Statistical Software, vol. 33, no. 1, pp. 1--22, 2010.

Crossref

Google Scholar

[2]

A. Genkin, D. D. Lewis, and D. Madigan, "Large-scale Bayesian logistic regression for text categorization," Technometrics, vol. 49, no. 3, pp. 291--304, 2007.

Crossref

Google Scholar

[3]

K. Koh, S.-J. Kim, and S. Boyd, "An interior-point method for large-scale l1-regularized logistic regression," Journal of Machine Learning Research, vol. 8, pp. 1519--1555, 2007.

Digital Library

Google Scholar

[4]

G. Andrew and J. Gao, "Scalable training of L1-regularized log-linear models," in Proceedings of the Twenty Fourth International Conference on Machine Learning (ICML), 2007.

Digital Library

Google Scholar

[5]

J. Liu, J. Chen, and J. Ye, "Large-scale sparse logistic regression," in Proceedings of The 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 547--556, 2009.

Digital Library

Google Scholar

[6]

M. Schmidt, G. Fung, and R. Rosales, "Fast optimization methods for l1 regularization: A comparative study and two new approaches," in Proceedings of European Conference on Machine Learning, pp. 286--297, 2007.

Digital Library

Google Scholar

[7]

G.-X. Yuan, K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "A comparison of optimization methods and software for large-scale l1-regularized linear classification," Journal of Machine Learning Research, vol. 11, pp. 3183--3234, 2010.

Digital Library

Google Scholar

[8]

P. Tseng and S. Yun, "A coordinate gradient descent method for nonsmooth separable minimization," Mathematical Programming, vol. 117, pp. 387--423, 2009.

Digital Library

Google Scholar

[9]

S. Yun and K.-C. Toh, "A coordinate gradient descent method for l1-regularized convex minimization," Computational Optimizations and Applications, vol. 48, no. 2, pp. 273--307, 2011.

Digital Library

Google Scholar

[10]

K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "Coordinate descent method for large-scale L2-loss linear SVM," Journal of Machine Learning Research, vol. 9, pp. 1369--1398, 2008.

Digital Library

Google Scholar

[11]

G.-X. Yuan, C.-H. Ho, and C.-J. Lin, "An improved GLMNET for l1-regularized logistic regression and support vector machines," tech. rep., National Taiwan University, 2011.

Google Scholar

[12]

H.-F. Yu, H.-Y. Lo, H.-P. Hsieh, J.-K. Lou, T. G. McKenzie, J.-W. Chou, P.-H. Chung, C.-H. Ho, C.-F. Chang, Y.-H. Wei, J.-Y. Weng, E.-S. Yan, C.-W. Chang, T.-T. Kuo, Y.-C. Lo, P. T. Chang, C. Po, C.-Y. Wang, Y.-H. Huang, C.-W. Hung, Y.-X. Ruan, Y.-S. Lin, S.-D. Lin, H.-T. Lin, and C.-J. Lin, "Feature engineering and classifier ensemble for KDD cup 2010," in JMLR Workshop and Conference Proceedings, 2011. To appear.

Google Scholar

[13]

R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, "LIBLINEAR: A library for large linear classification," Journal of Machine Learning Research, vol. 9, pp. 1871--1874, 2008.

Digital Library

Google Scholar

Cited By

View all

Lu FCurtin RRaff EFerraro FHolt JBaeza-Yates RBonchi F(2024)High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global UpdatesProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672038(2037-2047)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672038
Liu JZhang LHe FZhang CJiang SLi X(2024)Communication-Efficient Regret-Optimal Distributed Online Convex OptimizationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340388335:11(2270-2283)Online publication date: Nov-2024
https://doi.org/10.1109/TPDS.2024.3403883
Na KLee JKim E(2024)LF-Transformer: Latent Factorizer Transformer for Tabular LearningIEEE Access10.1109/ACCESS.2024.335497212(10690-10698)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3354972
Show More Cited By

Index Terms

An improved GLMNET for l1-regularized logistic regression
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees

Recommendations

Large-scale sparse logistic regression
KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

Logistic Regression is a well-known classification method that has been used widely in many applications of data mining, machine learning, computer vision, and bioinformatics. Sparse logistic regression embeds feature selection in the classification ...
An improved GLMNET for L1-regularized logistic regression

Recently, Yuan et al. (2010) conducted a comprehensive comparison on software for L1-regularized classification. They concluded that a carefully designed coordinate descent implementation CDN is the fastest among state-of-the-art solvers. In this paper, ...
An improved GLMNET for L1-regularized logistic regression

Recently, Yuan et al. (2010) conducted a comprehensive comparison on software for L1-regularized classification. They concluded that a carefully designed coordinate descent implementation CDN is the fastest among state-of-the-art solvers. In this paper, ...

Comments

Information & Contributors

Information

Published In

KDD '11: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2011

1446 pages

ISBN:9781450308137

DOI:10.1145/2020408

General Chair:
Chid Apte
IBM Research
,
Program Chairs:
Joydeep Ghosh
UT Austin
,
Padhraic Smyth
UC Irvine

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '11

Sponsor:

KDD '11: The 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 21 - 24, 2011

California, San Diego, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
771
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)5

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lu FCurtin RRaff EFerraro FHolt JBaeza-Yates RBonchi F(2024)High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global UpdatesProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672038(2037-2047)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672038
Liu JZhang LHe FZhang CJiang SLi X(2024)Communication-Efficient Regret-Optimal Distributed Online Convex OptimizationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340388335:11(2270-2283)Online publication date: Nov-2024
https://doi.org/10.1109/TPDS.2024.3403883
Na KLee JKim E(2024)LF-Transformer: Latent Factorizer Transformer for Tabular LearningIEEE Access10.1109/ACCESS.2024.335497212(10690-10698)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3354972
Wang PPriyadarshi R(2024)Policy Framework for Realizing Net-Zero Emission in Smart CitiesArchives of Computational Methods in Engineering10.1007/s11831-024-10131-5Online publication date: 30-Apr-2024
https://doi.org/10.1007/s11831-024-10131-5
vom Dahl SKanzow C(2024)An inexact regularized proximal Newton method without line searchComputational Optimization and Applications10.1007/s10589-024-00600-9Online publication date: 16-Aug-2024
https://doi.org/10.1007/s10589-024-00600-9
Ran WChen JZhao YZhang NLuo GZhao ZSong Y(2024) Global climate change‐driven impacts on the Asian distribution of Limassolla leafhoppers, with implications for biological and environmental conservation Ecology and Evolution10.1002/ece3.7000314:7Online publication date: 18-Jul-2024
https://doi.org/10.1002/ece3.70003
李飘(2023)A Smoothing Conjugate Gradient Algorithm for Sparse Logistic Regression ProblemsAdvances in Applied Mathematics10.12677/AAM.2023.12836412:08(3665-3683)Online publication date: 2023
https://doi.org/10.12677/AAM.2023.128364
Labrèche GAlvarez C(2023)SaaSyML: Software as a Service for Machine Learning On-board the OPS-SAT Spacecraft2023 IEEE Aerospace Conference10.1109/AERO55745.2023.10115531(1-9)Online publication date: 4-Mar-2023
https://doi.org/10.1109/AERO55745.2023.10115531
Orzek JArnold MVoelkle M(2023)Striving for Sparsity: On Exact and Approximate Solutions in Regularized Structural Equation ModelsStructural Equation Modeling: A Multidisciplinary Journal10.1080/10705511.2023.2189070(1-18)Online publication date: 11-May-2023
https://doi.org/10.1080/10705511.2023.2189070
Wee BSivakumar SLim KWong WJuwono F(2023)Diabetes detection based on machine learning and deep learning approachesMultimedia Tools and Applications10.1007/s11042-023-16407-583:8(24153-24185)Online publication date: 10-Aug-2023
https://doi.org/10.1007/s11042-023-16407-5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Large-scale sparse logistic regression

An improved GLMNET for L1-regularized logistic regression

An improved GLMNET for L1-regularized logistic regression