Article

Direct sparsity optimization based feature selection for multi-class classification

Authors:

Yong FanAuthors Info & Claims

IJCAI'16: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence

Pages 1918 - 1924

Published: 09 July 2016 Publication History

Abstract

A novel sparsity optimization method is proposed to select features for multi-class classification problems by directly optimizing a l _2,p -norm (0 < p ≤ 1) based sparsity function subject to data-fitting inequality constraints to obtain large between-class margins. The direct sparse optimization method circumvents the empirical tuning of regularization parameters in existing feature selection methods that adopt the sparsity model as a regularization term. To solve the direct sparsity optimization problem that is nonsmooth and non-convex when 0 < p < 1, we propose an efficient iterative algorithm with proved convergence by converting it to a convex and smooth optimization problem at every iteration step. The proposed algorithm has been evaluated based on publicly available datasets. The experiments have demonstrated that our algorithm could achieve feature selection performance competitive to state-of-the-art algorithms.

References

[1]

Large margin classifiers: convex loss, low noise, and convergence rates. Advances in Neural Information Processing Systems 16. 16: 1173-1180, 2004.

Digital Library

[2]

Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proceedings of the National Academy of Sciences of the United States of America. 98: 13790-13795, 2001.

[3]

Feature selection via concave minimization and support vector machines. Proceedings of the International Conference on Machine Learning: 82-90, 1998.

Digital Library

[4]

Decoding by linear programming. Ieee Transactions on Information Theory. 51: 4203-4215, 2005.

Digital Library

[5]

Sparse multinomial ogistic regression via Bayesian l1 regularisation. Advances in Neural Information Processing Systems: 209-216, 2006.

Digital Library

[6]

LIBSVM: A Library for Support Vector Machines. Acm Transactions on Intelligent Systems and Technology. 2, 2011.

Digital Library

[7]

Restricted isometry properties and nonconvex compressive sensing. Inverse Problems. 24, 2008.

[8]

Sparse solutions to linear inverse problems with multiple measurement vectors. Ieee Transactions on Signal Processing. 53: 2477-2488, 2005.

Digital Library

[9]

Cvx Research, I. CVX: Matlab software for disciplined convex programming, version 2.0, http://cvxr.com/cvx, 2011.

[10]

Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. Ieee Transactions on Signal Processing. 45: 600-616, 1997.

Digital Library

[11]

An introduction to variable and feature selection. Journal of Machine Learning Research. 3: 1157-1182, 2003.

Digital Library

[12]

Gene selection for cancer classification using support vector machines. Machine Learning. 46: 389-422, 2002.

Digital Library

[13]

Feature Selection via Joint Embedding Learning and Sparse Regression. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, 2011.

Digital Library

[14]

A Practical Approach to Feature-Selection. Machine Learning: 249-256, 1992.

Digital Library

[15]

Efficient Algorithms for Selecting Features with Arbitrary Group Constraints via Group Lasso. 2013 Ieee 13th International Conference on Data Mining (Icdm): 379-388, 2013.

[16]

Exclusive Feature Learning on Arbitrary Structures via L1,2-norm. Advances in Neural Information Processing Systems, 2014.

[17]

Sparse multinomial logistic regression: Fast algorithms and generalization bounds. Ieee Transactions on Pattern Analysis and Machine Intelligence. 27: 957-968, 2005.

Digital Library

[18]

Feature-Selection and Feature-Extraction for Text Categorization. Speech and Natural Language: 212-217, 1992.

Digital Library

[19]

Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery. International Conference on Machine Learning, 2009.

Digital Library

[20]

Multi-Task Feature Learning Via Efficient L2,1-Norm Minimization. Uncertainty in Artificial Intelligence, 2009.

[21]

Support vector machines with adaptive L-q penalty. Computational Statistics & Data Analysis. 51: 6380-6394, 2007.

Digital Library

[22]

Exact 1-norm support vector machines via unconstrained convex differentiable minimization. Journal of Machine Learning Research: 1517-1530, 2006.

Digital Library

[23]

An Effcient Approach to Sparse Linear Discriminant Analysis. International Conference on Machine Learning, 2012.

Digital Library

[24]

Efficient and Robust Feature Selection via Joint L2,1-Norms Minimization. Advances in Neural Information Processing Systems, 2010.

[25]

Multi-task feature selection. Technical report, Department of Statistics, University of California, Berkeley, 2006.

[26]

Supplementary material of "Direct Sparsity Optimization Based Feature Selection for Multi-Class Classification", 2016.

[27]

Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. Ieee Transactions on Pattern Analysis and Machine Intelligence. 27: 1226-1238, 2005.

Digital Library

[28]

Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society Series B-Methodological. 58: 267-288, 1996.

[29]

Hybrid huberized support vector machines for microarray classification and gene selection. Bioinformatics. 24: 412-419, 2008.

Digital Library

[30]

Discriminative Least Squares Regression for Multiclass Classification and Feature Selection. Ieee Transactions on Neural Networks and Learning Systems. 23: 1738-1754, 2012.

[31]

L2,1-Norm Regularized Discriminative Feature Selection for Unsupervised Learning. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, 2011.

Digital Library

[32]

Feature Selection at the Discrete Limit. Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.

Digital Library

Cited By

Li JCheng KWang SMorstatter FTrevino RTang JLiu H(2017)Feature SelectionACM Computing Surveys10.1145/313662550:6(1-45)Online publication date: 6-Dec-2017
https://dl.acm.org/doi/10.1145/3136625
Peng HFan Y(2017)Feature selection by optimizing a lower bound of conditional mutual informationInformation Sciences: an International Journal10.1016/j.ins.2017.08.036418:C(652-667)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1016/j.ins.2017.08.036

Index Terms

Direct sparsity optimization based feature selection for multi-class classification
1. Computing methodologies
  1. Machine learning

Index terms have been assigned to the content through auto-classification.

Recommendations

Particle swarm optimisation for sparsity-based feature selection in multi-label classification
GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Multi-label classification (MLC) is an emerging real-world problem in which each instance is associated with multiple class labels simultaneously. Multi-label classification is challenging due to the complex interactions of features and multiple labels. ...
A novel low-rank hypergraph feature selection for multi-view classification

In order to select informative features from a high-dimensional multi-view dataset, we have proposed a feature selection method that simultaneously embedding the low-rank constraint, sparse representation, global and local structure learning into a ...
Robust multi-class feature selection via l2,0-norm regularization minimization

Feature selection is an important data preprocessing in data mining and machine learning, that can reduce the number of features without deteriorating model’s performance. Recently, sparse regression has received considerable attention in feature ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

IJCAI'16: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence

July 2016

4277 pages

ISBN:9781577357704

Editor:
Gerhard Brewka
Leipzig University, Germany

Sponsors

Sony: Sony Corporation
Arizona State University: Arizona State University
Microsoft: Microsoft
Facebook: Facebook
AI Journal: AI Journal

Publisher

AAAI Press

Publication History

Published: 09 July 2016

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li JCheng KWang SMorstatter FTrevino RTang JLiu H(2017)Feature SelectionACM Computing Surveys10.1145/313662550:6(1-45)Online publication date: 6-Dec-2017
https://dl.acm.org/doi/10.1145/3136625
Peng HFan Y(2017)Feature selection by optimizing a lower bound of conditional mutual informationInformation Sciences: an International Journal10.1016/j.ins.2017.08.036418:C(652-667)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1016/j.ins.2017.08.036

View Options

View options

Media

Figures

Other

Tables

View Table of Contents