Article

Online feature selection using grafting

Authors:

Simon Perkins and

James TheilerAuthors Info & Claims

ICML'03: Proceedings of the Twentieth International Conference on International Conference on Machine Learning

August 2003

Pages 592 - 599

Published: 21 August 2003 Publication History

Abstract

In the standard feature selection problem, we are given a fixed set of candidate features for use in a learning problem, and must select a subset that will be used to train a model that is "as good as possible" according to some criterion. In this paper, we present an interesting and useful variant, the online feature selection problem, in which, instead of all features being available from the start, features arrive one at a time. The learner's task is to select a subset of features and return a corresponding model at each time step which is as good as possible given the features seen so far. We argue that existing feature selection methods do not perform well in this scenario, and describe a promising alternative method, based on a stagewise gradient descent technique which we call grafting.

References

[1]

Blake, C., & Merz, C. (1998). UCI repository of machine learning databases. www.ics.uci.edu/~mlearn/MLRepository.html. University of California, Irvine, Dept. of Information and Computer Science.

[2]

Boser, B., Guyon, I., & Vapnik, V. (1992). A training algorithm for optimal margin classifiers. Proc. Fifth Annual Workshop on Computational Learning Theory (pp. 144-152). Pittsburgh, ACM.

Digital Library

[3]

Chang, C., & Lin, C. (2001). LIBSVM: A library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.

[4]

Fletcher, R. (1987). Practical methods of optimization. Wiley. 2nd edition.

Digital Library

[5]

Freund, Y., & Schapire, R. (1996). Experiments with a new boosting algorithm. Machine Learning: Proc. 13th Int. Conf. (pp. 148-156). Morgan Kaufmann.

[6]

Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28, 337-307.

[7]

Hall, M. (2000). Correlation-based feature selection for discrete and numeric class machine learning. Proc. Int. Conf. Machine Learning (pp. 359-365). Morgan Kaufmann.

Digital Library

[8]

Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning. Springer.

[9]

Hoerl, A., & Kennard, R. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12, 55-67.

[10]

Kira, K., & Rendell, L. (1992). A practical approach to feature selection. Proc. Int. Conf. on Machine Learning (pp. 249-256). Morgan Kaufmann.

Digital Library

[11]

Kohavi, R., & John, G. (1997). Wrappers for feature subset selection. Artificial Intelligence, 97, 273-324.

Digital Library

[12]

Mallat, S., & Zhang, Z. (1993). Matching pursuit with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41, 3397-3415.

Digital Library

[13]

Perkins, S., Harvey, N.R., Brumby, S. P., & Lacker, K. (2001). Support vector machines for broad area feature classification in remotely sensed images. Proc. SPIE 4381, Aerosense 2001. Orlando.

[14]

Perkins, S., Lacker, K., & Theiler, J. (2003). Grafting: Fast, incremental feature selection by gradient descent in function space. Journal of Machine Learning Research. In press. Also at: http://niswww.lanl.gov/~simes/pubs.

Digital Library

[15]

Tibshirani, R. (1994). Regression shrinkage and selection via the lasso (Technical Report). Dept. of Statistics, University of Toronto.

Cited By

Zhao LGao YYe JChen FYe YLu CRamakrishnan N(2021)Spatio-Temporal Event Forecasting Using Incremental Multi-Source Feature LearningACM Transactions on Knowledge Discovery from Data10.1145/346497616:2(1-28)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1145/3464976
Chelmis CZois D(2021)Dynamic, Incremental, and Continuous Detection of Cyberbullying in Online Social MediaACM Transactions on the Web10.1145/344801415:3(1-33)Online publication date: 13-May-2021
https://dl.acm.org/doi/10.1145/3448014
Paul DKumar RSaha SMathew J(2021)Multi-objective Cuckoo Search-based Streaming Feature Selection for Multi-label DatasetACM Transactions on Knowledge Discovery from Data10.1145/344758615:6(1-24)Online publication date: 19-May-2021
https://dl.acm.org/doi/10.1145/3447586
Show More Cited By

Recommendations

Online streaming feature selection
ICML'10: Proceedings of the 27th International Conference on International Conference on Machine Learning

We study an interesting and challenging problem, online streaming feature selection, in which the size of the feature set is unknown, and not all features are available for learning while leaving the number of observations constant. In this problem, the ...
Read More
Online group feature selection
IJCAI '13: Proceedings of the Twenty-Third international joint conference on Artificial Intelligence

Online feature selection with dynamic features has become an active research area in recent years. However, in some real-world applications such as image analysis and email spam filtering, features may arrive by groups. Existing online feature selection ...
Read More
Online Feature Selection with Group Structure Analysis
Online selection of dynamic features has attracted intensive interest in recent years. However, existing online feature selection methods evaluate features individually and ignore the underlying structure of a feature stream. For instance, in image ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ICML'03: Proceedings of the Twentieth International Conference on International Conference on Machine Learning

August 2003

935 pages

ISBN:1577351894

Editors:
Tom Fawcett
HP Labs
,
Nina Mishra
HP Labs and Stanford University

Sponsors

Kluwer Academic Publishers
NSF: National Science Foundation
Kaidara Software
AAAI: American Association for Artificial Intelligence
Microsoft Research: Microsoft Research
HP: HP

Publisher

AAAI Press

Publication History

Published: 21 August 2003

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Zhao LGao YYe JChen FYe YLu CRamakrishnan N(2021)Spatio-Temporal Event Forecasting Using Incremental Multi-Source Feature LearningACM Transactions on Knowledge Discovery from Data10.1145/346497616:2(1-28)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1145/3464976
Chelmis CZois D(2021)Dynamic, Incremental, and Continuous Detection of Cyberbullying in Online Social MediaACM Transactions on the Web10.1145/344801415:3(1-33)Online publication date: 13-May-2021
https://dl.acm.org/doi/10.1145/3448014
Paul DKumar RSaha SMathew J(2021)Multi-objective Cuckoo Search-based Streaming Feature Selection for Multi-label DatasetACM Transactions on Knowledge Discovery from Data10.1145/344758615:6(1-24)Online publication date: 19-May-2021
https://dl.acm.org/doi/10.1145/3447586
Yao MChelmis CZois D(2019)Cyberbullying Ends Here: Towards Robust Detection of Cyberbullying in Social MediaThe World Wide Web Conference10.1145/3308558.3313462(3427-3433)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313462
Javidi MEskandari S(2019)Online streaming feature selectionPattern Analysis & Applications10.1007/s10044-018-0690-722:3(949-963)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10044-018-0690-7
Hajj NAwad M(2019)A piecewise weight update rule for a supervised training of cortical algorithmsNeural Computing and Applications10.1007/s00521-017-3167-531:6(1915-1930)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s00521-017-3167-5
Kim M(2018)Dynamic sparse coding for sparse time-series modeling via first-order smooth optimizationApplied Intelligence10.5555/3288064.328809648:11(3889-3901)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.5555/3288064.3288096
Hu XZhou PLi PWang JWu X(2018)A survey on online feature selection with streaming featuresFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-016-5489-312:3(479-493)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11704-016-5489-3
Li JCheng KWang SMorstatter FTrevino RTang JLiu H(2017)Feature SelectionACM Computing Surveys10.1145/313662550:6(1-45)Online publication date: 6-Dec-2017
https://dl.acm.org/doi/10.1145/3136625
Wu YHoi SMei TYu N(2017)Large-Scale Online Feature Selection for Ultra-High Dimensional Sparse DataACM Transactions on Knowledge Discovery from Data10.1145/307064611:4(1-22)Online publication date: 29-Jun-2017
https://dl.acm.org/doi/10.1145/3070646
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents