research-article

Stable feature selection via dense feature groups

Authors:

Steven LoscalzoAuthors Info & Claims

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 803 - 811

https://doi.org/10.1145/1401890.1401986

Published: 24 August 2008 Publication History

Abstract

Many feature selection algorithms have been proposed in the past focusing on improving classification accuracy. In this work, we point out the importance of stable feature selection for knowledge discovery from high-dimensional data, and identify two causes of instability of feature selection algorithms: selection of a minimum subset without redundant features and small sample size. We propose a general framework for stable feature selection which emphasizes both good generalization and stability of feature selection results. The framework identifies dense feature groups based on kernel density estimation and treats features in each dense group as a coherent entity for feature selection. An efficient algorithm DRAGS (Dense Relevant Attribute Group Selector) is developed under this framework. We also introduce a general measure for assessing the stability of feature selection algorithms. Our empirical study based on microarray data verifies that dense feature groups remain stable under random sample hold out, and the DRAGS algorithm is effective in identifying a set of feature groups which exhibit both high classification accuracy and stability.

References

[1]

U. Alon, N. Barkai, D. A. Notterman, et al. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl Acad. Sci. USA, 96:6745--6750, 1999

[2]

A. Appice, M. Ceci, S. Rawles, and P. Flach. Redundant feature elimination for multi-class problems. In Proceedings of the 21st International Conference on Machine learning, pages 33--40, 2004.

Digital Library

[3]

W. Au, K. C. C. Chan, A. K. C. Wong, and Y. Wang. Attribute clustering for grouping, selection, and classification of gene expression data. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2(2):83--101, 2005.

Digital Library

[4]

M. Berens, H. Liu, L. Parsons, L. Yu, and Z. Zhao. Fostering biological relevance in feature selection for microarray data. IEEE Intelligent Systems, 20(6):29--32, 2005.

[5]

R. Butterworth, G. Piatetsky-Shapiro, and D. A. Simovici. On feature selection through clustering. In Proceedings of the Fifth IEEE International Conference on Data Mining, pages 581--584, 2005.

Digital Library

[6]

B. Cao, D. Shen, J. Sun, Q. Yang, and Z. Chen. Feature selection in a kernel space. In Proceedings of the 24th International Conference on Machine learning, pages 121--127, 2007.

Digital Library

[7]

Y. Cheng. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17:790--799, 1995.

Digital Library

[8]

D. Comaniciu and P. Meer. Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:603--619, 2002.

Digital Library

[9]

M. Dash and H. Liu. Consistency-based search in feature selection. Artificial Intelligence, 151(1-2):155--176, 2003.

Digital Library

[10]

C. A. Davis, F. Gerick, V. Hintermair, C. C. Friedel, K. Fundel, R. Küffner, and R. Zimmer. Reliable gene signatures for microarray classification: assessment of stability and performance. Bioinformatics, 22:2356--2363, 2006.

Digital Library

[11]

C. Ding and H. Peng. Minimum redundancy feature selection from microarray gene expression data. In Proceedings of the Computational Systems Bioinformatics conference (CSB'03), pages 523--529, 2003.

Digital Library

[12]

L. Ein-Dor, I. Kela, G. Getz, D. Givol, and E. Domany. Outcome signature genes in breast cancer: is there a unique set? Bioinformatics, 21:171--178, 2005.

Digital Library

[13]

G. Forman. An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research, 3:1289--1305, 2003.

Digital Library

[14]

I. Guyon, J. Weston, S. Barnhill, and V. Vapnik. Gene selection for cancer classification using support vector machines. Machine Learning, 46:389--422, 2002.

Digital Library

[15]

T. Hastie, R. Tibshirani, D. Botstein, and P. Brown. Supervised harvesting of expression trees. Genome Biology, 2:0003.1-0003.12, 2001.

[16]

R. Jörnsten and B. Yu. Simultaneous gene clustering and subset selection for sample classification via MDL. Bioinformatics, 19:1100--1109, 2003.

[17]

A. Kalousis, J. Prados, and M. Hilario. Stability of feature selection algorithms: a study on high-dimensional spaces. Knowledge and Information Systems, 12:95--116, 2007.

Digital Library

[18]

R. Kohavi and G. H. John. Wrappers for feature subset selection. Artificial Intelligence, 97(1-2):273--324, 1997.

Digital Library

[19]

D. Koller and M. Sahami. Toward optimal feature selection. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 284--292, 1996.

Digital Library

[20]

T. Li, C. Zhang, and M. Ogihara. A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics, 20:2429--2437, 2004.

Digital Library

[21]

H. Liu and L. Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Trans. on Knowledge and Data Engineering, 17(3):1--12, 2005.

Digital Library

[22]

H. Liu and L. Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Trans. on Knowledge and Data Engineering, 17(3):1--12, 2005.

Digital Library

[23]

M. S. Pepe, R. Etzioni, Z. Feng, et al. Phases of biomarker development for early detection of cancer. J Natl Cancer Inst, 93:1054{1060, 2001.

[24]

M. Robnik-Sikonja and I. Kononenko. Theoretical and empirical analysis of Relief and ReliefF. Machine Learning, 53:23--69, 2003.

Digital Library

[25]

R. L. Somorjai, B. Dolenko, and R. Baumgartner. Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions. Bioinformatics, 19:1484--1491, 2003.

[26]

M. P. Wand and M. C. Jones. Kernel Smoothing. Chapman and Hall, 1995.

[27]

L. Wang, J. Zhu, and H. Zou. Hybrid huberized support vector machines for microarray classification. In Proceedings of the 24th International Conference on Machine learning, pages 983--990, 2007.

Digital Library

[28]

I. H. Witten and E. Frank. Data Mining - Pracitcal Machine Learning Tools and Techniques. Morgan Kaufmann Publishers, 2005.

Digital Library

[29]

E. Xing, M. Jordan, and R. Karp. Feature selection for high-dimensional genomic microarray data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 601--608, 2001.

Digital Library

[30]

L. Yu and H. Liu. Efficient feature selection via analysis of relevance and redundancy. Journal of Machine Learning Research, 5:1205--1224, 2004.

Digital Library

[31]

Y. Zhang, C. Ding, and T. Li. A two-stage gene selection algorithm by combining reliefF and mRMR. Proceedings of IEEE Conference of Bioinformatics and Bioengineering (BIBE2007), 2007.

Cited By

Marion RFrénay B(2024)Improving the Feature Selection Stability of the Delta Test in RegressionIEEE Transactions on Artificial Intelligence10.1109/TAI.2023.33131295:5(1911-1917)Online publication date: May-2024
https://doi.org/10.1109/TAI.2023.3313129
Han YSalido-Monzú DButt JSchweizer SWieser A(2024)A feature selection method for multimodal multispectral LiDAR sensingISPRS Journal of Photogrammetry and Remote Sensing10.1016/j.isprsjprs.2024.04.022212(42-57)Online publication date: Jun-2024
https://doi.org/10.1016/j.isprsjprs.2024.04.022
Kuzudisli CBakir-Gungor BBulut NQaqish BYousef M(2023)Review of feature selection approaches based on grouping of featuresPeerJ10.7717/peerj.1566611(e15666)Online publication date: 17-Jul-2023
https://doi.org/10.7717/peerj.15666
Show More Cited By

Index Terms

Stable feature selection via dense feature groups
1. Computing methodologies
  1. Machine learning
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Consensus group stable feature selection
KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining

Stability is an important yet under-addressed issue in feature selection from high-dimensional and small sample data. In this paper, we show that stability of feature selection has a strong dependency on sample size. We propose a novel framework for ...
A variance reduction framework for stable feature selection

Stability of feature selection is an important but under-addressed issue in knowledge discovery from high-dimensional data. In this study, we present a theoretical framework about the relationship between the stability and the accuracy of feature ...
Sparse regularization based feature selection: A survey
Abstract
Feature selection, as an essential preprocessing tool, aims to identify a subset of crucial features by eliminating redundant and noisy features according to a predefined criterion. In recent years, sparse learning has received considerable ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2008

1116 pages

ISBN:9781605581934

DOI:10.1145/1401890

General Chair:
Ying Li
Microsoft adCenter Labs
,
Program Chairs:
Bing Liu
University of Illinois at Chicago
,
Sunita Sarawagi
Indian Institute of Technology, Bombay

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD08

Sponsor:

KDD08: The 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2008

Nevada, Las Vegas, USA

Acceptance Rates

KDD '08 Paper Acceptance Rate 118 of 593 submissions, 20%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

117
Total Citations
View Citations
1,359
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)8

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Marion RFrénay B(2024)Improving the Feature Selection Stability of the Delta Test in RegressionIEEE Transactions on Artificial Intelligence10.1109/TAI.2023.33131295:5(1911-1917)Online publication date: May-2024
https://doi.org/10.1109/TAI.2023.3313129
Han YSalido-Monzú DButt JSchweizer SWieser A(2024)A feature selection method for multimodal multispectral LiDAR sensingISPRS Journal of Photogrammetry and Remote Sensing10.1016/j.isprsjprs.2024.04.022212(42-57)Online publication date: Jun-2024
https://doi.org/10.1016/j.isprsjprs.2024.04.022
Kuzudisli CBakir-Gungor BBulut NQaqish BYousef M(2023)Review of feature selection approaches based on grouping of featuresPeerJ10.7717/peerj.1566611(e15666)Online publication date: 17-Jul-2023
https://doi.org/10.7717/peerj.15666
BÜYÜKKEÇECİ MOKUR M(2023)A Comprehensive Review of Feature Selection and Feature Selection Stability in Machine LearningGazi University Journal of Science10.35378/gujs.99376336:4(1506-1520)Online publication date: 1-Dec-2023
https://doi.org/10.35378/gujs.993763
Spooner AMohammadi GSachdev PBrodaty HSowmya A(2023)Ensemble feature selection with data-driven thresholding for Alzheimer's disease biomarker discoveryBMC Bioinformatics10.1186/s12859-022-05132-924:1Online publication date: 9-Jan-2023
https://doi.org/10.1186/s12859-022-05132-9
Wang YLi XRuiz R(2023)Feature Selection With Maximal Relevance and Minimal Supervised RedundancyIEEE Transactions on Cybernetics10.1109/TCYB.2021.313989853:2(707-717)Online publication date: Feb-2023
https://doi.org/10.1109/TCYB.2021.3139898
García-Torres MRuiz RDivina F(2023)Evolutionary feature selection on high dimensional data using a search space reduction approachEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105556117(105556)Online publication date: Jan-2023
https://doi.org/10.1016/j.engappai.2022.105556
Rakesh DAnwit RJana P(2023)A new ranking-based stability measure for feature selection algorithmsSoft Computing10.1007/s00500-022-07767-527:9(5377-5396)Online publication date: 3-Jan-2023
https://doi.org/10.1007/s00500-022-07767-5
Zhou PZhao SYan YWu X(2022)Online Scalable Streaming Feature Selection via Dynamic DecisionACM Transactions on Knowledge Discovery from Data10.1145/350273716:5(1-20)Online publication date: 9-Mar-2022
https://dl.acm.org/doi/10.1145/3502737
Ling ZLi YZhang YYu KZhou PLi BWu X(2022)A Light Causal Feature Selection Approach to High-Dimensional DataIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3218786(1-13)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3218786
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents