research-article

Comprehensive study of feature selection methods to solve multicollinearity problem according to evaluation criteria

Expert Systems with Applications: An International Journal, Volume 76, Issue C

Pages 1 - 11

https://doi.org/10.1016/j.eswa.2017.01.048

Published: 15 June 2017 Publication History

Abstract

Complete study of feature selection methods in multicollinearity case was performed.The quadratic programming approach to treat multicollinearity problem was proposed.Test data sets representing diverse multicollinearity cases were used in experiments.The proposed approach outperforms other feature selection methods on data sets. This paper provides a new approach to feature selection based on the concept of feature filters, so that feature selection is independent of the prediction model. Data fitting is stated as a single-objective optimization problem, where the objective function indicates the error of approximating the target vector as some function of given features. Linear dependence between features induces the multicollinearity problem and leads to instability of the model and redundancy of the feature set. This paper introduces a feature selection method based on quadratic programming. This approach takes into account the mutual dependence of the features and the target vector, and selects features according to relevance and similarity measures defined according to the specific problem. The main idea is to minimize mutual dependence and maximize approximation quality by varying a binary vector that indicates the presence of features. The selected model is less redundant and more stable. To evaluate the quality of the proposed feature selection method and compare it with others, we use several criteria to measure instability and redundancy. In our experiments, we compare the proposed approach with several other feature selection methods, and show that the quadratic programming approach gives superior results according to the criteria considered for the test and real data sets.

References

[1]

Aha, D. W., & Bankert, R. L. (1996). A comparative evaluation of sequential feature selection algorithms. In Learning from data (pp. 199-206). Springer.

[2]

Askin, R. G. (1982). Multicollinearity in regression: Review and examples. Journal of Forecasting, 1(3), 281-292.

[3]

Belsley, D. A., Kuh, E., & Welsch, R. E. (2005). Regression diagnostics: Identifying influential data and sources of collinearity. New York: John Wiley & Sons.

[4]

Efron, B., Hastie, T., Johnstone, I., Tibshirani, R., et al. (2004). Least angle regression. The Annals of Statistics, 32(2), 407-499.

[5]

El-Dereny, M., & Rashwan, N. (2011). Solving multicollinearity problem using ridge regression models. International Journal of Contemporary Mathematical Sciences, 6, 585-600.

[6]

Estévez, P. A., Tesmer, M., Perez, C. A., & Zurada, J. M. (2009). Normalized mutual information feature selection. Neural Networks, IEEE Transactions on, 20(2), 189-201.

Digital Library

[7]

Ghamisi, P., & Benediktsson, J. A. (2015). Feature selection based on hybridization of genetic algorithm and particle swarm optimization. IEEE Geoscience and Remote Sensing Letters, 12(2), 309-313.

[8]

Gilmour, S. G. (1996). The interpretation of Mallows's C_p -Statistic. Journal of the Royal Statistical Society. Series D (The Statistician), 45(1), 49-56.

[9]

Grant, M., & Boyd, S. (2008). Graph implementations for nonsmooth convex programs. In V. Blondel, S. Boyd, & H. Kimura (Eds.), Recent advances in learning and control. In Lecture Notes in Control and Information Sciences (pp. 95-110). Springer-Verlag Limited. http://stanford.edu/~boyd/graph_dcp.html.

[10]

Grant, M., & Boyd, S. (2014). CVX: Matlab software for disciplined convex programming, version 2.1. http://cvxr.com/cvx.

[11]

Guyon, I. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157-1182.

Digital Library

[12]

Hall, M. A. (1999). Correlation-based feature selection for machine learning. The University of Waikato. Ph.D. thesis.

[13]

Harrell, F. E. (2001). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. Springer.

Digital Library

[14]

Katrutsa, A., & Strijov, V. (2015). Stress test procedure for feature selection algorithms. Chemometrics and Intelligent Laboratory Systems, 142, 172-183.

[15]

Ladha, L., & Deepa, T. (2011). Feature selection methods and algorithms. International Journal on Computer Science and Engineering, 3(5), 1787-1797.

[16]

Leamer, E. E. (1973). Multicollinearity: A Bayesian interpretation. The Review of Economics and Statistics, 55(3), 371-380.

[17]

Leardi, R. (2001). Genetic algorithms in chemometrics and chemistry: A review. Journal of Chemometrics, 15(7), 559-569.

[18]

Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2016). Feature selection: A data perspective. arXiv preprint arXiv:1601.07996.

[19]

Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.

[20]

Liu, H., & Motoda, H. (2012). Feature selection for knowledge discovery and data mining: 454. Springer Science & Business Media.

[21]

McQuarrie, A. D., & Tsai, C.-L. (1998). Regression and time series model selection. World Scientific.

[22]

Naghibi, T., Hoffmann, S., & Pfister, B. (2015). A semidefinite programming based search strategy for feature selection with mutual information measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8), 1529-1541.

Digital Library

[23]

Oluleye, B., Armstrong, L., Leng, J., & Diepeveen, D. (2014). A genetic algorithm-based feature selection. British Journal of Mathematics & Computer Science, In-Press.

[24]

Paul, R. K. (2006). Multicollinearity: Causes, Effects and Remedies. Technical Report. Working paper, unknown date. Accessed Apr. 23, 2013, http://pb8.ru/7hy.

[25]

Paul, S., & Das, S. (2015). Simultaneous feature selection and weighting--an evolutionary multi-objective optimization approach. Pattern Recognition Letters, 65, 51-59.

Digital Library

[26]

Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1226-1238.

Digital Library

[27]

Rodriguez-Lujan, I., Huerta, R., Elkan, C., & Cruz, C. S. (2010). Quadratic programming feature selection. Journal of Machine Learning Research, 11(Apr), 1491-1516.

Digital Library

[28]

Tibshirani, R. (1994). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58, 267-288.

[29]

Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67, 301-320.

Cited By

Tan JGui NQiu Z(2024)GAEFSKnowledge-Based Systems10.1016/j.knosys.2024.111523290:COnline publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111523
Chai CFan GYu HHuang ZDing JGuan Y(2024)Exploring better alternatives to size metrics for explainable software defect predictionSoftware Quality Journal10.1007/s11219-023-09656-y32:2(459-486)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11219-023-09656-y
Papageorgiou GSarlis VTjortjis C(2024)Evaluating the effectiveness of machine learning models for performance forecasting in basketball: a comparative studyKnowledge and Information Systems10.1007/s10115-024-02092-966:7(4333-4375)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s10115-024-02092-9
Show More Cited By

Comprehensive study of feature selection methods to solve multicollinearity problem according to evaluation criteria
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms

Recommendations

Individual optimal feature selection based on comprehensive evaluation indexs
ICIC'12: Proceedings of the 8th international conference on Intelligent Computing Theories and Applications

Individual optimal feature selection algorithm is simple and effective. Generally, single criterion is chosen for feature selection. But in this case, it is possible that features with good performance on other criteria will be neglected, and causing a ...
Empirical study of feature selection methods based on individual feature evaluation for classification problems

The use of feature selection can improve accuracy, efficiency, applicability and understandability of a learning process and its resulting model. For this reason, many methods of automatic feature selection have been developed. By using a modularization ...
Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods
Abstract
The evaluation of feature selection methods for text classification with small sample datasets must consider classification performance, stability, and efficiency. It is, thus, a multiple criteria decision-making (MCDM) problem. Yet ...
Highlights
- Evaluating feature selection methods for text classification with small datasets.

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 76, Issue C

June 2017

166 pages

ISSN:0957-4174

Issue’s Table of Contents

Copyright © Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 15 June 2017

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tan JGui NQiu Z(2024)GAEFSKnowledge-Based Systems10.1016/j.knosys.2024.111523290:COnline publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111523
Chai CFan GYu HHuang ZDing JGuan Y(2024)Exploring better alternatives to size metrics for explainable software defect predictionSoftware Quality Journal10.1007/s11219-023-09656-y32:2(459-486)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11219-023-09656-y
Papageorgiou GSarlis VTjortjis C(2024)Evaluating the effectiveness of machine learning models for performance forecasting in basketball: a comparative studyKnowledge and Information Systems10.1007/s10115-024-02092-966:7(4333-4375)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s10115-024-02092-9
Smiti SSoui MGhedira K(2024)Tri-XGBoost model improved by BLSmote-ENN: an interpretable semi-supervised approach for addressing bankruptcy predictionKnowledge and Information Systems10.1007/s10115-024-02067-w66:7(3883-3920)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s10115-024-02067-w
Singh PSingh SPaprzycki M(2023)Detection and elimination of multicollinearity in regression analysisInternational Journal of Knowledge-based and Intelligent Engineering Systems10.3233/KES-22162227:1(105-111)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/KES-221622
Zheng JGu YXie HWu G(2023)Linking innovation, empowerment to facilitate project performanceJournal of Engineering and Technology Management10.1016/j.jengtecman.2023.10175068:COnline publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.jengtecman.2023.101750
Saxena SJena BMohapatra BGupta NKalra MScartozzi MSaba LSuri J(2023)Fused deep learning paradigm for the prediction of o6-methylguanine-DNA methyltransferase genotype in glioblastoma patientsComputers in Biology and Medicine10.1016/j.compbiomed.2022.106492153:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.compbiomed.2022.106492
Derraz RMelissa Muharam FNurulhuda KAhmad Jaafar NKeng Yap N(2023)Ensemble and single algorithm models to handle multicollinearity of UAV vegetation indices for predicting rice biomassComputers and Electronics in Agriculture10.1016/j.compag.2023.107621205:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.compag.2023.107621
Ferrari Dacrema MMoroni FNembrini RFerro NFaggioli GCremonesi PAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Towards Feature Selection for Ranking and Classification Exploiting Quantum AnnealersProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531755(2814-2824)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531755
Khaire UDhanalakshmi R(2022)Stability of feature selection algorithmJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2019.06.01234:4(1060-1073)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2019.06.012
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents