article

Support vector regression based hybrid rule extraction methods for forecasting

Authors:

M. A. H. Farquad,

S. Bapi RajuAuthors Info & Claims

Expert Systems with Applications: An International Journal, Volume 37, Issue 8

Pages 5577 - 5589

https://doi.org/10.1016/j.eswa.2010.02.055

Published: 01 August 2010 Publication History

Abstract

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM) introduced by Vapnik (1995). The main drawback of these newer techniques is their lack of interpretability. In other words, it is difficult for the human analyst to understand the knowledge learnt by these models during training. The most popular way to overcome this difficulty is to extract if-then rules from SVM and SVR. Rules provide explanation capability to these models and improve the comprehensibility of the system. Over the last decade, different algorithms for extracting rules from SVM have been developed. However rule extraction from SVR is not widely available yet. In this paper a novel hybrid approach for extracting rules from SVR is presented. The proposed hybrid rule extraction procedure has two phases: (1) Obtain the reduced training set in the form of support vectors using SVR (2) Train the machine leaning techniques (with explanation capability) using the reduced training set. Machine learning techniques viz., Classification And Regression Tree (CART), Adaptive Network based Fuzzy Inference System (ANFIS) and Dynamic Evolving Fuzzy Inference System (DENFIS) are used in the phase 2. The proposed hybrid rule extraction procedure is compared to stand-alone CART, ANFIS and DENFIS. Extensive experiments are conducted on five benchmark data sets viz. Auto MPG, Body Fat, Boston Housing, Forest Fires and Pollution, to demonstrate the effectiveness of the proposed approach in generating accurate regression rules. The efficiency of these techniques is measured using Root Mean Squared Error (RMSE). From the results obtained, it is concluded that when the support vectors with the corresponding predicted target values are used, the SVR based hybrids outperform the stand-alone intelligent techniques and also the case when the support vectors with the corresponding actual target values are used.

References

[1]

Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control. v25. 821-837.

[2]

Ancona, N. (1999). Classification properties of support vector machines for regression. Technical report, R.I.-IESI/CNR-Nr.,02/99.

[3]

UCI machine learning repository. University of California, School of Information and Computer Science, Irvine, CA.

[4]

Barakat, N. H., & Bradley, A. P. (2006). Rule extraction from support vector machines: Measuring the explanation capability using the area under the ROC curve. In The 18th international conference on pattern recognition (ICPR'06), Hong Kong.

Digital Library

[5]

Barakat, N. H., & Diederich, J. (2004). Learning-based rule-extraction from support vector machines. In Proceedings of the 14th international conference on computer theory and applications ICCTA'2004, Alexandria, Egypt.

[6]

Rule extraction from support vector machines: A sequential covering approach. IEEE Transactions on Knowledge and Data Engineering. v19 i6. 729-741.

Digital Library

[7]

Eclectic rule-extraction from support vector machines. International Journal of Computational Intelligence. v2 i1. 59-62.

[8]

Robust linear programming discrimination of two linearly inseparable sets. Optimization Methods and Software. v1. 23-34.

[9]

Classification and regression trees. Wadsworth International Group, Belmont, California.

[10]

A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery. v2. 1-43.

Digital Library

[11]

Chaves, Ad. C. F., Vellasco, M. M. B. R. & Tanscheit, R. (2005). Fuzzy rule extraction from support vector machines. In Fifth international conference on hybrid intelligent systems, November 06-09, 2005. Rio de Janeiro, Brazil.

Digital Library

[12]

A multiple kernel support vector machine scheme for feature selection and rule extraction from gene expression data of cancer tissue. Artificial Intelligence in Medicine. v41. 161-175.

Digital Library

[13]

Support vector networks. Machine Learning. v20. 273-297.

Digital Library

[14]

Cortez, P., & Morais, A. (2007). A data mining approach to predict forest fires using meteorological data. In J. Neves, M. F. Santos, & J. Machado (Eds.), New trends in artificial intelligence, proceedings of the 13th EPIA2007 - Portuguese conference on artificial intelligence, Guimarães, Portugal (pp. 512-523).

[15]

Craven, M. W., & Shavlik, J. W. (1994). Using sampling and queries to extract rules from trained neural networks. In Proceedings of the eleventh international conference on machine learning, San Francisco, CA, USA.

Digital Library

[16]

An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, New York, NY, USA.

Digital Library

[17]

Production rules as a representation for a knowledge-based consultation program. Artificial Intelligence. v8 i1. 15-45.

[18]

Farquad, M. A. H., Ravi, V., & Raju, S. B. (2008). Support vector machine based hybrid classifiers and rule extraction thereof: Application to bankruptcy prediction in banks. In E. Soria, J. D. Martí***n, R. Magdalena, M. Martí***nez, & A. J. Serrano (Eds.), Handbook of research on machine learning applications and trends: Algorithms, methods and techniques, IGI Global, USA.

[19]

Farquad, M. A. H., Ravi, V., & Raju, S. B. (2008). Rule extraction using support vector machine based hybrid classifier. Presented in TENCON-2008, IEEE region 10 conference, 19-21 November, Hyderabad, India.

[20]

Practical methods of optimization. John Wiley and Sons, New York.

Digital Library

[21]

Fu, X., Ong, C. J., Keerthi, S., Hung, G. G., & Goh, L. (2004). Extracting the knowledge embedded in support vector machines. In International joint conference on neural networks (IJCNN'04), Budapest, Hungary.

[22]

Rule extraction from linear support vector machines. In: KDD'05: proceeding of the eleventh ACM SIGKDD international conference on knowledge discovery in data mining, ACM Press, New York, NY, USA. pp. 32-40.

Digital Library

[23]

Connectionist expert systems. Communications of the ACM. v31 i2. 152-169.

Digital Library

[24]

Ingo, M., Michael, W., Ralf, K., Martin, S., & Timm, E. (2006). YALE: Rapid prototyping for complex data mining tasks. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining (KDD-06), Philadelphia, PA, USA.

Digital Library

[25]

ANFIS: Adaptive-network-based fuzzy inference system. IEEE Transactions on Systems, Man, Cybernetics. v23 i5/6. 665-685.

[26]

Joachims, T. (1998). Making large-scale SVM learning practical. Technical report, LS-8-24, Computer Science Department, University of Dortmund.

[27]

DENFIS: Dynamic, evolving neural-fuzzy inference systems and its application for time-series prediction. IEEE Transactions on Fuzzy Systems. v10. 144-154.

Digital Library

[28]

Decompositional rule extraction from support vector machines by active learning. IEEE Transactions on Knowledge and Data Engineering. v21 i2. 178-191.

Digital Library

[29]

Instabilities of regression estimates relating air pollution to mortality. Technometrics. v15. 463-482.

[30]

Learning machines: Foundations of trainable pattern classifying systems. McGraw-Hill.

[31]

Nunez, H., Angulo, C., & Catata, A. (2002). Rule extraction from support vector machines. In European symposium on artificial neural networks proceedings (pp. 107-112).

[32]

Osuna, E. E., Freund, R., & Girosi, F. (1997). Support vector machines: Training and applications. Technical report, Massachusetts Institute of Technology, Artificial Intelligence Laboratory, AI Memo No. 1602.

Digital Library

[33]

Penrose, K. W., Nelson, A. G., & Fisher, A. G. (1985). FACSM, Human Performance Research Center, Brigham Young University, Provo, Utah 84602 as listed in Medicine and Science in Sports and Exercise, 17 (2), 189.

[34]

Pontil, M., & Verri, A. (1997). Properties of support vector machines. Technical Report, Massachusetts Institute of Technology, Boston MA, USA.

Digital Library

[35]

Learning with kernals. MIT Press, Massachusetts, USA.

[36]

Gross composition of the body. In: Lawrence, J.H., Tobias, C.A. (Eds.), Advances in biological and medical physics, IV. Academic Press Inc., New York, USA.

[37]

Smola, A. J., & Scholkopf, B. (1998). A tutorial on support vector regression. NEUROCOLT2 Technical report series, NC2-TR-1998-030.

[38]

A tutorial on support vector regression. Statistics and Computing. v14 i3. 199-222.

Digital Library

[39]

Fuzzy identification of systems and its applications to modeling and control. IEEE Transactions on Systems Man and Cybernet. vSMC-15 i1. 116-132.

[40]

The nature of statistical learning theory. Springer-Verlag, New York Inc., New York, NY, USA.

Digital Library

[41]

Werbos, P. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University.

[42]

Fuzzy sets. Information and Control. v8. 338-353.

[43]

. Lecture notes in computer science, 2005.Springer, Berlin, Heidelberg.

Cited By

Tomasiello SPedrycz WLoia V(2022)On Fractional Tikhonov Regularization: Application to the Adaptive Network-Based Fuzzy Inference System for Regression ProblemsIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2022.315794730:11(4717-4727)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1109/TFUZZ.2022.3157947
Tomasiello SUzair M(2021)Some Remarks on ANFIS for Forest Fires Prediction2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ45933.2021.9494463(1-5)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1109/FUZZ45933.2021.9494463
Singh NSingh PBhagat D(2019)A rule extraction approach from support vector machines for diagnosing hypertension among diabeticsExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.04.029130:C(188-205)Online publication date: 15-Sep-2019
https://dl.acm.org/doi/10.1016/j.eswa.2019.04.029
Show More Cited By

Support vector regression based hybrid rule extraction methods for forecasting
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Rule extraction for support vector machine using input space expansion
ACIIDS'11: Proceedings of the Third international conference on Intelligent information and database systems - Volume Part II

Fuzzy Rule-Based System (FRB) in the form of human comprehensible IF-THEN rules can be extracted from Support Vector Machine (SVM) which is regarded as a black-boxed system. We first prove that SVM decision network and the zero-ordered Sugeno FRB type ...
Decompositional Rule Extraction from Support Vector Machines by Active Learning

Support vector machines (SVMs) are currently state-of-the-art for the classification task and, generally speaking, exhibit good predictive performance due to their ability to model nonlinearities. However, their strength is also their main weakness, as ...
Rule extraction from support vector machines based on consistent region covering reduction

Due to good performance in classification and regression, support vector machines have attracted much attention and become one of the most popular learning machines in last decade. As a black box, the support vector machine is difficult for users' ...

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 37, Issue 8

August, 2010

597 pages

ISSN:0957-4174

Issue’s Table of Contents

Copyright © Elsevier Ltd © 2010.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 August 2010

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tomasiello SPedrycz WLoia V(2022)On Fractional Tikhonov Regularization: Application to the Adaptive Network-Based Fuzzy Inference System for Regression ProblemsIEEE Transactions on Fuzzy Systems10.1109/TFUZZ.2022.315794730:11(4717-4727)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1109/TFUZZ.2022.3157947
Tomasiello SUzair M(2021)Some Remarks on ANFIS for Forest Fires Prediction2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ45933.2021.9494463(1-5)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1109/FUZZ45933.2021.9494463
Singh NSingh PBhagat D(2019)A rule extraction approach from support vector machines for diagnosing hypertension among diabeticsExpert Systems with Applications: An International Journal10.1016/j.eswa.2019.04.029130:C(188-205)Online publication date: 15-Sep-2019
https://dl.acm.org/doi/10.1016/j.eswa.2019.04.029
Ravi VTejasviram VSharma AKhansama RChakraborty PGupta MDey LRoy S(2017)Prediction Intervals via Support Vector-quantile Regression Random Forest HybridProceedings of the 10th Annual ACM India Compute Conference10.1145/3140107.3140122(109-113)Online publication date: 16-Nov-2017
https://dl.acm.org/doi/10.1145/3140107.3140122
Stoean CStoean R(2014)Post-evolution of variable-length class prototypes to unlock decision making within support vector machinesApplied Soft Computing10.1016/j.asoc.2014.09.01725:C(159-173)Online publication date: 1-Dec-2014
https://dl.acm.org/doi/10.1016/j.asoc.2014.09.017
Vadlamani RSharma A(2014)Support VectorQuantile Regression Random Forest Hybrid for Regression ProblemsProceedings of the 8th International Workshop on Multi-disciplinary Trends in Artificial Intelligence - Volume 887510.1007/978-3-319-13365-2_14(149-160)Online publication date: 8-Dec-2014
https://dl.acm.org/doi/10.1007/978-3-319-13365-2_14
Stoean RStoean C(2013)Modeling medical decision making by support vector machines, explaining by rules of evolutionary algorithms with feature selectionExpert Systems with Applications: An International Journal10.1016/j.eswa.2012.11.00740:7(2677-2686)Online publication date: 1-Jun-2013
https://dl.acm.org/doi/10.1016/j.eswa.2012.11.007
De Paz JBajo JGonzález ARodríguez SCorchado J(2012)Combining case-based reasoning systems and support vector regression to evaluate the atmosphere---ocean interactionKnowledge and Information Systems10.5555/3225656.322593930:1(155-177)Online publication date: 1-Jan-2012
https://dl.acm.org/doi/10.5555/3225656.3225939
Farquad MRavi VRaju S(2010)Rule extraction from support vector machine using modified active learning based approachProceedings of the 14th international conference on Knowledge-based and intelligent information and engineering systems: Part I10.5555/1893893.1893949(461-470)Online publication date: 8-Sep-2010
https://dl.acm.org/doi/10.5555/1893893.1893949

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents