Article

Rule extraction from linear support vector machines

Authors:

Sathyakama Sandilya,

R. Bharat RaoAuthors Info & Claims

KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

Pages 32 - 40

https://doi.org/10.1145/1081870.1081878

Published: 21 August 2005 Publication History

Abstract

We describe an algorithm for converting linear support vector machines and any other arbitrary hyperplane-based linear classifiers into a set of non-overlapping rules that, unlike the original classifier, can be easily interpreted by humans. Each iteration of the rule extraction algorithm is formulated as a constrained optimization problem that is computationally inexpensive to solve. We discuss various properties of the algorithm and provide proof of convergence for two different optimization criteria We demonstrate the performance and the speed of the algorithm on linear classifiers learned from real-world datasets, including a medical dataset on detection of lung cancer from medical images. The ability to convert SVM's and other "black-box" classifiers into a set of human-understandable rules, is critical not only for physician acceptance, but also to reducing the regulatory barrier for medical-decision support systems based on such classifiers.

References

[1]

D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, 1995.

[2]

Dimitri P. Bertsekas. Projected Newton methods for optimization problems with simple constraints. SIAM Journal on Control and Optimization, 20:221--246, 1982.

Digital Library

[3]

F. Beyer, L. Zierott, J. Stoeckel, W. Heindel, and D. Wormanns. Computer-assisted detection (cad) of pulmonary nodules at mdct: Can cad be used as concurrent reader? In Proceeding of the 11th European Congress of Radiology, Viena, Austria, March 2005. To appear.

[4]

E.H. Shortliffe B.G. Buchanan. Rule-Based Expert Systems: the MYCIN experiments of the Stanford Heuristic Programming Project. Addison-Wesley, Reading, MA, 1984.

Digital Library

[5]

P. S. Bradley and O. L. Mangasarian. Feature selection via concave minimization and support vector machines. In J. Shavlik, editor, Machine Learning Proceedings of the Fifteenth International Conference(ICML '98), pages 82--90, San Francisco, California, 1998. Morgan Kaufmann. ftp://ftp.cs.wisc.edu/math-prog/tech-reports/9803.ps.

Digital Library

[6]

V. Cherkassky and F. Mulier. Learning from Data - Concepts, Theory and Methods. John Wiley & Sons, New York, 1998.

Digital Library

[7]

G. Fung and O. L. Mangasarian. Proximal support vector machine classifiers. In F. Provost and R. Srikant, editors, Proceedings KDD-2001: Knowledge Discovery and Data Mining, August 26--29, 2001, San Francisco, CA, pages 77--86, New York, 2001. Asscociation for Computing Machinery. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/01-02.ps.

Digital Library

[8]

G. Fung, O. L. Mangasarian, and J. Shavlik. Knowledge-based support vector machine classifiers. Technical Report 01-09, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, November 2001. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/01-09.ps, NIPS 2002 Proceedings, to appear.

[9]

Glenn Fung. The disputed federalist papers: Svm feature selection via concave minimization. In TAPIA '03: Proceedings of the 2003 conference on Diversity in computing, pages 42--46. ACM Press, 2003.

Digital Library

[10]

F. J. Kurfes. Neural networks and structured knowledge: Rule extraction and applications. Applied Intelligence (Special Issue), 12(1-2):7--13, 2000.

Digital Library

[11]

O. L. Mangasarian. Generalized support vector machines. In A. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 135--146, Cambridge, MA, 2000. MIT Press. ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98-14.ps.

[12]

O. L. Mangasarian, W. N. Street, and W. H. Wolberg. Breast cancer diagnosis and prognosis via linear programming. Operations Research, 43(4):570--577, July-August 1995.

Digital Library

[13]

S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K.-R. Müller. Fisher discriminant analysis with kernels. In Y.-H. Hu, J. Larsen, E. Wilson, and S. Douglas, editors, Neural Networks for Signal Processing IX, pages 41--48. IEEE, 1999.

[14]

P. M. Murphy and D. W. Aha. UCI machine learning repository, 1992. www.ics.uci.edu/ mlearn/MLRepository.html.

[15]

Haydemar Nunez, Cecilio Angulo, and Andreu Catal. Rule extraction from support vector machines. In ESANN'2002 proceedings - European Symposium on Artificial Neural Networks, pages 107-112. d-side, 2002.

[16]

K. Preston. Computer processing of biomedical images. Computer, 9:54--68, 1976.

Digital Library

[17]

A. Tickle R. Andrews, J. Diederich. A survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8:373--389, 1995.

Digital Library

[18]

L. B. Lusted R. S. Ledley. Reasoning foundations of medical diagnosis. Science, 130:9--21, 1959.

[19]

J. Roehrig. The promise of cad in digital mammography. European Journal of Radiology, 31:35--39, 1999.

[20]

G. Towell & J. Shavlik. The extraction of refined rules from knowledge-based neural networks. Machine Learning, 13:71--101, 1993.

Digital Library

[21]

J. A. K. Suykens and J. Vandewalle. Least squares support vector machine classifiers. Neural Processing Letters, 9(3):293--300, 1999.

Digital Library

[22]

V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, second edition, 2000.

Digital Library

Cited By

Kuo RChiu T(2024)Hybrid of jellyfish and particle swarm optimization algorithm-based support vector machine for stock market trend predictionApplied Soft Computing10.1016/j.asoc.2024.111394154(111394)Online publication date: Mar-2024
https://doi.org/10.1016/j.asoc.2024.111394
Vilone GLongo L(2023)Development of a Human-Centred Psychometric Test for the Evaluation of Explanations Produced by XAI MethodsExplainable Artificial Intelligence10.1007/978-3-031-44070-0_11(205-232)Online publication date: 21-Oct-2023
https://doi.org/10.1007/978-3-031-44070-0_11
Mishra Ph.D. KPandey Ph.D. SMishra Ph.D. KPandey Ph.D. S(2023)Behavior and Sentiment Analysis of Smart Digital Societies Using Deep Machine Learning TechnologiesCloud-IoT Technologies in Society 5.010.1007/978-3-031-28711-4_3(55-85)Online publication date: 22-Apr-2023
https://doi.org/10.1007/978-3-031-28711-4_3
Show More Cited By

Index Terms

Rule extraction from linear support vector machines
1. Computing methodologies
  1. Machine learning

Recommendations

Decompositional Rule Extraction from Support Vector Machines by Active Learning

Support vector machines (SVMs) are currently state-of-the-art for the classification task and, generally speaking, exhibit good predictive performance due to their ability to model nonlinearities. However, their strength is also their main weakness, as ...
Rule extraction for support vector machine using input space expansion
ACIIDS'11: Proceedings of the Third international conference on Intelligent information and database systems - Volume Part II

Fuzzy Rule-Based System (FRB) in the form of human comprehensible IF-THEN rules can be extracted from Support Vector Machine (SVM) which is regarded as a black-boxed system. We first prove that SVM decision network and the zero-ordered Sugeno FRB type ...
Rule extraction from trained support vector machines
PAKDD'05: Proceedings of the 9th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining

Support vector machine (SVM) is applied to many research fields because of its good generalization ability and solid theoretical foundation. However, as the model generated by SVM is like a black box, it is difficult for user to interpret and understand ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

August 2005

844 pages

ISBN:159593135X

DOI:10.1145/1081870

General Chair:
Robert Grossman
University of Illinois at Chicago & Open Data Partners, USA
,
Program Chairs:
Roberto Bayardo
IBM Almaden Research, USA
,
Kristin Bennett
RPI, USA

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD05

Sponsor:

KDD05: The Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 21 - 24, 2005

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

107
Total Citations
View Citations
1,175
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)2

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kuo RChiu T(2024)Hybrid of jellyfish and particle swarm optimization algorithm-based support vector machine for stock market trend predictionApplied Soft Computing10.1016/j.asoc.2024.111394154(111394)Online publication date: Mar-2024
https://doi.org/10.1016/j.asoc.2024.111394
Vilone GLongo L(2023)Development of a Human-Centred Psychometric Test for the Evaluation of Explanations Produced by XAI MethodsExplainable Artificial Intelligence10.1007/978-3-031-44070-0_11(205-232)Online publication date: 21-Oct-2023
https://doi.org/10.1007/978-3-031-44070-0_11
Mishra Ph.D. KPandey Ph.D. SMishra Ph.D. KPandey Ph.D. S(2023)Behavior and Sentiment Analysis of Smart Digital Societies Using Deep Machine Learning TechnologiesCloud-IoT Technologies in Society 5.010.1007/978-3-031-28711-4_3(55-85)Online publication date: 22-Apr-2023
https://doi.org/10.1007/978-3-031-28711-4_3
Goethals SMartens DEvgeniou T(2022)The non-linear nature of the cost of comprehensibilityJournal of Big Data10.1186/s40537-022-00579-29:1Online publication date: 7-Mar-2022
https://doi.org/10.1186/s40537-022-00579-2
Straub J(2022)Automating the design and development of gradient descent trained expert system networksKnowledge-Based Systems10.1016/j.knosys.2022.109465254:COnline publication date: 27-Oct-2022
https://dl.acm.org/doi/10.1016/j.knosys.2022.109465
Vilone GLongo L(2021)Classification of Explainable Artificial Intelligence Methods through Their Output FormatsMachine Learning and Knowledge Extraction10.3390/make30300323:3(615-661)Online publication date: 4-Aug-2021
https://doi.org/10.3390/make3030032
Kassaie BIrving ETompa F(2021)Computer-Assisted Cohort Identification in PracticeACM Transactions on Computing for Healthcare10.1145/34834113:2(1-28)Online publication date: 20-Dec-2021
https://dl.acm.org/doi/10.1145/3483411
Dhebar YDeb K(2021)Interpretable Rule Discovery Through Bilevel Optimization of Split-Rules of Nonlinear Decision Trees for Classification ProblemsIEEE Transactions on Cybernetics10.1109/TCYB.2020.303300351:11(5573-5584)Online publication date: Nov-2021
https://doi.org/10.1109/TCYB.2020.3033003
Moradi MSamwald M(2021)Explaining Black-Box Models for Biomedical Text ClassificationIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2021.305674825:8(3112-3120)Online publication date: Aug-2021
https://doi.org/10.1109/JBHI.2021.3056748
Hirsch LNuovo AHaddela P(2021)Document Clustering with Evolved Single Word Search Queries2021 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC45853.2021.9504770(280-287)Online publication date: 28-Jun-2021
https://doi.org/10.1109/CEC45853.2021.9504770
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents