research-article

Intelligible models for classification and regression

Authors:

Rich Caruana, and

Johannes GehrkeAuthors Info & Claims

KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2012

Pages 150 - 158

https://doi.org/10.1145/2339530.2339556

Published: 12 August 2012 Publication History

Abstract

Complex models for regression and classification have high accuracy, but are unfortunately no longer interpretable by users. We study the performance of generalized additive models (GAMs), which combine single-feature models called shape functions through a linear function. Since the shape functions can be arbitrarily complex, GAMs are more accurate than simple linear models. But since they do not contain any interactions between features, they can be easily interpreted by users.

We present the first large-scale empirical comparison of existing methods for learning GAMs. Our study includes existing spline and tree-based methods for shape functions and penalized least squares, gradient boosting, and backfitting for learning GAMs. We also present a new method based on tree ensembles with an adaptive number of leaves that consistently outperforms previous work. We complement our experimental results with a bias-variance analysis that explains how different shape models influence the additive model. Our experiments show that shallow bagged trees with gradient boosting distinguish itself as the best method on low- to medium-dimensional datasets.

Supplementary Material

MP4 File (311b_m_talk_2.mp4)

Download
150.15 MB

References

[1]

http://archive.ics.uci.edu/ml/.

[2]

http://www.liaad.up.pt/~ltorgo/ Regression/DataSets.html.

[3]

http://www.cs.toronto.edu/~delve/data/ datasets.html.

[4]

http://osmot.cs.cornell.edu/kddcup/.

[5]

http://additivegroves.net.

[6]

E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning, 36(1):105--139, 1999.

Digital Library

[7]

H. Binder and G. Tutz. A comparison of methods for the fitting of generalized additive models. Statistics and Computing, 18(1):87--99, 2008.

Digital Library

[8]

L. Breiman. Random forests. Machine learning, 45(1):5--32, 2001.

Digital Library

[9]

L. Breiman and J. Friedman. Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association, pages 580--598,1985.

[10]

R. Caruana and A. Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In ICML, 2006.

Digital Library

[11]

G. Forman, M. Scholz, and S. Rajaram. Feature shaping for linear svm classifiers. In KDD, 2009.

Digital Library

[12]

J. Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29:1189--1232,2001.

[13]

J. Friedman. Stochastic gradient boosting. Computational Statistics and Data Analysis, 38:367--378, 2002.

Digital Library

[14]

T. Hastie and R. Tibshirani. Generalized additive models (with discussion). Statistical Science, 1:297--318, 1986.

[15]

T. Hastie and R. Tibshirani. Generalized additive models. Chapman & Hall/CRC, 1990.

[16]

G. Hooker. Generalized functional anova diagnostics for high-dimensional functions of dependent variables. Journal of Computational and Graphical Statistics, 16(3):709--732, 2007.

[17]

B. Panda, J. Herbach, S. Basu, and R. Bayardo. Planet: massively parallel learning of tree ensembles with mapreduce. PVLDB, 2009.

Digital Library

[18]

P. Ravikumar, H. Liu, J. Lafferty, and L. Wasserman. Sparse additive models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(5):1009--1030, 2009.

[19]

D. Sorokina, R. Caruana, and M. Riedewald. Additive groves of regression trees. In ECML, 2007.

Digital Library

[20]

S. Tyree, K. Weinberger, K. Agrawal, and J. Paykin. Parallel boosted regression trees for web search ranking. In WWW, 2011.

Digital Library

[21]

S. Wood. Thin plate regression splines. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(1):95--114, 2003.

[22]

S. Wood. Generalized additive models: an introduction with R. CRC Press, 2006.

Digital Library

Cited By

Al-Quraishi TKeong NG CMahdi OGyasi AAl-Quraishi N(2024)Advanced Ensemble Classifier Techniques for Predicting Tumor Viability in Osteosarcoma Histological Slide ImagesApplied Data Science and Analysis10.58496/ADSA/2024/0062024(52-68)Online publication date: 29-May-2024
https://doi.org/10.58496/ADSA/2024/006
Servati MVaccaro CDiller EPellegrino Da Silva RMafra FCao SStanley KCohen-Gadol AParker J(2024)Metabolic Insight into Glioma Heterogeneity: Mapping Whole Exome Sequencing to In Vivo Imaging with Stereotactic Localization and Deep LearningMetabolites10.3390/metabo1406033714:6(337)Online publication date: 16-Jun-2024
https://doi.org/10.3390/metabo14060337
Sun YChen CXu YXie SBlum RVenkitasubramaniam P(2024)On the generalization discrepancy of spatiotemporal dynamics-informed graph convolutional networksFrontiers in Mechanical Engineering10.3389/fmech.2024.139713110Online publication date: 12-Jul-2024
https://doi.org/10.3389/fmech.2024.1397131
Show More Cited By

Index Terms

Intelligible models for classification and regression
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Logical and relational learning
        Inductive logic learning

Recommendations

Convex Hull Ensemble Machine for Regression and Classification

We propose a new ensemble algorithm called Convex Hull Ensemble Machine (CHEM). CHEM in Hilbert space is first developed and modified for regression and classification problems. We prove that the ensemble model converges to the optimal model in Hilbert ...
Read More
Novel ensemble methods for regression via classification problems

Regression via classification (RvC) is a method in which a regression problem is converted into a classification problem. A discretization process is used to covert continuous target value to classes. The discretized data can be used with classifiers as ...
Read More
Bagging different instead of similar models for regression and classification problems

Even though many ensemble techniques have been proposed, there is as yet no clear picture of which method is best. In this study, we propose a technique that uses different subsets of the same training dataset with the concurrent usage of a voting (for ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining

August 2012

1616 pages

ISBN:9781450314626

DOI:10.1145/2339530

General Chair:
Qiang Yang
Hong Kong University of Science and Technology
,
Program Chairs:
Deepak Agarwal
LinkedIn
,
Jian Pei
Simon Fraser University

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '12

Sponsor:

KDD '12: The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 12 - 16, 2012

Beijing, China

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '24

Sponsor:
sigkdd
sigkdd

The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

233
Total Citations
View Citations
2,098
Total Downloads

Downloads (Last 12 months)188
Downloads (Last 6 weeks)22

Other Metrics

View Author Metrics

Citations

Cited By

Al-Quraishi TKeong NG CMahdi OGyasi AAl-Quraishi N(2024)Advanced Ensemble Classifier Techniques for Predicting Tumor Viability in Osteosarcoma Histological Slide ImagesApplied Data Science and Analysis10.58496/ADSA/2024/0062024(52-68)Online publication date: 29-May-2024
https://doi.org/10.58496/ADSA/2024/006
Servati MVaccaro CDiller EPellegrino Da Silva RMafra FCao SStanley KCohen-Gadol AParker J(2024)Metabolic Insight into Glioma Heterogeneity: Mapping Whole Exome Sequencing to In Vivo Imaging with Stereotactic Localization and Deep LearningMetabolites10.3390/metabo1406033714:6(337)Online publication date: 16-Jun-2024
https://doi.org/10.3390/metabo14060337
Sun YChen CXu YXie SBlum RVenkitasubramaniam P(2024)On the generalization discrepancy of spatiotemporal dynamics-informed graph convolutional networksFrontiers in Mechanical Engineering10.3389/fmech.2024.139713110Online publication date: 12-Jul-2024
https://doi.org/10.3389/fmech.2024.1397131
Wang X(2024)Human-Centered Evaluation of Explanations in AI-Assisted Decision-MakingCompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645239(134-136)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645239
An SCao Y(2024)Relative Keys: Putting Feature Explanation into ContextProceedings of the ACM on Management of Data10.1145/36392632:1(1-28)Online publication date: 26-Mar-2024
https://doi.org/10.1145/3639263
Vengroff D(2024)Impact Charts: A Tool for Identifying Systematic Bias in Social Systems and DataProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658965(1187-1198)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658965
Borisov VLeemann TSeßler KHaug JPawelczyk MKasneci G(2024)Deep Neural Networks and Tabular Data: A SurveyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.322916135:6(7499-7519)Online publication date: Jun-2024
https://doi.org/10.1109/TNNLS.2022.3229161
Jeyasothy ASuresh SRamasamy SSundararajan N(2024)Development of a Novel Transformation of Spiking Neural Classifier to an Interpretable ClassifierIEEE Transactions on Cybernetics10.1109/TCYB.2022.318118154:1(3-12)Online publication date: Jan-2024
https://doi.org/10.1109/TCYB.2022.3181181
Ortigossa EGonçalves TNonato L(2024)EXplainable Artificial Intelligence (XAI)—From Theory to Methods and ApplicationsIEEE Access10.1109/ACCESS.2024.340984312(80799-80846)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3409843
Tutul ANirjhar EChaspari T(2024)Investigating Trust in Human-AI Collaboration for a Speech-Based Data Analytics TaskInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2328910(1-19)Online publication date: 22-Mar-2024
https://doi.org/10.1080/10447318.2024.2328910
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents