Article

Lookahead-based algorithms for anytime induction of decision trees

Authors:

Shaul MarkovitchAuthors Info & Claims

ICML '04: Proceedings of the twenty-first international conference on Machine learning

Page 33

https://doi.org/10.1145/1015330.1015373

Published: 04 July 2004 Publication History

Abstract

The majority of the existing algorithms for learning decision trees are greedy---a tree is induced top-down, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Furthermore, the greedy algorithms require a fixed amount of time and are not able to generate a better tree if additional time is available. To overcome this problem, we present two lookahead-based algorithms for anytime induction of decision trees, thus allowing tradeoff between tree quality and learning time. The first one is depth-k lookahead, where a larger time allocation permits larger k. The second algorithm uses a novel strategy for evaluating candidate splits; a stochastic version of ID3 is repeatedly invoked to estimate the size of the tree in which each split results, and the one that minimizes the expected size is preferred. Experimental results indicate that for several hard concepts, our proposed approach exhibits good anytime behavior and yields significantly better decision trees when more time is available.

References

[1]

Baram, Y., El-Yaniv, R., & Luz, K. (2003). Online choice of active learning algorithms. ICML'03 (pp. 19--26).

[2]

Blake, C. L., & Merz, C. J. (1998). UCI repository of machine learning databases.

[3]

Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1987). Occam's Razor. Information Processing Letters, 24, 377--380.

Digital Library

[4]

Boddy, M., & Dean, T. L. (1994). Deliberation scheduling for problem solving in time constrained environments. Artificial Intelligence, 67, 245--285.

Digital Library

[5]

Bouckaert, R. R. (2003). Choosing between two learning algorithms based on calibrated tests. ICML'03 (pp. 51--58).

[6]

Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and regression trees. Monterey, CA: Wadsworth and Brooks.

[7]

Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2, 121--167.

Digital Library

[8]

Dong, M., & Kothari, R. (2001). Look-ahead based fuzzy decision tree induction. IEEE-FS, 9, 461--468.

Digital Library

[9]

Hyafil, L., & Rivest, R. L. (1976). Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5, 15--17.

[10]

Kononenko, I., & Kovacic, M. (1992). Learning as optimization: Stochastic generation of multiple knowledge. ICML'92 (pp. 257--262).

Digital Library

[11]

Minsky, M., & Papert, S. (1969). Perceptrons: Introduction to computational geometry. MIT Press, Cambridge, MA.

Digital Library

[12]

Murphy, O. J., & McCraw, R. L. (1991). Designing storage efficient decision trees. IEEE Transactions on Computers, 40, 315--320.

Digital Library

[13]

Murphy, P. M., & Pazzani, M. J. (1994). Exploring the decision forest: an empirical investigation of Occam's Razor in decision tree induction. Journal of Artificial Intelligence Research, 1, 257--275.

Digital Library

[14]

Murthy, S. K., & Salzberg, S. (1995). Lookahead and pathology in decision tree induction. IJCAI'95 (pp. 1025--1033).

Digital Library

[15]

Norton, S. W. (1989). Generating better decision trees. IJCAI'89 (pp. 800--805).

[16]

Page, D., & Ray, S. (2003). Skewing: An efficient alternative to lookahead for decision tree induction. IJCAI'03.

Digital Library

[17]

Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81--106.

[18]

Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Mateo, CA: Morgan Kaufmann.

Digital Library

[19]

Ragavan, H., & Rendell, L. (1993). Lookahead feature construction for learning hard concepts. ICML'93 (pp. 252--259).

[20]

Russell, S. J., & Zilberstein, S. (1991). Composing real-time systems. IJCAI'91 (pp. 212--217).

[21]

Sarkar, U. K., Chakrabarti, P., Ghose, S., & DeSarkar, S. C. (1994). Improving greedy algorithms by lookahead search. Journal of Algorithms, 16, 1--23.

Digital Library

[22]

Schapire, R. (1999). A brief introduction to boosting. IJCAI'99 (pp. 1401--1406).

Digital Library

[23]

Webb, G. I. (1996). Further experimental evidence against the utility of Occam's Razor. Journal of Artificial Intelligence Research, 4, 397--417.

Digital Library

Cited By

Kiossou HSchaus PNijssen SAglin G(2024)Efficient Lookahead Decision TreesAdvances in Intelligent Data Analysis XXII10.1007/978-3-031-58553-1_11(133-144)Online publication date: 16-Apr-2024
https://doi.org/10.1007/978-3-031-58553-1_11
Najat RHaitam EJaafar A(2023)A Breast Cancer Detection Problem using various Machine Learning Techniques in the Context of Health Prediction SystemE3S Web of Conferences10.1051/e3sconf/202341201093412(01093)Online publication date: 17-Aug-2023
https://doi.org/10.1051/e3sconf/202341201093
Haitam ENajat RJaafar A(2023)Machine Learning for a Medical Prediction System “Breast Cancer Detection” as a use caseE3S Web of Conferences10.1051/e3sconf/202341201092412(01092)Online publication date: 17-Aug-2023
https://doi.org/10.1051/e3sconf/202341201092
Show More Cited By

Lookahead-based algorithms for anytime induction of decision trees
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
2. Mathematics of computing
  1. Discrete mathematics
    1. Graph theory

Recommendations

Lookahead and pathology in decision tree induction
IJCAI'95: Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2

The standard approach to decision tree induction is a top-down greedy agorithm that makes locally optimal irrevocable decisions at each node of a tree. In this paper we empircally study an alternative approach in which the algorithms use one-level ...
Anytime Learning of Decision Trees

The majority of existing algorithms for learning decision trees are greedy---a tree is induced top-down, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Even the few non-greedy ...
Anytime Learning of Decision Trees

The majority of existing algorithms for learning decision trees are greedy---a tree is induced top-down, making locally optimal decisions at each node. In most cases, however, the constructed tree is not globally optimal. Even the few non-greedy ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '04: Proceedings of the twenty-first international conference on Machine learning

July 2004

934 pages

ISBN:1581138385

DOI:10.1145/1015330

Conference Chair:
Carla Brodley
Purdue University/Tufts University

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 July 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
856
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kiossou HSchaus PNijssen SAglin G(2024)Efficient Lookahead Decision TreesAdvances in Intelligent Data Analysis XXII10.1007/978-3-031-58553-1_11(133-144)Online publication date: 16-Apr-2024
https://doi.org/10.1007/978-3-031-58553-1_11
Najat RHaitam EJaafar A(2023)A Breast Cancer Detection Problem using various Machine Learning Techniques in the Context of Health Prediction SystemE3S Web of Conferences10.1051/e3sconf/202341201093412(01093)Online publication date: 17-Aug-2023
https://doi.org/10.1051/e3sconf/202341201093
Haitam ENajat RJaafar A(2023)Machine Learning for a Medical Prediction System “Breast Cancer Detection” as a use caseE3S Web of Conferences10.1051/e3sconf/202341201092412(01092)Online publication date: 17-Aug-2023
https://doi.org/10.1051/e3sconf/202341201092
Yu YZhang WO’Gara DLi JChang S(2023)A moment kernel machine for clinical data mining to inform medical decision makingScientific Reports10.1038/s41598-023-36752-713:1Online publication date: 28-Jun-2023
https://doi.org/10.1038/s41598-023-36752-7
Nanfack GTemple PFrénay B(2022)Constraint Enforcement on Decision Trees: A SurveyACM Computing Surveys10.1145/350673454:10s(1-36)Online publication date: 13-Sep-2022
https://dl.acm.org/doi/10.1145/3506734
Paranjape PDhabu MDeshpande P(2021)dSubSign: Classification of Instance-Feature Data Using Discriminative Subgraphs as Class SignaturesInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402150028531:07(917-947)Online publication date: 23-Jul-2021
https://doi.org/10.1142/S0218194021500285
Donick DLera S(2021)Uncovering feature interdependencies in high-noise environments with stepwise lookahead decision forestsScientific Reports10.1038/s41598-021-88571-311:1Online publication date: 29-Apr-2021
https://doi.org/10.1038/s41598-021-88571-3
Berthold MFillbrunn ASiebes A(2021)Widening: using parallel resources to improve model qualityData Mining and Knowledge Discovery10.1007/s10618-021-00749-5Online publication date: 9-Apr-2021
https://doi.org/10.1007/s10618-021-00749-5
Paranjape PDhabu MDeshpande P(2020)A novel classifier for multivariate instance using graph class signaturesFrontiers of Computer Science10.1007/s11704-019-8263-514:4Online publication date: 3-Jan-2020
https://doi.org/10.1007/s11704-019-8263-5
Tao YWang LAlmirall D(2018)Tree-based reinforcement learning for estimating optimal dynamic treatment regimesThe Annals of Applied Statistics10.1214/18-AOAS113712:3Online publication date: 1-Sep-2018
https://doi.org/10.1214/18-AOAS1137
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents