Abstract
Axis-parallel decision trees perform poorly on that multidimensional sparse data that are frequently input in many tasks. A straightforward solution is to create decision trees that have oblique splits; however, most training approaches have low performance. These models can easily overfit, so they should be combined with a random ensemble. This paper proposes an algorithm to train kernel decision trees. At each stump, the algorithm optimizes a loss function with a margin rescaling approach that simultaneously optimizes the margin and impurity criteria. We performed an experimental evaluation of several tasks, such as studying the reaction of social media users and image recognition. The experimental results show that the proposed algorithm trains ensembles that outperform other oblique or kernel forests in many datasets.
This is a preview of subscription content, log in via an institution to check access.
Explore related subjects
Find the latest articles, discoveries, and news in related topics.REFERENCES
Breiman, L., Friedman, J., Olshen, R., and Stone, C., Classification And Regression Trees, Routledge, 2017. https://doi.org/10.1201/9781315139470
Chen, T. and Guestrin, C., XGBoost: A scalable tree boosting system, Proceedings of the 22nd ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, San Francisco, 2016, New York: Association for Computing Machinery, 2016, pp. 785–794. https://doi.org/10.1145/2939672.2939785
Breiman, L., Random forests, Mach. Learn., 2001, vol. 45, no. 1, pp. 5–32. https://doi.org/10.1023/a:1010933404324
Golea, M., Bartlett, P., Lee, W.S., and Mason, L., Generalization in decision trees and DNF: Does size matter?, Advances in Neural Information Processing Systems, Jordan, M., Kearns, M., and Solla, S., Eds., MIT Press, 1997, vol. 10. https://proceedings.neurips.cc/paper_files/paper/1997/file/4f87658ef0de194413056248-a00ce009-Paper.pdf.
Vapnik, V.N., An overview of statistical learning theory, IEEE Trans. Neural Networks, 1999, vol. 10, no. 5, pp. 988–999. https://doi.org/10.1109/72.788640
Breiman, L., Technical note: Some properties of splitting criteria, Mach. Learn., 1996, vol. 24, no. 1, pp. 41–47. https://doi.org/10.1007/bf00117831
Liu, W. and Tsang, I., Sparse perceptron decision tree for millions of dimensions, Proc. AAAI Conf. Artif. Intell., 2016, vol. 30, no. 1. https://doi.org/10.1609/aaai.v30i1.10247
Liu, W. and Tsang, I.W., Making decision trees feasible in ultrahigh feature and label dimensions, J. Mach. Learn. Res., 2017, vol. 18, no. 1, pp. 2814–2849. https://doi.org/10.1142/9789811205675_0011
Bennett, K.P. and Blue, J.A., A support vector machine approach to decision trees, 1998 IEEE Int. Joint Conf. on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence, Anchorage, Alaska, 1998, IEEE, 1998, pp. 2396–2401. https://doi.org/10.1109/ijcnn.1998.687237
Menze, B.H., Kelm, B.M., Splitthoff, D.N., Koethe, U., and Hamprecht, F.A., On oblique random forests, Machine Learning and Knowledge Discovery in Databases, Gunopulos, D., Hofmann, T., Malerba, D., and Vazirgiannis, M., Eds., Lecture Notes in Computer Science, vol. 6912, Berlin: Springer, 2011, pp. 453–469. https://doi.org/10.1007/978-3-642-23783-6_29
Tibshirani, R. and Hastie, T., Margin trees for highdimensional classification, J. Mach. Learn. Res., 2007, vol. 8, no. 3, pp. 637–652.
Manwani, N. and Sastry, P.S., Geometric decision tree, IEEE Trans. Syst., Man, Cybern., Part B (Cybern.), 2011, vol. 42, no. 1, pp. 181–192. https://doi.org/10.1109/tsmcb.2011.2163392
Hofmann, T., Schölkopf, B., and Smola, A.J., Kernel methods in machine learning, Ann. Stat., 2008, vol. 36, no. 3, pp. 1171–1220. https://doi.org/10.1214/009053607000000677
Norouzi, M., Collins, M.D., Fleet, D.J., and Kohli, P., CO2 forest: Improved random forest by continuous optimization of oblique splits, 2015. https://doi.org/10.48550/arXiv.1506.06155
Tsochantaridis, I., Jochims, T., Hofmann, T., and Altun, Ya., Large margin methods for structured and interdependent output variables, J. Mach. Learn. Res., 2005, vol. 6, no. 9, pp. 1453–1484.
Yuille, A.L. and Rangarajan, A., The concave-convex procedure, Neural Comput., 2003, vol. 15, no. 4, pp. 915–936. https://doi.org/10.1162/08997660360581958
DeSalvo, G. and Mohri, M., Random composite forests, Proc. AAAI Conf. Artif. Intell., 2016, vol. 30, no. 1. https://doi.org/10.1609/aaai.v30i1.10203
Hehn, T.M., Kooij, J.F.P., and Hamprecht, F.A., End-to-end learning of decision trees and forests, Int. J. Comput. Vision, 2020, vol. 128, no. 4, pp. 997–1011. https://doi.org/10.1007/s11263-019-01237-6
İrsoy, O. and Alpaydin, E., Autoencoder trees, Proc. Mach. Learn. Res., 2016, vol. 45, pp. 378–390. http://proceedings.mlr.press/v45/Irsoy15.
Chai, Z. and Zhao, C., Multiclass oblique random forests with dual-incremental learning capacity, IEEE Trans. Neural Networks Learn. Syst., 2020, vol. 31, no. 12, pp. 5192–5203. https://doi.org/10.1109/tnnls.2020.2964737
Hecht-Nielsen, R., Theory of the backpropagation neural network, Neural Networks for Perception: Computation, Learning, and Architectures, Wechsler, H., Ed., Academic, 1992, pp. 65–93. https://doi.org/10.1016/b978-0-12-741252-8.50010-8
Yang, B., Shen, S., and Gao, W., Weighted oblique decision trees, Proc. AAAI Conf. Artif. Intell., 2019, vol. 33, no. 1, pp. 5621–5627. https://doi.org/10.1609/aaai.v33i01.33015621
Carreira-Perpinán, M.A. and Tavallali, P., Alternating optimization of decision trees, with application to learning sparse oblique trees, Adv. Neural Inf. Process. Syst., 2018, vol. 31. https://proceedings.neurips.cc/paper/2018/hash/185c29dc24325934ee377cfda20e414c-Abstract.html.
Kumar, M.A. and Gopal, M., A hybrid SVM based decision tree, Pattern Recognit., 2010, vol. 43, no. 12, pp. 3977–3987. https://doi.org/10.1016/j.patcog.2010.06.010
Krizhevsky, A., Learning multiple layers of features from tiny images, MSc Thesis, Toronto: Univ. of Toronto, 2009.
YouTube channels dataset. http://keen.isa.ru/youtube. Cited July 14, 2022.
Blake, C., UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html. Cited July 14, 2022.
Funding
This work was supported by ongoing institutional funding. No additional grants to carry out or direct this particular research were obtained.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors declare that they have no conflicts of interest.
Additional information
Publisher’s Note.
Allerton Press remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Devyatkin, D.A., Grigoriev, O.G. Method of Training a Kernel Tree. Sci. Tech. Inf. Proc. 50, 390–396 (2023). https://doi.org/10.3103/S0147688223050040
Received:
Published:
Issue Date:
DOI: https://doi.org/10.3103/S0147688223050040