Abstract
In this paper we aim to show that instance-based classification can replace the classifier component of a rule learner and of maximum-entropy modeling, thereby improving the generalization accuracy of both algorithms. We describe hybrid algorithms that combine rule learning models and maximum-entropy modeling with instance-based classification. Experimental results show that both hybrids are able to outperform the parent algorithm. We analyze and compare the overlap in errors and the statistical bias and variance of the hybrids, their parent algorithms, and a plain instance-based learner. We observe that the successful hybrid algorithms have a lower statistical bias component in the error than their parent algorithms; the fewer errors they make are also less systematic.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Cover, T.M., Hart, P.E.: Nearest neighbor pattern classification. Institute of Electrical and Electronics Engineers Transactions on Information Theory 13, 21–27 (1967)
Aha, D.W., Kibler, D., Albert, M.: Instance-based learning algorithms. Machine Learning 6, 37–66 (1991)
Domingos, P.: Unifying instance-based and rule-based induction. Machine Learning 24, 141–168 (1996)
Ting, K., Cameron-Jones, R.: Exploring a framework for instance based learning and naive bayesian classifiers. In: Proceedings of the Seventh Australian Joint Conference on Artificial Intelligence, pp. 100–107 (1994)
Sebag, M., Schoenauer, M.: A rule-based similarity measure. In: Wess, S., Richter, M., Althoff, K.-D. (eds.) EWCBR 1993. LNCS, vol. 837, pp. 119–130. Springer, Heidelberg (1994)
Cost, S., Salzberg, S.: A weighted nearest neighbour algorithm for learning with symbolic features. Machine Learning 10, 57–78 (1993)
Wettschereck, D., Aha, D.W., Mohri, T.: A review and comparative evaluation of feature-weighting methods for a class of lazy learning algorithms. Artificial Intelligence Review, special issue on Lazy Learning 11, 273–314 (1997)
Daelemans, W., Zavrel, J., Van der Sloot, K., Van den Bosch, A.: TiMBL: Tilburg Memory Based Learner, version 5.1, reference manual. Technical Report ILK-0402, ILK, Tilburg University (2004)
Guiasu, S., Shenitzer, A.: The principle of maximum entropy. The Mathematical Intelligencer 7 (1985)
Nocedal, J.: Updating quasi-Newton matrices with limited storage. Mathematics of Computation 35, 773–782 (1980)
Cohen, W.: Fast effective rule induction. In: Proceedings 12th International Conference on Machine Learning, pp. 115–123 (1995)
Hendrickx, I., Van den Bosch, A.: Maximum-entropy parameter estimation for the k-nn modified value-difference kernel. In: Proceedings of the 16th Belgian-Dutch Conference on Artificial Intelligence, pp. 19–26 (2004)
Van den Bosch, A.: Feature transformation through rule induction, a case study with the k-nn classifier. In: Proceedings on the workshop on advances in Inductive rule learning at the ECML/PKDD 2004, pp. 1–15 (2004)
Blake, C., Merz, C.: UCI repository of machine learning databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Kohavi, R., John, G.: Wrappers for feature subset selection. Artificial Intelligence Journal 97, 273–324 (1997)
Van den Bosch, A.: Wrapped progressive sampling search for optimizing learning algorithm parameters. In: Proceedings of the 16th Belgian-Dutch Conference on Artificial Intelligence, pp. 219–226 (2004)
Fayyad, U., Irani, K.: Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of the 13th International Joint Conference on Artificial Intelligence, pp. 1022–1027 (1993)
Brill, E., Wu, J.: Classifier combination for improved lexical disambiguation. In: Proceedings of the COLING-ACL 1998, pp. 191–195 (1998)
Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proceedings of the thirteenth International Conference on Machine Learning, pp. 275–283 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hendrickx, I., van den Bosch, A. (2005). Hybrid Algorithms with Instance-Based Classification. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds) Machine Learning: ECML 2005. ECML 2005. Lecture Notes in Computer Science(), vol 3720. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11564096_19
Download citation
DOI: https://doi.org/10.1007/11564096_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29243-2
Online ISBN: 978-3-540-31692-3
eBook Packages: Computer ScienceComputer Science (R0)