Abstract
Combining multiple classifier systems (MCS’) has been shown to outperform single classifier system. It has been demonstrated that improvement for ensemble performance depends on either the diversity among or the performance of individual systems. A variety of diversity measures and ensemble methods have been proposed and studied. It remains a challenging problem to estimate the ensemble performance in terms of the performance of and the diversity among individual systems. In this paper, we establish upper and lower bounds for P m (performance of the ensemble using majority voting) in terms of P̄(average performance of individual systems) and D̄ (average entropy diversity measure among individual systems). These bounds are shown to be tight using the concept of a performance distribution pattern (PDP) for the input set. Moreover, we showed that when P̄ is big enough, the ensemble performance P m resulting from a maximum (information-theoretic) entropy PDP is an increasing function with respect to the diversity measure D̄. Five experiments using data sets from various applications domains are conducted to demonstrate the complexity, richness, and diverseness of the problem in estimating the ensemble performance.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ho, T., Hull, J., Srihari, S.: Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 66–75 (1994)
Ho, T.K.: Multiple classifier combination: Lessons and next steps. In: Bunke, H., Kandel, A. (eds.) Hybrid Methods in Pattern Recognition, pp. 171–198. World Scientific, Singapore (2002)
Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons, Chichester (2004)
Sharkey, A. (ed.): Combining Artificial Neural Nets. Springer, Heidelberg (1999)
Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)
Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51, 181–207 (2003)
Kuncheva, L.: That elusive diversity in classifier ensembles. In: Perales, F.J., et al. (eds.) IbPRIA 2003. LNCS, vol. 2652, pp. 1126–1138. Springer, Heidelberg (2003)
Hansen, L., Salamon, P.: Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(10), 993–1001 (1990)
Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Volume (1995)
Tumer, K., Ghosh, J.: Linear and order statistics combiners for pattern classification. In: Sharkey, A. (ed.) Combining Artificial Neural Nets, pp. 127–162. Springer, Heidelberg (1999)
Fumera, G., Roli, F.: A theoretical and experimental analysis of linear combiners and multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(6), 942–956 (2005)
Aksela, M., Laaksonen, J.: Using diversity of errors for selecting members of a committee classifier. Pattern Recognition 39, 608–623 (2006)
Partridge, D., Krzanowski, W.: Refining multiple classifier system diversity. Technical report, Computer Science Department, University of Exeter, UK (2003)
Ruta, D.: Classifier Diversity in Combined Pattern Recognition Systems. PhD thesis, University of Paisley, Scotland, UK (2003)
Shipp, C., Kuncheva, L.: Relationships between combination methods and measures of diversity in combining classifiers. Information Fusion 3(2), 135–148 (2002)
Narasimhamurthy, A.: Evaluation of diversity measures for binary classifier ensembles. In: Oza, N.C., et al. (eds.) MCS 2005. LNCS, vol. 3541, pp. 267–277. Springer, Heidelberg (2005)
Hsu, D.F., Chung, Y.S., Kristal, B.S.: Combinatorial fusion analysis: Methods and practices of combining multiple scoring systems. In: Hsu, H.H. (ed.) Advanced Data Mining Technologies in Bioinformatics, pp. 32–62. Idea Group, Hershey (2006)
Hsu, D.F., Taksa, I.: Comparing rank and score combination methods for data fusion in information retrieval. Information Retrieval 8(3), 449–480 (2005)
Kuncheva, L.I., et al.: Limits on the majority vote accuracy in classifier fusion. Pattern Analysis and Applications 6, 22–31 (2003)
Ruta, D., Gabrys, B.: A theoretical analysis of the majority voting errors for multiple classifier systems. Pattern Analysis and Applications 5, 333–350 (2002)
Kittler, J., et al.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998)
Kittler, J., Alkoot, F.: Sum versus vote fusion in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(1), 110–115 (2003)
Lam, L., Suen, C.Y.: Application of majority voting to pattern recognition: An analysis of its behaviour and performance. IEEE Transactions on Systems, Man, and Cybernetics 27(5), 533–568 (1997)
Zenobi, G., Cunningham, P.: Using diversity in preparing ensembles of classifiers based on different feature subsets to minimize generalization error. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 576–587. Springer, Heidelberg (2001)
Narasimhamurthy, A.: Theoretical bounds of majority voting performance for a binary classification problem. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(12), 1988–1995 (2005)
Roli, F., Kittler, J.: Fusion of multiple classifiers. Information Fusion (Editorial) 3(4), 243 (2002)
Kuncheva, L.I.: Diversity in multiple classifier systems. Information Fusion (Editorial) 6(1), 3–4 (2005)
Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)
Matan, O.: On voting ensembles of classifiers. In: Proc. AAAI-96, Integrating Multiple Learned Models Workshop, pp. 84–88 (1996)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Chichester (1991)
Kuncheva, L.I.: Fuzzy Classifier Design. Springer, Heidelberg (2000)
Breiman, L.: Arcing classifiers. The Annals of Statistics 26(3), 801–849 (1998)
Kendall, M., Gibbons, J.D.: Rank Correlation Methods. Edward Arnold, London (1990)
Chung, Y.S., Hsu, D.F., Tang, C.Y.: On the diversity-performance relationship for plurality voting and disagreement diversity in classifier ensembles. Manuscript (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Chung, YS., Hsu, D.F., Tang, C.Y. (2007). On the Diversity-Performance Relationship for Majority Voting in Classifier Ensembles. In: Haindl, M., Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2007. Lecture Notes in Computer Science, vol 4472. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72523-7_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-72523-7_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72481-0
Online ISBN: 978-3-540-72523-7
eBook Packages: Computer ScienceComputer Science (R0)