Article

Using query-specific variance estimates to combine Bayesian classifiers

Authors:

Shaojun WangAuthors Info & Claims

ICML '06: Proceedings of the 23rd international conference on Machine learning

Pages 529 - 536

https://doi.org/10.1145/1143844.1143911

Published: 25 June 2006 Publication History

Abstract

Many of today's best classification results are obtained by combining the responses of a set of base classifiers to produce an answer for the query. This paper explores a novel "query specific" combination rule: After learning a set of simple belief network classifiers, we produce an answer to each query by combining their individual responses, using weights based inversely on their respective variances around their responses. These variances are based on the uncertainty of the network parameters, which in turn depend on the training datasample. In essence, this variance quantifies the base classifier's confidence of its response to this query. Our experimental results show that these "mixture-using-variance belief net classifiers" MUVS work effectively, especially when the base classifiers are learned using balanced bootstrap samples and when their results are combined using James-Stein shrinkage. We also found that our variance-based combination rule performed better than both bagging and AdaBoost, even on the set of base classifiers produced by AdaBoost itself. Finally, this framework is extremely efficient, as both the learning and the classification components require only straight-line code.

References

[1]

Bauer, E., & Kohavi, R. (1999). An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 36.

Digital Library

[2]

Breiman, L. (1996). Bagging predictors. Machine Learning, 24.

Digital Library

[3]

Cooper, G., & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9.

Digital Library

[4]

Duda, R., Hart, P., & Stork, D. (2002). Pattern classification, 2nd ed. Wiley.

Digital Library

[5]

Freund, Y., & Schapire, R. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55.

Digital Library

[6]

Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29.

Digital Library

[7]

Hastie, T., Tibshirani, R., & Friedman, J. (2002). The elements of statistical learning. Springer.

[8]

Heckerman, D. (1998). A tutorial on learning with Bayesian networks. Learning in Graphical Models.

Digital Library

[9]

Holte, R. (1993). Very simple classification rules perform well on most commonly used datasets. Machine Learning, 3.

Digital Library

[10]

Inza, I., Larranaga, P., Lozano, J., & Pena, J. (2005). Special issue of Machine Learning Journal: Probabilistic graphical models for classification, vol. 59.

[11]

Jacobs, R., Jordan, M., Nowlan, S., & Hinton, G. (1991). Adaptive mixtures of local experts. Neural Computation.

[12]

Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. NIPS (pp. 231--238).

[13]

Lehmann, E., & Casella, G. (2003). Theory of point estimation. Springer.

[14]

Mitchell, T. (1997). Machine learning. McGraw-Hill.

Digital Library

[15]

Opitz, D., & Maclin, R. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11.

[16]

Pearl, J. (1988). Probabilistic reasoning in intelligent systems. Morgan Kaufmann.

Digital Library

[17]

Perrone, M., & Cooper, L. (1993). When networks disagree: ensemble method for neural networks. Artificial Neural Networks for Speech and Vision.

[18]

Sollich, P., & Krogh, A. (1996). Learning with ensembles: How overfitting can be useful. NIPS.

[19]

Taniguchi, M., & Tresp, V. (1997). Averaging regularized estimators. Neural Computation, 9.

Digital Library

[20]

UCI (2006). http://www.ics.uci.edu/~mlearn/.

[21]

Van Allen, T., Greiner, R., & Hooper, P. (2001). Bayesian error-bars for belief net inference. UAI.

Digital Library

[22]

Web (A). http://www.cs.ualberta.ca/~greiner/MUV.

[23]

Web (B). http://www.cs.ualberta.ca/~greiner/BNvar.

Cited By

Xu LAmari S(2012)Combining Classifiers and Learning Mixture-of-ExpertsMachine Learning10.4018/978-1-60960-818-7.ch209(243-252)Online publication date: 2012
https://doi.org/10.4018/978-1-60960-818-7.ch209
Dikici EOrderud FTorp H(2012)Best linear unbiased estimator for Kalman filter based left ventricle tracking in 3D+T echocardiography2012 IEEE Workshop on Mathematical Methods in Biomedical Image Analysis10.1109/MMBIA.2012.6164741(201-208)Online publication date: Jan-2012
https://doi.org/10.1109/MMBIA.2012.6164741
Dikici EOrderud F(2011)Maximum likelihood and James-Stein edge estimators for left ventricle tracking in 3D echocardiographyProceedings of the Second international conference on Machine learning in medical imaging10.5555/2046063.2046069(43-50)Online publication date: 18-Sep-2011
https://dl.acm.org/doi/10.5555/2046063.2046069
Show More Cited By

Index Terms

Using query-specific variance estimates to combine Bayesian classifiers
1. Computing methodologies
  1. Machine learning
2. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Statistical graphics

Recommendations

Improving classification accuracy by using confidence measures to combine classifiers
Learning to combine discriminative classifiers: confidence based
KDD '10: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining

Much of research in data mining and machine learning has led to numerous practical applications. Spam filtering, fraud detection, and user query-intent analysis has relied heavily on machine learned classifiers, and resulted in improvements in robust ...
Combination of Multiple Classifiers Using Local Accuracy Estimates

This paper presents a method for combining classifiers that uses estimates of each individual classifier's local accuracy in small regions of feature space surrounding an unknown test sample. An empirical evaluation using five real data sets confirms ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '06: Proceedings of the 23rd international conference on Machine learning

June 2006

1154 pages

ISBN:1595933832

DOI:10.1145/1143844

Program Chairs:
William Cohen,
Andrew Moore

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

ICML '06 Paper Acceptance Rate 140 of 548 submissions, 26%;

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu LAmari S(2012)Combining Classifiers and Learning Mixture-of-ExpertsMachine Learning10.4018/978-1-60960-818-7.ch209(243-252)Online publication date: 2012
https://doi.org/10.4018/978-1-60960-818-7.ch209
Dikici EOrderud FTorp H(2012)Best linear unbiased estimator for Kalman filter based left ventricle tracking in 3D+T echocardiography2012 IEEE Workshop on Mathematical Methods in Biomedical Image Analysis10.1109/MMBIA.2012.6164741(201-208)Online publication date: Jan-2012
https://doi.org/10.1109/MMBIA.2012.6164741
Dikici EOrderud F(2011)Maximum likelihood and James-Stein edge estimators for left ventricle tracking in 3D echocardiographyProceedings of the Second international conference on Machine learning in medical imaging10.5555/2046063.2046069(43-50)Online publication date: 18-Sep-2011
https://dl.acm.org/doi/10.5555/2046063.2046069
Lee CRao BKrishnapuram BTomkins AYang Q(2010)Learning to combine discriminative classifiersProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining10.1145/1835804.1835899(743-752)Online publication date: 25-Jul-2010
https://dl.acm.org/doi/10.1145/1835804.1835899
Hooper PAbbasi-Yadkori YGreiner RHoehn BMcAllester D(2009)Improved mean and variance approximations for belief net responses via network doublingProceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence10.5555/1795114.1795142(232-239)Online publication date: 18-Jun-2009
https://dl.acm.org/doi/10.5555/1795114.1795142
Van Allen TSingh AGreiner RHooper P(2008)Quantifying the uncertainty of a belief net responseArtificial Intelligence10.1016/j.artint.2007.09.004172:4-5(483-513)Online publication date: 1-Mar-2008
https://dl.acm.org/doi/10.1016/j.artint.2007.09.004
Collins-Thompson KCallan JKraaij Wde Vries AClarke CFuhr NKando N(2007)Estimation and use of uncertainty in pseudo-relevance feedbackProceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval10.1145/1277741.1277795(303-310)Online publication date: 23-Jul-2007
https://dl.acm.org/doi/10.1145/1277741.1277795
Shoham P(2007)How Relevant is Game Theory to Intelligent Agent Technology?Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence10.1109/WI.2007.65Online publication date: 2-Nov-2007
https://dl.acm.org/doi/10.1109/WI.2007.65
Bennett P(2007)Neighborhood-Based Local SensitivityProceedings of the 18th European conference on Machine Learning10.1007/978-3-540-74958-5_7(30-41)Online publication date: 17-Sep-2007
https://dl.acm.org/doi/10.1007/978-3-540-74958-5_7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents