article

Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension

Authors:

David Haussler,

Michael Kearns,

Robert E. SchapireAuthors Info & Claims

Machine Learning, Volume 14, Issue 1

Pages 83 - 113

https://doi.org/10.1023/A:1022698821832

Published: 01 January 1994 Publication History

Publisher Site

Abstract

In this paper we study a Bayesian or average-case model of concept learning with a twofold goal: to provide more precise characterizations of learning curve (sample complexity) behavior that depend on properties of both the prior distribution over concepts and the sequence of instances seen by the learner, and to smoothly unite in a common framework the popular statistical physics and VC dimension theories of learning curves. To achieve this, we undertake a systematic investigation and comparison of two fundamental quantities in learning and information theory: the probability of an incorrect prediction for an optimal learning algorithm, and the Shannon information gain. This study leads to a new understanding of the sample complexity of learning in several existing models.

Cited By

View all

Laviolette FMorvant ERalaivola LRoy J(2017)Risk upper bounds for general ensemble methods with an application to multiclass classificationNeurocomputing10.1016/j.neucom.2016.09.016219:C(15-25)Online publication date: 5-Jan-2017
https://dl.acm.org/doi/10.1016/j.neucom.2016.09.016
Chen COzolek JWang WRohde G(2011)A general system for automatic biomedical image segmentation using intensity neighborhoodsJournal of Biomedical Imaging10.5555/1992576.20433162011(1-12)Online publication date: 1-Jan-2011
https://dl.acm.org/doi/10.5555/1992576.2043316
Tamaddoni-Nezhad AMuggleton S(2010)Stochastic refinementProceedings of the 20th international conference on Inductive logic programming10.5555/2022735.2022763(222-237)Online publication date: 27-Jun-2010
https://dl.acm.org/doi/10.5555/2022735.2022763
Show More Cited By

Index Terms

Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Probabilistic reasoning
      2. Vagueness and fuzzy logic
  2. Machine learning
2. Theory of computation
  1. Models of computation
    1. Probabilistic computation

Recommendations

A Bayesian/Information Theoretic Model of Learning to Learn viaMultiple Task Sampling
Special issue on inductive transfer

A Bayesian model of learning to learn by sampling from multiple tasks is presented. The multiple tasks are themselves generated by sampling from a distribution over an environment of related tasks. Such an environment is shown to be naturally modelled ...
BOUNDS ON THE SAMPLE COMPLEXITY OF BAYESIAN LEARNING USINGINFORMATION THEORY AND THE VC DIMENSION
Stochastic complexities of general mixture models in variational Bayesian learning

In this paper, we focus on variational Bayesian learning of general mixture models. Variational Bayesian learning was proposed as an approximation of Bayesian learning. While it has provided computational tractability and good generalization in many ...

Comments

Information & Contributors

Information

Published In

Machine Language Volume 14, Issue 1

Special issue on computational learning theory

Jan. 1994

125 pages

ISSN:0885-6125

Editors:
Thomas Dietterich
Oregon State Univ., Corvallis
,
Ming Li,
Leslie G. Valiant
Harvard Univ., Cambridge, MA

Issue’s Table of Contents

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 January 1994

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Laviolette FMorvant ERalaivola LRoy J(2017)Risk upper bounds for general ensemble methods with an application to multiclass classificationNeurocomputing10.1016/j.neucom.2016.09.016219:C(15-25)Online publication date: 5-Jan-2017
https://dl.acm.org/doi/10.1016/j.neucom.2016.09.016
Chen COzolek JWang WRohde G(2011)A general system for automatic biomedical image segmentation using intensity neighborhoodsJournal of Biomedical Imaging10.5555/1992576.20433162011(1-12)Online publication date: 1-Jan-2011
https://dl.acm.org/doi/10.5555/1992576.2043316
Tamaddoni-Nezhad AMuggleton S(2010)Stochastic refinementProceedings of the 20th international conference on Inductive logic programming10.5555/2022735.2022763(222-237)Online publication date: 27-Jun-2010
https://dl.acm.org/doi/10.5555/2022735.2022763
Yang LHanneke SCarbonell J(2010)Bayesian active learning using arbitrary binary valued queriesProceedings of the 21st international conference on Algorithmic learning theory10.5555/1893193.1893204(50-58)Online publication date: 6-Oct-2010
https://dl.acm.org/doi/10.5555/1893193.1893204
Guyon ISaffari ADror GCawley G(2010)Model Selection: Beyond the Bayesian/Frequentist DivideThe Journal of Machine Learning Research10.5555/1756006.175600911(61-87)Online publication date: 1-Mar-2010
https://dl.acm.org/doi/10.5555/1756006.1756009
Muggleton STamaddoni-Nezhad A(2008)QG/GAMachine Language10.1007/s10994-007-5029-370:2-3(121-133)Online publication date: 1-Mar-2008
https://dl.acm.org/doi/10.1007/s10994-007-5029-3
Lappas G(2007)Estimating the size of neural networks from the number of available training dataProceedings of the 17th international conference on Artificial neural networks10.5555/1776814.1776823(68-77)Online publication date: 9-Sep-2007
https://dl.acm.org/doi/10.5555/1776814.1776823
Su HLi QDang J(2006)A hybrid bayesian optimal classifier based on neuro-fuzzy logicProceedings of the Second international conference on Advances in Natural Computation - Volume Part I10.1007/11881070_49(341-350)Online publication date: 24-Sep-2006
https://dl.acm.org/doi/10.1007/11881070_49
Perevozchikova OTul'Chinskii VKharchenko A(2003)Distinctive Features of Minimization of a Risk Functional in Mass Data SetsCybernetics and Systems Analysis10.1023/B:CASA.0000003500.91054.5b39:4(501-508)Online publication date: 1-Jul-2003
https://dl.acm.org/doi/10.1023/B%3ACASA.0000003500.91054.5b
Schmitt M(2002)On the complexity of computing and learning with multiplicative neural networksNeural Computation10.1162/0899766025274112114:2(241-301)Online publication date: 1-Feb-2002
https://dl.acm.org/doi/10.1162/08997660252741121
Show More Cited By

Abstract

Cited By

Index Terms

Recommendations

A Bayesian/Information Theoretic Model of Learning to Learn viaMultiple Task Sampling