Article

PAC-Bayes risk bounds for sample-compressed Gibbs classifiers

Authors:

François Laviolette,

Mario MarchandAuthors Info & Claims

ICML '05: Proceedings of the 22nd international conference on Machine learning

Pages 481 - 488

https://doi.org/10.1145/1102351.1102412

Published: 07 August 2005 Publication History

Get Access

Abstract

We extend the PAC-Bayes theorem to the sample-compression setting where each classifier is represented by two independent sources of information: a compression set which consists of a small subset of the training data, and a message string of the additional information needed to obtain a classifier. The new bound is obtained by using a prior over a data-independent set of objects where each object gives a classifier only when the training data is provided. The new PAC-Bayes theorem states that a Gibbs classifier defined on a posterior over sample-compressed classifiers can have a smaller risk bound than any such (deterministic) sample-compressed classifier.

References

[1]

Cover, T. M., & Thomas, J. A. (1991). Elements of information theory, chapter 12. Wiley.]]

Digital Library

Google Scholar

[2]

Floyd, S., & Warmuth, M. (1995). Sample compression, learnability, and the Vapnik-Chervonenkis dimension. Machine Learning, 21, 269--304.]]

Digital Library

Google Scholar

[3]

Graepel, T., Herbrich, R., & Shawe-Taylor, J. (2000). Generalisation error bounds for sparse linear classifiers. Proceedings of the Thirteenth Annual Conference on Computational Learning Theory (pp. 298--303).]]

Digital Library

Google Scholar

[4]

Graepel, T., Herbrich, R., & Williamson, R. C. (2001). From margin to sparsity. Advances in neural information processing systems (pp. 210--216).]]

Google Scholar

[5]

Langford, J. (2005). Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 6, 273--306.]]

Digital Library

Google Scholar

[6]

Langford, J., & Shawe-Taylor, J. (2003). PAC-Bayes & margins. In S. T. S. Becker and K. Obermayer (Eds.), Advances in neural information processing systems 15, 423--430. Cambridge, MA: MIT Press.]]

Google Scholar

[7]

Littlestone, N., & Warmuth, M. (1986). Relating data compression and learnability (Technical Report). University of California Santa Cruz, Santa Cruz, CA.]]

Google Scholar

[8]

Marchand, M., & Shawe-Taylor, J. (2002). The set covering machine. Journal of Machine Learning Research, 3, 723--746.]]

Digital Library

Google Scholar

[9]

McAllester, D. (1999). Some PAC-Bayesian theorems. Machine Learning, 37, 355--363.]]

Digital Library

Google Scholar

[10]

McAllester, D. (2003a). PAC-Bayesian stochastic model selection. Machine Learning, 51, 5--21. A priliminary version appeared in proceedings of COLT'99.]]

Digital Library

Google Scholar

[11]

McAllester, D. (2003b). Simplified PAC-Bayesian margin bounds. Proceedings of the 16th Annual Conference on Learning Theory, Lecture Notes in Artificial Intelligence, 2777, 203--215.]]

Google Scholar

[12]

Seeger, M. (2002). PAC-Bayesian generalization bounds for gaussian processes. Journal of Machine Learning Research, 3, 233--269.]]

Digital Library

Google Scholar

Cited By

View all

Oneto LOneto L(2019)PAC-Bayes TheoryModel Selection and Error Estimation in a Nutshell10.1007/978-3-030-24359-3_8(75-86)Online publication date: 18-Jul-2019
https://doi.org/10.1007/978-3-030-24359-3_8
Oneto LOneto L(2019)The “Five W” of MS and EEModel Selection and Error Estimation in a Nutshell10.1007/978-3-030-24359-3_2(5-11)Online publication date: 18-Jul-2019
https://doi.org/10.1007/978-3-030-24359-3_2
Oneto L(2018)Model selection and error estimation without the agonizing painWIREs Data Mining and Knowledge Discovery10.1002/widm.12528:4Online publication date: 22-Mar-2018
https://doi.org/10.1002/widm.1252
Show More Cited By

PAC-Bayes risk bounds for sample-compressed Gibbs classifiers
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

PAC-Bayes Risk Bounds for Stochastic Averages and Majority Votes of Sample-Compressed Classifiers

We propose a PAC-Bayes theorem for the sample-compression setting where each classifier is described by a compression subset of the training data and a message string of additional information. This setting, which is the appropriate one to describe many ...
Tighter PAC-Bayes bounds
NIPS'06: Proceedings of the 19th International Conference on Neural Information Processing Systems

This paper proposes a PAC-Bayes bound to measure the performance of Support Vector Machine (SVM) classifiers. The bound is based on learning a prior over the distribution of classifiers with a part of the training samples. Experimental work shows that ...
PAC-bayes bounds with data dependent priors

This paper presents the prior PAC-Bayes bound and explores its capabilities as a tool to provide tight predictions of SVMs' generalization. The computation of the bound involves estimating a prior of the distribution of classifiers from the available ...

Comments

Information & Contributors

Information

Published In

ICML '05: Proceedings of the 22nd international conference on Machine learning

August 2005

1113 pages

ISBN:1595931805

DOI:10.1145/1102351

General Chair:
Saso Dzeroski
Jozef Stefan Institute, Slovenia
,
Program Chairs:
Luc De Raedt,
Stefan Wrobel

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
184
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Oneto LOneto L(2019)PAC-Bayes TheoryModel Selection and Error Estimation in a Nutshell10.1007/978-3-030-24359-3_8(75-86)Online publication date: 18-Jul-2019
https://doi.org/10.1007/978-3-030-24359-3_8
Oneto LOneto L(2019)The “Five W” of MS and EEModel Selection and Error Estimation in a Nutshell10.1007/978-3-030-24359-3_2(5-11)Online publication date: 18-Jul-2019
https://doi.org/10.1007/978-3-030-24359-3_2
Oneto L(2018)Model selection and error estimation without the agonizing painWIREs Data Mining and Knowledge Discovery10.1002/widm.12528:4Online publication date: 22-Mar-2018
https://doi.org/10.1002/widm.1252
Germain PLacasse ALaviolette FMarchand MRoy J(2015)Risk bounds for the majority voteThe Journal of Machine Learning Research10.5555/2789272.283114016:1(787-860)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.5555/2789272.2831140

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

PAC-Bayes Risk Bounds for Stochastic Averages and Majority Votes of Sample-Compressed Classifiers

Tighter PAC-Bayes bounds

PAC-bayes bounds with data dependent priors

Comments

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

PAC-Bayes Risk Bounds for Stochastic Averages and Majority Votes of Sample-Compressed Classifiers

Tighter PAC-Bayes bounds

PAC-bayes bounds with data dependent priors

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations