Article

Free access

Improved lower bounds for learning from noisy examples: an information-theoretic approach

Authors:

Claudio Gentile,

David P. HelmboldAuthors Info & Claims

COLT' 98: Proceedings of the eleventh annual conference on Computational learning theory

Pages 104 - 115

https://doi.org/10.1145/279943.279965

Published: 24 July 1998 Publication History

PDF eReader

References

[1]

D. Angluin, "Learning regular sets from queries and counterexamples", Information and Computation, vol. 75, n. 2, pp. 87-106, 1987.

Digital Library

Google Scholar

[2]

D. Angluin, "Queries and concept learning", Machine Learning, vol. 2, n. 4, pp 319-342, 1988.

Crossref

Google Scholar

[3]

D. Angluin, M. Krikis, R. Sloan, G. Turfin, "Malicious Omissions and Errors in Answers to Membership Queries", Machine Learning, vol. 28, no. 2/3, pp. 211- 255, 1997.

Digital Library

Google Scholar

[4]

D. Angluin, P.D. Laird, "Learning from Noisy Examples'', Machine Learning, vol. 2, no. 2, pp. 343-370, 1988.

Digital Library

Google Scholar

[5]

B. Apolloni, C. Gentile, "Sample Size Lower Bounds in PAC Learning by Algorithmic Complexity Theory", Theor. Comp. Sc., to appear.

Digital Library

Google Scholar

[6]

J.A. Aslam, S.E. Decatur, "On the sample complexity of noise-tolerant learning", Inf. Proc. Lett., vol. 57, pp. 189-195, 1996.

Digital Library

Google Scholar

[7]

P. Bartlett, "Learning with a slowly changing distribution'', in Proc. of the 5th Workshop on Comput. Learn. Th., 1992, pp. 243-252.

Digital Library

Google Scholar

[8]

P. Bartlett, D. Helmbold, manuscript, 1996.

Google Scholar

[9]

R.D. Barve, P.M. Long, "On the complexity of learning from drifting distributions", Information and Computation, vol. 138, no. 2, pp. 170-193, 1997.

Digital Library

Google Scholar

[10]

G. Benedek, A. Itai, "Leamability by Fixed Distributions'', Theor. Comp. Sc., vol. 86, no. 2, pp. 377-389, 1991.

Digital Library

Google Scholar

[11]

A. Blumer, A. Ehrenfeucht, D. Haussler, M. Warmuth, "Leamability and the Vapnik-Chervonenkis Dimension'', J. ofACM, vol. 36, pp. 929-965, 1989.

Digital Library

Google Scholar

[12]

N. Cesa-Bianchi, E. Dichterman, P. Fischer, E. Shamir, H.U. Simon, "Sample-efficient Strategies for Learning in the Presence of Noise", eCOLT Tech. Rep. 97- 003, WWW.' http://ecolt.informatik, uni-dortmund, de/. Preliminary versions in 28th STOC, 1996 and 3rd EuroCOLT, 1997

Google Scholar

[13]

T.M. Cover, J.A. Thomas, Elements of information theory. NY: John Wiley & Sons, Inc., 1991.

Digital Library

Google Scholar

[14]

R.L. Dobrushin, "General formulation of Shannon's main theorem in information theory", Uspelchi Mat. Nauk, vol. 14, pp. 3-104, 1959; translated in Amer. Math. Soc. Translations, Ser. 2, vol. 33, pp. 323-438, 1963.

Google Scholar

[15]

R.M. Dudley, A Course on Empirical Processes. Lecture Notes in Mathematics, vol. 1097, Spfinger-Vefiag, Berlin/New York, 1984.

Google Scholar

[16]

A. Ehrenfeucht, D. Haussler, M. Kearns, L. Valiant, "A General Lower Bound on the Number of Examples Needed for Learning", Information and Computation, vol. 82, no. 3, pp. 247-261, 1989.

Digital Library

Google Scholar

[17]

B. Eisenberg, R.L. Rivest, "On the Sample Complexity of Pac-Leaming using Random and Chosen Examples'', in Proc. of the 3th Workshop on Comput. Learn. Th., 1990, pp. 154~162.

Digital Library

Google Scholar

[18]

C. Gentile, "A note on sample size lower bounds for PAC-leaming", manuscript, 1997.

Google Scholar

[19]

D. Haussler, A. Barron, "How well do Bayes methods work for on-line prediction of {-1,+ 1 } values'?.", in Proc. of the 3rd NEC Symposium on Computation and Cognition, 1992, pp. 74-100.

Google Scholar

[20]

D. Haussler, M. Keams, R. Schapire, "Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension", Machine Learning, vol. 14, pp. 84-114, 1994.

Digital Library

Google Scholar

[21]

D. Haussler, N. Littlestone, and M. K. Warmuth. Predicting {0, 1 } functions on randomly drawn points. Information and Computation, 115(2):284-293, 1994.

Digital Library

Google Scholar

[22]

D. Haussler, M. Opper, "Mutual Information, Metric Entropy, and Cumulative Relative Entropy Risk", Annals of Statistics, 1997. To appear.

Google Scholar

[23]

D. Helmbold, N. Littlestone, P. Long, "Apple Tasting'', manuscript 1997. An extended abstract appeared in Proc. of the 33rd Symposium on the Foundations of Comp. Sci., 1992, pp.493-502.

Google Scholar

[24]

S. Ihara, Information theory for continuous systems. River Edge, NJ: World Scientific, 1993.

Google Scholar

[25]

M. Keams, M. Li, "Learning in the presence of malicious errors", SIAM J. Comput, vol. 22, pp. 807-837, 1993.

Digital Library

Google Scholar

[26]

A.N. Kolmogorov, V.M. Tihomirov, "e-entropy and ecapacity of sets in functional spaces", Amer. Math. Soc. Translations (Ser. 2), vol. 17, pp. 277-364, 1961.

Google Scholar

[27]

P. Laird, Learning from Good and Bad Data. Kluwer International Series in Engineering and Computer Science, Kluwer Academic Publishers, Boston, MA, 1988.

Digital Library

Google Scholar

[28]

W. Maass, G. Turfin, "On the complexity of learning from counterexamples and membership queries", in Proc. of the 31th Symposium on the Foundations of Comp. Sci., 1990, pp. 203-210.

Digital Library

Google Scholar

[29]

K. Sakakibara, "On learning from queries and counterexamples in the presence of noise", Inf, Proc. Lett., vol. 37, no. 5, pp. 279-284, 1991.

Digital Library

Google Scholar

[30]

H.U. Simon, "General Bounds on the Number of Examples Needed for Learning Probabilistic Concepts", Journal of Comp. System Sci., vol. 52, no. 2, pp. 239- 254, 1996.

Digital Library

Google Scholar

[31]

R. Sloan, "Four types of noise in data for PAC learning'', Inf. Proc. Lett., vol. 54, pp. 157-162, 1995.

Digital Library

Google Scholar

[32]

G. Shackelford, D. Volper, "Leaming k-DNF with noise in the attributes", in Proc. of the 1988 Workshop on Comput. Learn. Th., 1988, pp. 97-103.

Digital Library

Google Scholar

[33]

G. Turhn, "Lower bounds for PAC learning with queries", in Proc. of the 6th Workshop on Comput. Learn. Th., 1993, pp. 384-391.

Digital Library

Google Scholar

[34]

L. Valiant, "A theory of the learnable", Communication ofACM, vol. 27, no. ll,pp. 1134-1142, 1984.

Digital Library

Google Scholar

[35]

V.N. Vapnik, Estimation of dependencies based on empirical data. NY: Springer Verlag, 1982.

Digital Library

Google Scholar

[36]

V.N. Vapnik, The Nature of Statistical Learning Theory. NY: Springer Verlag, 1995.

Digital Library

Google Scholar

[37]

B. Yu, "Lower Bounds on Expected Redundancy for Nonparametric Classes", IEEE Trans. on Inf. Th., vol. 42, no. 1, pp. 272-275, 1996.

Digital Library

Google Scholar

Cited By

View all

Wang RLiu TTao D(2018)Multiclass Learning With Partially Corrupted LabelsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2017.269978329:6(2568-2580)Online publication date: Jul-2018
https://doi.org/10.1109/TNNLS.2017.2699783
Frenay BVerleysen M(2014)Classification in the Presence of Label Noise: A SurveyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2013.229289425:5(845-869)Online publication date: May-2014
https://doi.org/10.1109/TNNLS.2013.2292894
Sabato SSrebro NTishby N(2010)Tight sample complexity of Large-margin learningProceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 210.5555/2997046.2997123(2038-2046)Online publication date: 6-Dec-2010
https://dl.acm.org/doi/10.5555/2997046.2997123
Show More Cited By

Index Terms

Improved lower bounds for learning from noisy examples: an information-theoretic approach
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees
  2. Modeling and simulation
2. Mathematics of computing
  1. Information theory

Recommendations

Improved Lower Bounds for Learning from Noisy Examples

This paper presents a general information-theoretic approach for obtaining lower bounds on the number of examples required for Probably Approximately Correct (PAC) learning in the presence of noise. This approach deals directly with the fundamental ...
Improved upper bounds on sizes of codes

Let A(n,d) denote the maximum possible number of codewords in a binary code of length n and minimum Hamming distance d. For large values of n, the best known upper bound, for fixed d, is the Johnson bound. We give a new upper bound which is at least as ...
Average-Case Lower Bounds for Noisy Boolean Decision Trees

We present a new method for deriving lower bounds to the expected number of queries made by noisy decision trees computing Boolean functions. The new method has the feature that expectations are taken with respect to a uniformly distributed random input, ...

Comments

Information & Contributors

Information

Published In

COLT' 98: Proceedings of the eleventh annual conference on Computational learning theory

July 1998

304 pages

ISBN:1581130570

DOI:10.1145/279943

Chairmen:
Peter Bartlett
Australian National Univ., Australia
,
Yishay Mansour
Tel-Aviv Univ., Israel

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 1998

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

COLT98

Sponsor:

University of Wisconsin
SIGAI
SIGACT
UC @ Santa Cruz

COLT98: The 11th Annual Conference on Computational Learning Theroy

July 24 - 26, 1998

Wisconsin, Madison, USA

Acceptance Rates

Overall Acceptance Rate 35 of 71 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
422
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)11

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wang RLiu TTao D(2018)Multiclass Learning With Partially Corrupted LabelsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2017.269978329:6(2568-2580)Online publication date: Jul-2018
https://doi.org/10.1109/TNNLS.2017.2699783
Frenay BVerleysen M(2014)Classification in the Presence of Label Noise: A SurveyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2013.229289425:5(845-869)Online publication date: May-2014
https://doi.org/10.1109/TNNLS.2013.2292894
Sabato SSrebro NTishby N(2010)Tight sample complexity of Large-margin learningProceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 210.5555/2997046.2997123(2038-2046)Online publication date: 6-Dec-2010
https://dl.acm.org/doi/10.5555/2997046.2997123
Kääriäinen M(2006)Active learning in the non-realizable caseProceedings of the 17th international conference on Algorithmic Learning Theory10.1007/11894841_9(63-77)Online publication date: 7-Oct-2006
https://dl.acm.org/doi/10.1007/11894841_9

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations