Efficient agnostic PAC-learning with simple hypothesis

Reviewer: Sally Goldman

Most work in computational learning theory assumes the learner knows a set of classification rules (or hypotheses), one of which correctly classifies most examples of interest. While no such knowledge is available for most real-life problems, when designing a learning algorithm for such a problem, there is a set of classification rules from which the algorithm will select its final hypothesis. The agnostic probably approximately correct (PAC) learning model, defined by Haussler, does not require the learner to find a hypothesis having low error with respect to the correct classification; instead, it requires the learner to output a hypothesis h from the set H of classification rules whose error is near that of a best rule from H . This paper gives efficient algorithms for agnostic PAC learning when the hypothesis class H is the set of rectangles in the plane, the set of unions of two rectangles in the plane, or the set of k intervals over the real line. Thus for any classification problem, if you restrict your attention to rules from one such hypothesis class H , the algorithm will find a hypothesis from H whose error rate is nearly as good as the best hypothesis from H . While the hypothesis classes studied are fairly simple, this paper discusses empirical studies showing that simple prediction rules work well for many real-world classification problems. This paper nicely defines the formal models, and thus should be appropriate for people from the learning theory and machine learning communities who have some familiarity with the PAC model.