Nils J. Nilsson - Introduction To Machine Learning

INTRODUCTION
TO
MACHINE LEARNING
AN EARLY DRAFT OF A PROPOSED
TEXTBOOK
Nils J. Nilsson
Robotics Laboratory
Department of Computer Science
Stanford University
Stanford, CA 94305
e-mail: nilsson@cs.stanford.edu
September 26, 1996
Copyright c 1996 Nils J. Nilsson

This material may not be copied, reproduced, or distributed without the
written permission of the copyright holder. It is being made available on
the world-wide web in draft form to students, faculty, and researchers
solely for the purpose of preliminary evaluation.
Contents
1 Preliminaries 1
1.1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : 1
1.1.1 What is Machine Learning? : : : : : : : : : : : : : : 1
1.1.2 Wellsprings of Machine Learning : : : : : : : : : : : 3
1.1.3 Varieties of Machine Learning : : : : : : : : : : : : : 5
1.2 Learning Input-Output Functions : : : : : : : : : : : : : : : 6
1.2.1 Types of Learning : : : : : : : : : : : : : : : : : : : 6
1.2.2 Input Vectors : : : : : : : : : : : : : : : : : : : : : : 8
1.2.3 Outputs : : : : : : : : : : : : : : : : : : : : : : : : : 9
1.2.4 Training Regimes : : : : : : : : : : : : : : : : : : : : 9
1.2.5 Noise : : : : : : : : : : : : : : : : : : : : : : : : : : 10
1.2.6 Performance Evaluation : : : : : : : : : : : : : : : : 10
1.3 Learning Requires Bias : : : : : : : : : : : : : : : : : : : : : 10
1.4 Sample Applications : : : : : : : : : : : : : : : : : : : : : : 13
1.5 Sources : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 14
1.6 Bibliographical and Historical Remarks : : : : : : : : : : : 15
2 Boolean Functions 17
2.1 Representation : : : : : : : : : : : : : : : : : : : : : : : : : 17
2.1.1 Boolean Algebra : : : : : : : : : : : : : : : : : : : : 17
2.1.2 Diagrammatic Representations : : : : : : : : : : : : 18
2.2 Classes of Boolean Functions : : : : : : : : : : : : : : : : : 19
2.2.1 Terms and Clauses : : : : : : : : : : : : : : : : : : : 19
2.2.2 DNF Functions : : : : : : : : : : : : : : : : : : : : : 20
i
2.2.3 CNF Functions : : : : : : : : : : : : : : : : : : : : : 24
2.2.4 Decision Lists : : : : : : : : : : : : : : : : : : : : : : 25
2.2.5 Symmetric and Voting Functions : : : : : : : : : : : 26
2.2.6 Linearly Separable Functions : : : : : : : : : : : : : 26
2.3 Summary : : : : : : : : : : : : : : : : : : : : : : : : : : : : 27
3 Using Version Spaces for Learning 29
3.1 Version Spaces and Mistake Bounds : : : : : : : : : : : : : 29
3.2 Version Graphs : : : : : : : : : : : : : : : : : : : : : : : : : 31
3.3 Learning as Search of a Version Space : : : : : : : : : : : : 34
3.4 The Candidate Elimination Method : : : : : : : : : : : : : 35
4 Neural Networks 39
4.1 Threshold Logic Units : : : : : : : : : : : : : : : : : : : : : 39
4.1.1 De nitions and Geometry : : : : : : : : : : : : : : : 39
4.1.2 Special Cases of Linearly Separable Functions : : : : 41
4.1.3 Error-Correction Training of a TLU : : : : : : : : : 42
4.1.4 Weight Space : : : : : : : : : : : : : : : : : : : : : : 45
4.1.5 The Widrow-Ho Procedure : : : : : : : : : : : : : : 46
4.1.6 Training a TLU on Non-Linearly-Separable Training
Sets : : : : : : : : : : : : : : : : : : : : : : : : : : : 49
4.2 Linear Machines : : : : : : : : : : : : : : : : : : : : : : : : 50
4.3 Networks of TLUs : : : : : : : : : : : : : : : : : : : : : : : 51
4.3.1 Motivation and Examples : : : : : : : : : : : : : : : 51
4.3.2 Madalines : : : : : : : : : : : : : : : : : : : : : : : : 54
4.3.3 Piecewise Linear Machines : : : : : : : : : : : : : : : 56
4.3.4 Cascade Networks : : : : : : : : : : : : : : : : : : : 57
4.4 Training Feedforward Networks by Backpropagation : : : : 58
4.4.1 Notation : : : : : : : : : : : : : : : : : : : : : : : : : 58
4.4.2 The Backpropagation Method : : : : : : : : : : : : : 60
4.4.3 Computing Weight Changes in the Final Layer : : : 62
4.4.4 Computing Changes to the Weights in Intermediate
Layers : : : : : : : : : : : : : : : : : : : : : : : : : : 64
ii
4.4.5 Variations on Backprop : : : : : : : : : : : : : : : : 66
4.4.6 An Application: Steering a Van : : : : : : : : : : : : 66
4.5 Synergies Between Neural Network and Knowledge-Based
Methods : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 68
5 Statistical Learning 69
5.1 Using Statistical Decision Theory : : : : : : : : : : : : : : : 69
5.1.1 Background and General Method : : : : : : : : : : : 69
5.1.2 Gaussian (or Normal) Distributions : : : : : : : : : 71
5.1.3 Conditionally Independent Binary Components : : : 75
5.2 Learning Belief Networks : : : : : : : : : : : : : : : : : : : 77
5.3 Nearest-Neighbor Methods : : : : : : : : : : : : : : : : : : : 77
6 Decision Trees 81
6.1 De nitions : : : : : : : : : : : : : : : : : : : : : : : : : : : : 81
6.2 Supervised Learning of Univariate Decision Trees : : : : : : 83
6.2.1 Selecting the Type of Test : : : : : : : : : : : : : : : 83
6.2.2 Using Uncertainty Reduction to Select Tests : : : : 84
6.2.3 Non-Binary Attributes : : : : : : : : : : : : : : : : : 88
6.3 Networks Equivalent to Decision Trees : : : : : : : : : : : : 88
6.4 Over tting and Evaluation : : : : : : : : : : : : : : : : : : 89
6.4.1 Over tting : : : : : : : : : : : : : : : : : : : : : : : 89
6.4.2 Validation Methods : : : : : : : : : : : : : : : : : : 90
6.4.3 Avoiding Over tting in Decision Trees : : : : : : : : 91
6.4.4 Minimum-Description Length Methods : : : : : : : : 92
6.4.5 Noise in Data : : : : : : : : : : : : : : : : : : : : : : 93
6.5 The Problem of Replicated Subtrees : : : : : : : : : : : : : 94
6.6 The Problem of Missing Attributes : : : : : : : : : : : : : : 96
6.7 Comparisons : : : : : : : : : : : : : : : : : : : : : : : : : : 96
iii
7 Inductive Logic Programming 97
7.1 Notation and De nitions : : : : : : : : : : : : : : : : : : : : 99
7.2 A Generic ILP Algorithm : : : : : : : : : : : : : : : : : : : 100
7.3 An Example : : : : : : : : : : : : : : : : : : : : : : : : : : : 103
7.4 Inducing Recursive Programs : : : : : : : : : : : : : : : : : 107
7.5 Choosing Literals to Add : : : : : : : : : : : : : : : : : : : 110
7.6 Relationships Between ILP and Decision Tree Induction : : 111
8 Computational Learning Theory 117
8.1 Notation and Assumptions for PAC Learning Theory 117 : : : :
8.2 PAC Learning 119

: : : : : : : : : : : : : : : : : : : : : : : : : :
8.2.1 The Fundamental Theorem 119

: : : : : : : : : : : : : :
8.2.2 Examples 121

: : : : : : : : : : : : : : : : : : : : : : : :
8.2.3 Some Properly PAC-Learnable Classes 122

: : : : : : : :
8.3 The Vapnik-Chervonenkis Dimension 124

: : : : : : : : : : : : :
8.3.1 Linear Dichotomies 124

: : : : : : : : : : : : : : : : : : :
8.3.2 Capacity 126

: : : : : : : : : : : : : : : : : : : : : : : :
8.3.3 A More General Capacity Result 127

: : : : : : : : : : :
8.3.4 Some Facts and Speculations About the VC Dimension129

8.4 VC Dimension and PAC Learning 129
: : : : : : : : : : : : : :
8.5 Bibliographical and Historical Remarks 130

: : : : : : : : : : :
9 Unsupervised Learning 131

9.1 What is Unsupervised Learning? : : : : : : : : : : : : : : : 131
9.2 Clustering Methods : : : : : : : : : : : : : : : : : : : : : : : 133
9.2.1 A Method Based on Euclidean Distance : : : : : : : 133
9.2.2 A Method Based on Probabilities : : : : : : : : : : : 136
9.3 Hierarchical Clustering Methods : : : : : : : : : : : : : : : 138
9.3.1 A Method Based on Euclidean Distance : : : : : : : 138
9.3.2 A Method Based on Probabilities : : : : : : : : : : : 138
iv
10 Temporal-Dierence Learning 145
10.1 Temporal Patterns and Prediction Problems : : : : : : : : : 145
10.2 Supervised and Temporal-Dierence Methods : : : : : : : : 146
10.3 Incremental Computation of the (W) i : : : : : : : : : : : 148
10.4 An Experiment with TD Methods : : : : : : : : : : : : : : 150
10.5 Theoretical Results: : : : : : : : : : : : : : : : : : : : : : : 152
10.6 Intra-Sequence Weight Updating : : : : : : : : : : : : : : : 153
10.7 An Example Application: TD-gammon : : : : : : : : : : : : 155
11 Delayed-Reinforcement Learning 159
11.1 The General Problem : : : : : : : : : : : : : : : : : : : : : 159
11.2 An Example : : : : : : : : : : : : : : : : : : : : : : : : : : : 160
11.3 Temporal Discounting and Optimal Policies : : : : : : : : : 161
11.4 -Learning
Q : : : : : : : : : : : : : : : : : : : : : : : : : : : 164
11.5 Discussion, Limitations, and Extensions of Q-Learning : : : 167
11.5.1 An Illustrative Example : : : : : : : : : : : : : : : : 167
11.5.2 Using Random Actions : : : : : : : : : : : : : : : : 169
11.5.3 Generalizing Over Inputs : : : : : : : : : : : : : : : 170
11.5.4 Partially Observable States : : : : : : : : : : : : : : 171
11.5.5 Scaling Problems : : : : : : : : : : : : : : : : : : : : 172
12 Explanation-Based Learning 175
12.1 Deductive Learning : : : : : : : : : : : : : : : : : : : : : : : 175
12.2 Domain Theories : : : : : : : : : : : : : : : : : : : : : : : : 176
12.3 An Example : : : : : : : : : : : : : : : : : : : : : : : : : : : 178
12.4 Evaluable Predicates : : : : : : : : : : : : : : : : : : : : : : 182
12.5 More General Proofs : : : : : : : : : : : : : : : : : : : : : : 183
12.6 Utility of EBL : : : : : : : : : : : : : : : : : : : : : : : : : 183
12.7 Applications : : : : : : : : : : : : : : : : : : : : : : : : : : : 183
12.7.1 Macro-Operators in Planning : : : : : : : : : : : : : 184
12.7.2 Learning Search Control Knowledge : : : : : : : : : 186
v
vi
Preface
These notes are in the process of becoming a textbook. The process is quite
un nished, and the author solicits corrections, criticisms, and suggestions
from students and other readers. Although I have tried to eliminate errors,
some undoubtedly remain|caveat lector. Many typographical infelicities
will no doubt persist until the nal version. More material has yet to
be added. Please let me have your suggestions about topics that are too Some of my
important to be left out. I hope that future versions will cover Hop eld plans for
additions and
nets, Elman nets and other recurrent nets, radial basis functions, grammar other
and automata learning, genetic algorithms, and Bayes networks . I am reminders are
:::
mentioned in
also collecting exercises and project suggestions which will appear in future marginal notes.
versions. Yes, the nal version will have a good index.
My intention is to pursue a middle ground between a theoretical text-
book and one that focusses on applications. The book concentrates on the
important ideas in machine learning. I do not give proofs of many of the
theorems that I state, but I do give plausibility arguments and citations to
formal proofs. And, I do not treat many matters that would be of practical
importance in applications the book is not a handbook of machine learn-
ing practice. Instead, my goal is to give the reader sucient preparation
to make the extensive literature on machine learning accessible.
Students in my Stanford courses on machine learning have already made
several useful suggestions, as have my colleague, Pat Langley, and my teach-
ing assistants, Ron Kohavi, Karl Peger, Robert Allen, and Lise Getoor.
vii
Chapter 1
Preliminaries
1.1 Introduction
1.1.1 What is Machine Learning?
Learning, like intelligence, covers such a broad range of processes that it is
dicult to dene precisely. A dictionary denition includes phrases such as
\to gain knowledge, or understanding of, or skill in, by study, instruction,
or experience," and \modication of a behavioral tendency by experience."
Zoologists and psychologists study learning in animals and humans. In
this book we focus on learning in machines. There are several parallels
between animal and machine learning. Certainly, many techniques in ma-
chine learning derive from the eorts of psychologists to make more precise
their theories of animal and human learning through computational mod-
els. It seems likely also that the concepts and techniques being explored by
researchers in machine learning may illuminate certain aspects of biological
learning.
As regards machines, we might say, very broadly, that a machine learns
whenever it changes its structure, program, or data (based on its inputs
or in response to external information) in such a manner that its expected
future performance improves. Some of these changes, such as the addition
of a record to a data base, fall comfortably within the province of other dis-
ciplines and are not necessarily better understood for being called learning.
But, for example, when the performance of a speech-recognition machine
improves after hearing several samples of a person's speech, we feel quite
justied in that case to say that the machine has learned.
1
2 CHAPTER 1. PRELIMINARIES
Machine learning usually refers to the changes in systems that perform
tasks associated with arti cial intelligence (AI). Such tasks involve recog-
nition, diagnosis, planning, robot control, prediction, etc. The \changes"
might be either enhancements to already performing systems or ab initio
synthesis of new systems. To be slightly more specic, we show the archi-
tecture of a typical AI \agent" in Fig. 1.1. This agent perceives and models
its environment and computes appropriate actions, perhaps by anticipating
their eects. Changes made to any of the components shown in the gure
might count as learning. Dierent learning mechanisms might be employed
depending on which subsystem is being changed. We will study several
dierent learning methods in this book.
Sensory signals Goals
Perception
Model
Planning and
Reasoning
Action
Computation
Actions
Figure 1.1: An AI System

One might ask \Why should machines have to learn? Why not design
machines to perform as desired in the rst place?" There are several reasons
why machine learning is important. Of course, we have already mentioned
that the achievement of learning in machines might help us understand how
animals and humans learn. But there are important engineering reasons as
well. Some of these are:
Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved.
1.1. INTRODUCTION 3
Some tasks cannot be dened well except by example that is, we
might be able to specify input/output pairs but not a concise rela-
tionship between inputs and desired outputs. We would like machines
to be able to adjust their internal structure to produce correct out-
puts for a large number of sample inputs and thus suitably constrain
their input/output function to approximate the relationship implicit
in the examples.
It is possible that hidden among large piles of data are important
relationships and correlations. Machine learning methods can often
be used to extract these relationships (data mining).
Human designers often produce machines that do not work as well as
desired in the environments in which they are used. In fact, certain
characteristics of the working environment might not be completely
known at design time. Machine learning methods can be used for
on-the-job improvement of existing machine designs.
The amount of knowledge available about certain tasks might be
too large for explicit encoding by humans. Machines that learn this
knowledge gradually might be able to capture more of it than humans
would want to write down.
Environments change over time. Machines that can adapt to a chang-
ing environment would reduce the need for constant redesign.
New knowledge about tasks is constantly being discovered by humans.
Vocabulary changes. There is a constant stream of new events in
the world. Continuing redesign of AI systems to conform to new
knowledge is impractical, but machine learning methods might be
able to track much of it.
1.1.2 Wellsprings of Machine Learning

Work in machine learning is now converging from several sources. These
dierent traditions each bring dierent methods and dierent vocabulary
which are now being assimilated into a more unied discipline. Here is a
brief listing of some of the separate disciplines that have contributed to
machine learning more details will follow in the the appropriate chapters:
Statistics: A long-standing problem in statistics is how best to use
samples drawn from unknown probability distributions to help decide
from which distribution some new sample is drawn. A related problem
is how to estimate the value of an unknown function at a new point
given the values of this function at a set of sample points. Statistical
methods for dealing with these problems can be considered instances
of machine learning because the decision and estimation rules depend
on a corpus of samples drawn from the problem environment. We
will explore some of the statistical methods later in the book. Details
about the statistical theory underlying these methods can be found
in statistical textbooks such as Anderson, 1958].
Brain Models: Non-linear elements with weighted inputs
have been suggested as simple models of biological neu-
rons. Networks of these elements have been studied by sev-
eral researchers including McCulloch & Pitts, 1943, Hebb, 1949,
Rosenblatt, 1958] and, more recently by Gluck & Rumelhart, 1989,
Sejnowski, Koch, & Churchland, 1988]. Brain modelers are inter-
ested in how closely these networks approximate the learning phe-
nomena of living brains. We shall see that several important machine
learning techniques are based on networks of nonlinear elements|
often called neural networks. Work inspired by this school is some-
times called connectionism, brain-style computation, or sub-symbolic
processing.
Adaptive Control Theory: Control theorists study the problem
of controlling a process having unknown parameters which must
be estimated during operation. Often, the parameters change dur-
ing operation, and the control process must track these changes.
Some aspects of controlling a robot based on sensory inputs rep-
resent instances of this sort of problem. For an introduction see
Bollinger & Due, 1988].
Psychological Models: Psychologists have studied the performance
of humans in various learning tasks. An early example is the EPAM
network for storing and retrieving one member of a pair of words when
given another Feigenbaum, 1961]. Related work led to a number of
early decision tree Hunt, Marin, & Stone, 1966] and semantic net-
work Anderson & Bower, 1973] methods. More recent work of this
sort has been inuenced by activities in articial intelligence which
we will be presenting.
Some of the work in reinforcement learning can be traced to eorts
to model how reward stimuli inuence the learning of goal-seeking
behavior in animals Sutton & Barto, 1987]. Reinforcement learning
is an important theme in machine learning research.
1.1. INTRODUCTION 5
Articial Intelligence: From the beginning, AI research has been
concerned with machine learning. Samuel developed a prominent
early program that learned parameters of a function for evaluating
board positions in the game of checkers Samuel, 1959]. AI researchers
have also explored the role of analogies in learning Carbonell, 1983]
and how future actions and decisions can be based on previous
exemplary cases Kolodner, 1993]. Recent work has been directed
at discovering rules for expert systems using decision-tree methods
Quinlan, 1990] and inductive logic programming Muggleton, 1991,
Lavrac & Dzeroski, 1994]. Another theme has been saving and
generalizing the results of problem solving using explanation-based
learning DeJong & Mooney, 1986, Laird, et al., 1986, Minton, 1988,
Etzioni, 1993].
Evolutionary Models:
In nature, not only do individual animals learn to perform better,
but species evolve to be better t in their individual niches. Since the
distinction between evolving and learning can be blurred in computer
systems, techniques that model certain aspects of biological evolution
have been proposed as learning methods to improve the performance
of computer programs. Genetic algorithms Holland, 1975] and ge-
netic programming Koza, 1992, Koza, 1994] are the most prominent
computational techniques for evolution.
1.1.3 Varieties of Machine Learning

Orthogonal to the question of the historical source of any learning technique
is the more important question of what is to be learned. In this book, we
take it that the thing to be learned is a computational structure of some
sort. We will consider a variety of dierent computational structures:
Functions
Logic programs and rule sets
Finite-state machines
Grammars
Problem solving systems
We will present methods both for the synthesis of these structures from
examples and for changing existing structures. In the latter case, the change
to the existing structure might be simply to make it more computationally
ecient rather than to increase the coverage of the situations it can handle.
Much of the terminology that we shall be using throughout the book is best
introduced by discussing the problem of learning functions, and we turn to
that matter rst.
1.2 Learning Input-Output Functions

We use Fig. 1.2 to help dene some of the terminology used in describing
the problem of learning a function. Imagine that there is a function, f ,
and the task of the learner is to guess what it is. Our hypothesis about the
function to be learned is denoted by h. Both f and h are functions of a
vector-valued input X = (x1 x2 : : : xi : : : xn ) which has n components.
We think of h as being implemented by a device that has X as input and
h(X) as output. Both f and h themselves may be vector-valued. We
assume a priori that the hypothesized function, h, is selected from a class
of functions H. Sometimes we know that f also belongs to this class or
to a subset of this class. We select h based on a training set, , of m
input vector examples. Many important details depend on the nature of
the assumptions made about all of these entities.
1.2.1 Types of Learning

There are two major settings in which we wish to learn a function. In one,
called supervised learning, we know (sometimes only approximately) the
values of f for the m samples in the training set, . We assume that if we
can nd a hypothesis, h, that closely agrees with f for the members of ,
then this hypothesis will be a good guess for f |especially if is large.
Curve-tting is a simple example of supervised learning of a function.
Suppose we are given the values of a two-dimensional function, f , at the
four sample points shown by the solid circles in Fig. 1.3. We want to t
these four points with a function, h, drawn from the set, H, of second-degree
functions. We show there a two-dimensional parabolic surface above the x1 ,
x2 plane that ts the points. This parabolic function, h, is our hypothesis
about the function, f , that produced the four samples. In this case, h = f
at the four samples, but we need not have required exact matches.
In the other setting, termed unsupervised learning, we simply have a
training set of vectors without function values for them. The problem in
this case, typically, is to partition the training set into subsets, 1 , : : : ,
R , in some appropriate way. (We can still regard the problem as one of
1.2. LEARNING INPUT-OUTPUT FUNCTIONS 7
Training Set:
Ξ = {X1, X2, . . . Xi, . . ., Xm}
x1
.
.
. h(X)
X= xi h
.
.
.
xn h∈H
Figure 1.2: An Input-Output Function
learning a function the value of the function is the name of the subset
to which an input vector belongs.) Unsupervised learning methods have
application in taxonomic problems in which it is desired to invent ways to
classify data into meaningful categories.
We shall also describe methods that are intermediate between super-
vised and unsupervised learning.
We might either be trying to nd a new function, h, or to modify an
existing one. An interesting special case is that of changing an existing
function into an equivalent one that is computationally more ecient. This
type of learning is sometimes called speed-up learning. A very simple exam-
ple of speed-up learning involves deduction processes. From the formulas
A B and B C , we can deduce C if we are given A. From this deductive
process, we can create the formula A C |a new formula but one that
does not sanction any more conclusions than those that could be derived
from the formulas that we previously had. But with this new formula we
can derive C more quickly, given A, than we could have done before. We
can contrast speed-up learning with methods that create genuinely new
functions|ones that might give dierent results after learning than they
did before. We say that the latter methods involve inductive learning. As
opposed to deduction, there are no correct inductions|only useful ones.
h sample f-value
1500
0
1000
00 10
500
00 5
0
-10 0
-5
0 -5 x2
5
10-10
x1
Figure 1.3: A Surface that Fits Four Points
1.2.2 Input Vectors

Because machine learning methods derive from so many dierent traditions,
its terminology is rife with synonyms, and we will be using most of them
in this book. For example, the input vector is called by a variety of names.
Some of these are: input vector, pattern vector, feature vector, sample, ex-
ample, and instance. The components, xi, of the input vector are variously
called features, attributes, input variables, and components.
The values of the components can be of three main types. They might be
real-valued numbers, discrete-valued numbers, or categorical values. As an
example illustrating categorical values, information about a student might
be represented by the values of the attributes class, major, sex, adviser. A
particular student would then be represented by a vector such as: (sopho-
more, history, male, higgins). Additionally, categorical values may be or-
dered (as in fsmall, medium, largeg) or unordered (as in the example just
given). Of course, mixtures of all these types of values are possible.
In all cases, it is possible to represent the input in unordered form by
listing the names of the attributes together with their values. The vector
form assumes that the attributes are ordered and given implicitly by a form.
As an example of an attribute-value representation, we might have: (major:
history, sex: male, class: sophomore, adviser: higgins, age: 19). We will be
using the vector form exclusively.
An important specialization uses Boolean values, which can be regarded
as a special case of either discrete numbers (1,0) or of categorical variables
1.2. LEARNING INPUT-OUTPUT FUNCTIONS 9
(True, False).
1.2.3 Outputs
The output may be a real number, in which case the process embodying
the function, h, is called a function estimator, and the output is called an
output value or estimate.
Alternatively, the output may be a categorical value, in which case
the process embodying h is variously called a classi er, a recognizer, or a
categorizer, and the output itself is called a label, a class, a category, or a
decision. Classiers have application in a number of recognition problems,
for example in the recognition of hand-printed characters. The input in
that case is some suitable representation of the printed character, and the
classier maps this input into one of, say, 64 categories.
Vector-valued outputs are also possible with components being real
numbers or categorical values.
An important special case is that of Boolean output values. In that
case, a training pattern having value 1 is called a positive instance, and a
training sample having value 0 is called a negative instance. When the input
is also Boolean, the classier implements a Boolean function. We study the
Boolean case in some detail because it allows us to make important general
points in a simplied setting. Learning a Boolean function is sometimes
called concept learning, and the function is called a concept.
1.2.4 Training Regimes

There are several ways in which the training set, , can be used to produce
a hypothesized function. In the batch method, the entire training set is
available and used all at once to compute the function, h. A variation
of this method uses the entire training set to modify a current hypothesis
iteratively until an acceptable hypothesis is obtained. By contrast, in the
incremental method, we select one member at a time from the training set
and use this instance alone to modify a current hypothesis. Then another
member of the training set is selected, and so on. The selection method
can be random (with replacement) or it can cycle through the training set
iteratively. If the entire training set becomes available one member at a
time, then we might also use an incremental method|selecting and using
training set members as they arrive. (Alternatively, at any stage all training
set members so far available could be used in a \batch" process.) Using the
training set members as they become available is called an online method.
Online methods might be used, for example, when the next training instance
is some function of the current hypothesis and the previous instance|as it
would be when a classier is used to decide on a robot's next action given
its current set of sensory inputs. The next set of sensory inputs will depend
on which action was selected.
1.2.5 Noise
Sometimes the vectors in the training set are corrupted by noise. There are
two kinds of noise. Class noise randomly alters the value of the function
attribute noise randomly alters the values of the components of the input
vector. In either case, it would be inappropriate to insist that the hypothe-
sized function agree precisely with the values of the samples in the training
set.
1.2.6 Performance Evaluation

Even though there is no correct answer in inductive learning, it is important
to have methods to evaluate the result of learning. We will discuss this
matter in more detail later, but, briey, in supervised learning the induced
function is usually evaluated on a separate set of inputs and function values
for them called the testing set . A hypothesized function is said to generalize
when it guesses well on the testing set. Both mean-squared-error and the
total number of errors are common measures.
1.3 Learning Requires Bias

Long before now the reader has undoubtedly asked why is learning a func-
tion possible at all? Certainly, for example, there are an uncountable num-
ber of dierent functions having values that agree with the four samples
shown in Fig. 1.3. Why would a learning procedure happen to select the
quadratic one shown in that gure? In order to make that selection we had
at least to limit a priori the set of hypotheses to quadratic functions and
then to insist that the one we chose passed through all four sample points.
This kind of a priori information is called bias, and useful learning without
bias is impossible.
We can gain more insight into the role of bias by considering the special
case of learning a Boolean function of n dimensions. There are 2n dierent
Boolean inputs possible. Suppose we had no bias that is H is the set of
all 22n Boolean functions, and we have no preference among those that t
1.3. LEARNING REQUIRES BIAS 11
the samples in the training set. In this case, after being presented with one
member of the training set and its value we can rule out precisely one-half
of the members of H|those Boolean functions that would misclassify this
labeled sample. The remaining functions constitute what is called a \ver-
sion space" we'll explore that concept in more detail later. As we present
more members of the training set, the graph of the number of hypotheses
not yet ruled out as a function of the number of dierent patterns presented
is as shown in Fig. 1.4. At any stage of the process, half of the remain-
ing Boolean functions have value 1 and half have value 0 for any training
pattern not yet seen. No generalization is possible in this case because the
training patterns give no clue about the value of a pattern not yet seen.
Only memorization is possible here, which is a trivial sort of learning.
|Hv| = no. of functions not ruled out
log2|Hv|
2n − j
2n
(generalization is not possible)
0
0 2n
j = no. of labeled
patterns already seen
Figure 1.4: Hypotheses Remaining as a Function of Labeled Patterns Pre-

sented
But suppose we limited H to some subset, Hc, of all Boolean functions.
Depending on the subset and on the order of presentation of training pat-
terns, a curve of hypotheses not yet ruled out might look something like the
one shown in Fig. 1.5. In this case it is even possible that after seeing fewer
than all 2n labeled samples, there might be only one hypothesis that agrees
with the training set. Certainly, even if there is more than one hypothesis
remaining, most of them may have the same value for most of the patterns
not yet seen! The theory of Probably Approximately Correct (PAC) learning
makes this intuitive idea precise. We'll examine that theory later.
|Hv| = no. of functions not ruled out
log2|Hv|
2n
depends on order
of presentation
log2|Hc|
0
0 2n
j = no. of labeled
patterns already seen
Figure 1.5: Hypotheses Remaining From a Restricted Subset

Let's look at a specic example of how bias aids learning. A Boolean
function can be represented by a hypercube each of whose vertices repre-
sents a dierent input pattern. We show a 3-dimensional version in Fig.
1.6. There, we show a training set of six sample patterns and have marked
those having a value of 1 by a small square and those having a value of 0
by a small circle. If the hypothesis set consists of just the linearly separa-
ble functions|those for which the positive and negative instances can be
separated by a linear surface, then there is only one function remaining in
this hypothsis set that is consistent with the training set. So, in this case,
even though the training set does not contain all possible patterns, we can
already pin down what the function must be|given the bias.
Machine learning researchers have identied two main varieties of bias,
absolute and preference. In absolute bias (also called restricted hypothesis-
space bias), one restricts H to a denite subset of functions. In our example
of Fig. 1.6, the restriction was to linearly separable Boolean functions. In
preference bias, one selects that hypothesis that is minimal according to
1.4. SAMPLE APPLICATIONS 13
x3
x2
x1
Figure 1.6: A Training Set That Completely Determines a Linearly Sepa-

rable Function
some ordering scheme over all hypotheses. For example, if we had some way
of measuring the complexity of a hypothesis, we might select the one that
was simplest among those that performed satisfactorily on the training set.
The principle of Occam's razor, used in science to prefer simple explanations
to more complex ones, is a type of preference bias. (William of Occam,
1285-?1349, was an English philosopher who said: \non sunt multiplicanda
entia praeter necessitatem," which means \entities should not be multiplied
unnecessarily.")
1.4 Sample Applications

Our main emphasis in this book is on the concepts of machine learning|
not on its applications. Nevertheless, if these concepts were irrelevant to
real-world problems they would probably not be of much interest. As mo-
tivation, we give a short summary of some areas in which machine learning
techniques have been successfully applied. Langley, 1992] cites some of the
following applications and others:
1. Rule discovery using a variant of ID3 for a printing industry problem
Evans & Fisher, 1992].
2. Electric power load forecasting using a k-nearest-neighbor rule system
Jabbour, K., et al., 1987].
3. Automatic \help desk" assistant using a nearest-neighbor system
Acorn & Walden, 1992].
4. Planning and scheduling for a steel mill using ExpertEase, a marketed
(ID3-like) system Michie, 1992].
5. Classication of stars and galaxies Fayyad, et al., 1993].
Many application-oriented papers are presented at the annual confer-
ences on Neural Information Processing Systems. Among these are papers
on: speech recognition, dolphin echo recognition, image processing, bio-
engineering, diagnosis, commodity trading, face recognition, music com-
position, optical character recognition, and various control applications
Various Editors, 1989-1994].
As additional examples, Hammerstrom, 1993] mentions:
1. Sharp's Japanese kanji character recognition system processes 200
characters per second with 99+% accuracy. It recognizes 3000+ char-
acters.
2. NeuroForecasting Centre's (London Business School and University
College London) trading strategy selection network earned an average
annual prot of 18% against a conventional system's 12.3%.
3. Fujitsu's (plus a partner's) neural network for monitoring a contin-
uous steel casting operation has been in successful operation since
early 1990.
In summary, it is rather easy nowadays to nd applications of machine
learning techniques. This fact should come as no surprise inasmuch as many
machine learning techniques can be viewed as extensions of well known
statistical methods which have been successfully applied for many years.
1.5 Sources
Besides the rich literature in machine learning (a small part of which is ref-
erenced in the Bibliography), there are several textbooks that are worth
mentioning Hertz, Krogh, & Palmer, 1991, Weiss & Kulikowski, 1991,
Natarjan, 1991, Fu, 1994, Langley, 1996]. Shavlik & Dietterich, 1990,
1.6. BIBLIOGRAPHICAL AND HISTORICAL REMARKS 15
Buchanan & Wilkins, 1993] are edited volumes containing some of the most
important papers. A survey paper by Dietterich, 1990] gives a good
overview of many important topics. There are also well established confer-
ences and publications where papers are given and appear including:
The Annual Conferences on Advances in Neural Information Process-
ing Systems
The Annual Workshops on Computational Learning Theory
The Annual International Workshops on Machine Learning
The Annual International Conferences on Genetic Algorithms
(The Proceedings of the above-listed four conferences are published
by Morgan Kaufmann.)
The journal Machine Learning (published by Kluwer Academic Pub-
lishers).
There is also much information, as well as programs and datasets, available
over the Internet through the World Wide Web.
1.6 Bibliographical and Historical Remarks

To be added.
Every chapter
will contain a
brief survey of
the history of
the material
covered in that
chapter.

Chapter 2
Boolean Functions
2.1 Representation
2.1.1 Boolean Algebra
Many important ideas about learning of functions are most easily presented
using the special case of Boolean functions. There are several important
subclasses of Boolean functions that are used as hypothesis classes for func-
tion learning. Therefore, we digress in this chapter to present a review of
Boolean functions and their properties. (For a more thorough treatment
see, for example, Unger, 1989].)
A Boolean function, f(x1 x2 : : : xn) maps an n-tuple of (0,1) values to
f 0 1 . Boolean algebra is a convenient notation for representing Boolean
g
functions. Boolean algebra uses the connectives , +, and . For example,

the and function of two variables is written x1 x2 . By convention, the
connective, \ " is usually suppressed, and the and function is written x1x2.
x1x2 has value 1 if and only if both x1 and x2 have value 1 if either x1 or x2
has value 0, x1x2 has value 0. The (inclusive) or function of two variables
is written x1 + x2. x1 + x2 has value 1 if and only if either or both of x1
or x2 has value 1 if both x1 and x2 have value 0, x1 + x2 has value 0. The
complement or negation of a variable, x, is written x. x has value 1 if and
only if x has value 0 if x has value 1 has value 1, x has value 0.
These denitions are compactly given by the following rules for Boolean
algebra:
1 + 1 = 1, 1 + 0 = 1, 0 + 0 = 0,
1 1 = 1, 1 0 = 0, 0 0 = 0, and
17
18 CHAPTER 2. BOOLEAN FUNCTIONS
1 = 0, 0 = 1.
Sometimes the arguments and values of Boolean functions are expressed
in terms of the constants T (True) and F (False) instead of 1 and 0, re-
spectively.
The connectives and + are each commutative and associative. Thus,
for example, x1(x2 x3) = (x1x2 )x3, and both can be written simply as
x1x2 x3. Similarly for +.
A Boolean formula consisting of a single variable, such as x1 is called
an atom. One consisting of either a single variable or its complement, such
as x1, is called a literal.
The operators and + do not commute between themselves. Instead,
we have DeMorgan's laws (which can be veried by using the above deni-
tions):
x1 x2 = x1 + x2 , and
x1 + x2 = x1 x2 .
2.1.2 Diagrammatic Representations

We saw in the last chapter that a Boolean function could be represented
by labeling the vertices of a cube. For a function of n variables, we would
need an n-dimensional hypercube. In Fig. 2.1 we show some 2- and 3-
dimensional examples. Vertices having value 1 are labeled with a small
square, and vertices having value 0 are labeled with a small circle.
Using the hypercube representations, it is easy to see how many Boolean
functions of n dimensions there are. A 3-dimensional cube has 23 = 8
vertices, and each may be labeled in two dierent ways thus there are
2(2n3) = 256 dierent Boolean functions of 3 variables. In general, there are
22 Boolean functions of n variables.
We will be using 2- and 3-dimensional cubes later to provide some in-
tuition about the properties of certain Boolean functions. Of course, we
cannot visualize hypercubes (for n > 3), and there are many surprising
properties of higher dimensional spaces, so we must be careful in using
intuitions gained in low dimensions. One diagrammatic technique for di-
mensions slightly higher than 3 is the Karnaugh map. A Karnaugh map
is an array of values of a Boolean function in which the horizontal rows
are indexed by the values of some of the variables and the vertical columns
are indexed by the rest. The rows and columns are arranged in such a
way that entries that are adjacent in the map correspond to vertices that
are adjacent in the hypercube representation. We show an example of the
4-dimensional even parity function in Fig. 2.2. (An even parity function is
2.2. CLASSES OF BOOLEAN FUNCTIONS 19
x2 x2
x1x2 x1 + x2
x1 x1
and or
x2 x3
x1x2x3 + x1x2x3
x1x2 + x1x2
+ x1x2x3 + x1x2x3
x1 x2
xor (exclusive or)
x1 even parity function
Figure 2.1: Representing Boolean Functions on Cubes
a Boolean function that has value 1 if there are an even number of its argu-
ments that have value 1 otherwise it has value 0.) Note that all adjacent
cells in the table correspond to inputs diering in only one component. Also describe
general logic
diagrams,
Wnek, et al., 1990].
2.2 Classes of Boolean Functions
2.2.1 Terms and Clauses
To use absolute bias in machine learning, we limit the class of hypotheses.
In learning Boolean functions, we frequently use some of the common sub-
classes of those functions. Therefore, it will be important to know about
these subclasses.
One basic subclass is called terms. A term is any function written
in the form l1 l2 lk , where the li are literals. Such a form is called a
conjunction of literals. Some example terms are x1 x7 and x1x2x4 . The size
of a term is the number of literals it contains. The examples are of sizes 2
and 3, respectively. (Strictly speaking, the class of conjunctions of literals
x3,x4
00 01 11 10
00 1 0 1 0
x1,x2 01 0 1 0 1
11 1 0 1 0
10 0 1 0 1
Figure 2.2: A Karnaugh Map
is called the monomials, and a conjunction of literals itself is called a term.

This distinction is a ne one which we elect to blur here.)
It is easy to show that there are exactly 3n possible terms of n vari-
Pki=0 C(2n
ables. The number of terms of size k or less is bounded from above by
i) = O(nk ), where C(i j) = (i;ij!)!j ! is the binomial coecient.
Probably I'll
put in a simple
term-learning A clause is any function written in the form l1 + l2 + + lk , where
algorithm
here|so we the li are literals. Such a form is called a disjunction of literals. Some
can get started example clauses are x3 + x5 + x6 and x1 + x4. The size of a clause is the
on learning!
Also for DNF
functions and
Pki=0 C(2n
number of literals it contains. There are 3n possible clauses and fewer than
i) clauses of size k or less. If f is a term, then (by De Morgan's
decision
lists|as they laws) f is a clause, and vice versa. Thus, terms and clauses are duals of
are dened in each other.
the next few
pages. In psychological experiments, conjunctions of literals seem easier for
humans to learn than disjunctions of literals.
2.2.2 DNF Functions

A Boolean function is said to be in disjunctive normal form (DNF) if it
can be written as a disjunction of terms. Some examples in DNF are:
f = x1x2 + x2x3x4 and f = x1 x3 + x2 x3 + x1x2 x3. A DNF expression
is called a k-term DNF expression if it is a disjunction of k terms it is in
the class k-DNF if the size of its largest term is k. The examples above are
2-term and 3-term expressions, respectively. Both expressions are in the
class 3-DNF.
Each term in a DNF expression for a function is called an implicant
because it \implies" the function (if the term has value 1, so does the
function). In general, a term, t, is an implicant of a function, f, if f has
value 1 whenever t does. A term, t, is a prime implicant of f if the term, t0,
formed by taking any literal out of an implicant t is no longer an implicant
of f. (The implicant cannot be \divided" by any term and remain an
implicant.)
Thus, both x2x3 and x1 x3 are prime implicants of f = x2x3 + x1 x3 +
x2x1 x3, but x2x1x3 is not.
The relationship between implicants and prime implicants can be geo-
metrically illustrated using the cube representation for Boolean functions.
Consider, for example, the function f = x2 x3 + x1 x3 + x2x1x3 . We illus-
trate it in Fig. 2.3. Note that each of the three planes in the gure \cuts
o" a group of vertices having value 1, but none cuts o any vertices hav-
ing value 0. These planes are pictorial devices used to isolate certain lower
dimensional subfaces of the cube. Two of them isolate one-dimensional
edges, and the third isolates a zero-dimensional vertex. Each group of ver-
tices on a subface corresponds to one of the implicants of the function, f,
and thus each implicant corresponds to a subface of some dimension. A
k-dimensional subface corresponds to an (n k)-size implicant term. The
;
function is written as the disjunction of the implicants|corresponding to

the union of all the vertices cut o by all of the planes. Geometrically,
an implicant is prime if and only if its corresponding subface is the largest
dimensional subface that includes all of its vertices and no other vertices
having value 0. Note that the term x2x1x3 is not a prime implicant of
f. (In this case, we don't even have to include this term in the function
because the vertex cut o by the plane corresponding to x2 x1x3 is already
cut o by the plane corresponding to x2x3.) The other two implicants are
prime because their corresponding subfaces cannot be expanded without
including vertices having value 0.
Note that all Boolean functions can be represented in DNF|trivially
by disjunctions of terms of size n where each term corresponds to one of the
vertices whose value is 1. Whereas there are 22n functions of n dimensions
in DNF (since any Boolean function can be written in DNF), there are just
2O ( nk)
functions in k-DNF.
All Boolean functions can also be represented in DNF in which each
term is a prime implicant, but that representation is not unique, as shown
in Fig. 2.4.
If we can express a function in DNF form, we can use the consensus
method to nd an expression for the function in which each term is a prime
implicant. The consensus method relies on two results: We may
replace this
section with
Consensus: one describing
the Quine-
McCluskey
Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved. method
instead.
x3
0, 0, 1 1, 0, 1
1, 1, 1
1, 0, 0
x2
x1
f = x2x3 + x1x3 + x2x1x3
= x2x3 + x1x3
x2x3 and x1x3 are prime implicants
Figure 2.3: A Function and its Implicants
xi f1 + xi f2 = xi f1 + xi f2 + f1 f2
where f1 and f2 are terms such that no literal appearing in f1 appears
complemented in f2 . f1 f2 is called the consensus of xi f1 and xi f2.
Readers familiar with the resolution rule of inference will note that
consensus is the dual of resolution.
Examples: x1 is the consensus of x1x2 and x1x2 . The terms x1 x2
and x1 x2 have no consensus since each term has more than one literal
appearing complemented in the other.
Subsumption:
xi f1 + f1 = f1
x3
0, 0, 1 1, 0, 1
1, 1, 1
1, 0, 0
x2
x1
f = x2x3 + x1x3 + x1x2
= x1x2 + x1x3
All of the terms are prime implicants, but there

is not a unique representation
Figure 2.4: Non-Uniqueness of Representation by Prime Implicants
where f1 is a term. We say that f1 subsumes xi f1 .

Example: x1 x4 x5 subsumes x1 x4 x2x5
The consensus method for nding a set of prime implicants for a func-
tion, f, iterates the following operations on the terms of a DNF expression
for f until no more such operations can be applied:
1. initialize the process with the set, , of terms in the DNF expression
T
of f,
2. compute the consensus of a pair of terms in and add the result to
T
T ,
3. eliminate any terms in that are subsumed by other terms in .
T T

When this process halts, the terms remaining in are all prime implicants
T
of f.
Example: Let f = x1 x2 +x1 x2 x3 +x1 x2 x3 x4 x5. We show a derivation
of a set of prime implicants in the consensus tree of Fig. 2.5. The circled
numbers adjoining the terms indicate the order in which the consensus and
subsumption operations were performed. Shaded boxes surrounding a term
indicate that it was subsumed. The nal form of the function in which all
terms are prime implicants is: f = x1 x2 + x1x3 + x1 x4x5. Its terms are all
of the non-subsumed terms in the consensus tree.
2 4
x1x2 x1x2x3 x1x2x3x4x5
1
x1x3
6
3 x1x2x4x5
f = x1x2 + x x + x x x
1 3 1 4 5
5
x1x4x5
Figure 2.5: A Consensus Tree
2.2.3 CNF Functions

Disjunctive normal form has a dual: conjunctive normal form (CNF). A
Boolean function is said to be in CNF if it can be written as a conjunction
of clauses. An example in CNF is: f = (x1 + x2)(x2 + x3 + x4 ). A CNF
expression is called a k-clause CNF expression if it is a conjunction of k
clauses it is in the class k-CNF if the size of its largest clause is k. The
example is a 2-clause expression in 3-CNF. If f is written in DNF, an
application of De Morgan's law renders f in CNF, and vice versa. Because
CNF and DNF are duals, there are also 2O(nk) functions in k-CNF.
2.2.4 Decision Lists

Rivest has proposed a class of Boolean functions called decision lists
Rivest, 1987]. A decision list is written as an ordered list of pairs:
(tq vq )
(tq;1 vq;1 )
(ti vi )
(t2 v2)
(T v1 )
where the vi are either 0 or 1, the ti are terms in (x1 : : : xn), and T is a
term whose value is 1 (regardless of the values of the xi). The value of a
decision list is the value of vi for the rst ti in the list that has value 1. (At
least one ti will have value 1, because the last one does v1 can be regarded
as a default value of the decision list.) The decision list is of size k, if the
size of the largest term in it is k. The class of decision lists of size k or less
is called k-DL.
An example decision list is:
f=
(x1 x2 1)
(x1 x2x3 0)
x2 x3 1)
(1 0)
f has value 0 for x1 = 0, x2 = 0, and x3 = 1. It has value 1 for x1 = 1,
x2 = 0, and x3 = 1. This function is in 3-DL.
It has been shown that the class k-DL is a strict superset of the union of
k-DNF and k-CNF. There are 2Onk k log(n)] functions in k-DL Rivest, 1987].
Interesting generalizations of decision lists use other Boolean functions
in place of the terms, ti. For example we might use linearly separable
functions in place of the ti (see below and Marchand & Golea, 1993]).
2.2.5 Symmetric and Voting Functions
A Boolean function is called symmetric if it is invariant under permutations
of the input variables. For example, any function that is dependent only on
the number of input variables whose values are 1 is a symmetric function.
The parity functions, which have value 1 depending on whether or not
the number of input variables with value 1 is even or odd is a symmetric
function. (The exclusive or function, illustrated in Fig. 2.1, is an odd-parity
function of two dimensions. The or and and functions of two dimensions
are also symmetric.)
An important subclass of the symmetric functions is the class of voting
functions (also called m-of-n functions). A k-voting function has value 1 if
and only if k or more of its n inputs has value 1. If k = 1, a voting function
is the same as an n-sized clause if k = n, a voting function is the same as
an n-sized term if k = (n + 1)=2 for n odd or k = 1 + n=2 for n even, we
have the majority function.
2.2.6 Linearly Separable Functions

The linearly separable functions are those that can be expressed as follows:
f = thresh(
Xn w x
i i )
i=1
where wi , i = 1 : : : n, are real-valued numbers called weights, is a real-
valued number called the threshold, and thresh( ) is 1 if and 0

otherwise. (Note that the concept of linearly separable functions can be

extended to non-Boolean inputs.) The k-voting functions are all members
of the class of linearly separable functions in which the weights all have unit
value and the threshold depends on k. Thus, terms and clauses are special
cases of linearly separable functions.
A convenient way to write linearly separable functions uses vector no-
tation:
f = thresh( X W )
X W
where = (x1 : : : xn) is an n-dimensional vector of input variables, =
(w1 : : : wn) is an n-dimensional vector of weight values, and XW is
the dot (or inner) product of the two vectors. Input vectors for which f
has value 1 lie in a half-space on one side of (and on) a hyperplane whose
W
orientation is normal to and whose position (with respect to the origin)
2.3. SUMMARY 27
is determined by . We saw an example of such a separating plane in Fig.
1.6. With this idea in mind, it is easy to see that two of the functions in
Fig. 2.1 are linearly separable, while two are not. Also note that the terms
in Figs. 2.3 and 2.4 are linearly separable functions as evidenced by the
separating planes shown.
There is no closed-form expression for the number of linearly separable
functions of n dimensions, but the following table gives the numbers for n
up to 6.
n Boolean Linearly Separable
Functions Functions
1 4 4
2 16 14
3 256 104
4 65,536 1,882
5 4:3 109
94,572
6 1:8 1019
15,028,134
Muroga, 1971] has shown that (for n > 1) there are no more than 2n2
linearly separable functions of n dimensions. (See also Winder, 1961,
Winder, 1962].)
2.3 Summary
The diagram in Fig. 2.6 shows some of the set inclusions of the classes of
Boolean functions that we have considered. We will be confronting these
classes again in later chapters.
The sizes of the various classes are given in the following table (adapted
from Dietterich, 1990, page 262]):
Class Size of Class
terms 3n
clauses 3n
k-term DNF 2 (kn)
O
k-clause CNF 2O(knk )
k-DNF 2O(n )
k-CNF 2Ok (nk)
k-DL 2On k log( n)]
lin sep O
2 n (n 2)
DNF 22
k-size-
k-DNF k-DL
terms
terms
DNF
lin sep (All)
Figure 2.6: Classes of Boolean Functions

To be added.

Chapter 3
Using Version Spaces for

Learning
3.1 Version Spaces and Mistake Bounds
The rst learning methods we present are based on the concepts of version
spaces and version graphs. These ideas are most clearly explained for the
case of Boolean function learning. Given an initial hypothesis set H (a
subset of all Boolean functions) and the values of (X) for each X in a
f
training set, , the version space is that subset of hypotheses, Hv , that is

consistent with these values. A hypothesis, , is consistent with the values
h
of X in if and only if (X) = (X) for all X in . We say that the

h f
hypotheses in H that are not consistent with the values in the training set
are ruled out by the training set.
We could imagine (conceptually only!) that we have devices for imple-
menting every function in H. An incremental training procedure could then
be de ned which presented each pattern in to each of these functions and
then eliminated those functions whose values for that pattern did not agree
with its given value. At any stage of the process we would then have left
some subset of functions that are consistent with the patterns presented so
far this subset is the version space for the patterns already presented. This
idea is illustrated in Fig. 3.1.
Consider the following procedure for classifying an arbitrary input pat-
tern, X: the pattern is put in the same class (0 or 1) as are the majority of
the outputs of the functions in the version space. During the learning pro-
cedure, if this majority is not equal to the value of the pattern presented,
29
30 CHAPTER 3. USING VERSION SPACES FOR LEARNING
A Subset, H, of all
Boolean Functions
h1 1 or 0
h2
Rule out hypotheses not

consistent with training patterns
X hi
hj
Hypotheses not ruled out

constitute the version space
hK
K = |H|
Figure 3.1: Implementing the Version Space
we say a mistake is made, and we revise the version space accordingly|

eliminating all those (majority of the) functions voting incorrectly. Thus,
whenever a mistake is made, we rule out at least half of the functions re-
maining in the version space.
How many mistakes can such a procedure make? Obviously, we can
make no more than log2 (jHj) mistakes, where jHj is the number of hy-
potheses in the original hypothesis set, H. (Note, though, that the number
of training patterns seen before this maximum number of mistakes is made
might be much greater.) This theoretical (and very impractical!) result
(due to Littlestone, 1988]) is an example of a mistake bound|an impor-
tant concept in machine learning theory. It shows that there must exist a
learning procedure that makes no more mistakes than this upper bound.
Later, we'll derive other mistake bounds.
As a special case, if our bias was to limit H to terms, we would make
no more than log2 (3n) = log2 (3) = 1 585 mistakes before exhausting the
n : n

3.2. VERSION GRAPHS 31
version space. This result means that if were a term, we would make no
f
more than 1 585 mistakes before learning , and otherwise we would make
: n f
no more than that number of mistakes before being able to decide that f
is not a term.
Even if we do not have sucient training patterns to reduce the ver-
sion space to a single function, it may be that there are enough training
patterns to reduce the version space to a set of functions such that most
of them assign the same values to most of the patterns we will see hence-
forth. We could select one of the remaining functions at random and be
reasonably assured that it will generalize satisfactorily. We next discuss a
computationally more feasible method for representing the version space.
3.2 Version Graphs

Boolean functions can be ordered by generality. A Boolean function, 1 , is f
more general than a function, 2 , (and 2 is more speci c than 1 ), if 1

f f f f
has value 1 for all of the arguments for which 2 has value 1, and 1 6= 2.
f f f
For example, 3 is more general than 2 3 but is not more general than
x x x
x 3 + 2.
x
We can form a graph with the hypotheses, f i g, in the version space

h
as nodes. A node in the graph, i , has an arc directed to node, j , if and

h h
only if j is more general than i . We call such a graph a version graph.

h h
In Fig. 3.2, we show an example of a version graph over a 3-dimensional

input space for hypotheses restricted to terms (with none of them yet ruled
out).
That function, denoted here by \1," which has value 1 for all inputs,
corresponds to the node at the top of the graph. (It is more general than
any other term.) Similarly, the function \0," is at the bottom of the graph.
Just below \1," is a row of nodes corresponding to all terms having just
one literal, and just below them is a row of nodes corresponding to terms
having two literals, and so on. There are 33 = 27 functions altogether (the
function \0," included in the graph, is technically not a term). To make
our portrayal of the graph less cluttered only some of the arcs are shown
each node in the actual graph has an arc directed to all of the nodes above
it that are more general.
We use this same example to show how the version graph changes as
we consider a set of labeled samples in a training set, . Suppose we
rst consider the training pattern (1, 0, 1) with value 0. Some of the
functions in the version graph of Fig. 3.2 are inconsistent with this training
pattern. These ruled out nodes are no longer in the version graph and are
x3
Version Graph for Terms
(none yet ruled out) x2

1 x1
x1 x1 x2 (k = 1)
x2 x3
x3
(k = 2)
x1 x3 x1x2
x1x2 x3
(k = 3)
0
(for simplicity, only some arcs in the graph are shown)
Figure 3.2: A Version Graph for Terms
shown shaded in Fig. 3.3. We also show there the three-dimensional cube
representation in which the vertex (1, 0, 1) has value 0.
In a version graph, there are always a set of hypotheses that are max-
imally general and a set of hypotheses that are maximally speci c. These
are called the general boundary set (gbs) and the speci c boundary set (sbs),
respectively. In Fig. 3.4, we have the version graph as it exists after learn-
ing that (1,0,1) has value 0 and (1, 0, 0) has value 1. The gbs and sbs are
shown.
Boundary sets are important because they provide an alternative to
representing the entire version space explicitly, which would be impractical.
Given only the boundary sets, it is possible to determine whether or not
any hypothesis (in the prescribed class of Boolean functions we are using)
3.2. VERSION GRAPHS 33
x3
1, 0, 1 has
value 0
x2
x1
1
New Version Graph
ruled out nodes
x1 x2 x3
x1 x2 x3
x1x2 x2x3
x1x3
x1x3 x1x2
x1x2 x3
x1x2x3
0
(only some arcs in the graph are shown)
Figure 3.3: The Version Graph Upon Seeing (1, 0, 1)
is a member or not of the version space. This determination is possible

because of the fact that any member of the version space (that is not a
member of one of the boundary sets) is more speci c than some member
of the general boundary set and is more general than some member of the
speci c boundary set.
If we limit our Boolean functions that can be in the version space to
terms, it is a simple matter to determine maximally general and maximally
speci c functions (assuming that there is some term that is in the version
space). A maximally speci c one corresponds to a subface of minimal
dimension that contains all the members of the training set labelled by a 1
and no members labelled by a 0. A maximally general one corresponds to a
subface of maximal dimension that contains all the members of the training
x3
1, 0, 1 has
value 0
1, 0, 0 has x2
value 1
x1
1
more specific than gbs, general boundary set
more general than sbs (gbs)
x3
x1 x1 x2 x2 x3
x1x3 x2x3
x1x2
x1x2 x
3
specific boundary set (sbs)
Figure 3.4: The Version Graph Upon Seeing (1, 0, 1) and (1, 0, 0)
set labelled by a 1 and no members labelled by a 0. Looking at Fig. 3.4,

we see that the subface of minimal dimension that contains (1, 0, 0) but
does not contain (1, 0, 1) is just the vertex (1, 0, 0) itself|corresponding
to the function 1 2 3. The subface of maximal dimension that contains
x x x
(1, 0, 0) but does not contain (1, 0, 1) is the bottom face of the cube|
corresponding to the function 3 . In Figs. 3.2 through 3.4 the sbs is always
x
singular. Version spaces for terms always have singular speci c boundary
sets. As seen in Fig. 3.3, however, the gbs of a version space for terms need
not be singular.
3.3 Learning as Search of a Version Space

To be written. Relate to term learning algorithm presented in Chapter
Two. Also discuss best- rst search methods. See Pat Langley's example
3.4. THE CANDIDATE ELIMINATION METHOD 35
using \pseudo-cells" of how to generate and eliminate hypotheses.]
Selecting a hypothesis from the version space can be thought of as a
search problem. One can start with a very general function and special-
ize it through various specialization operators until one nds a function
that is consistent (or adequately so) with a set of training patterns. Such
procedures are usually called top-down methods. Or, one can start with
a very special function and generalize it|resulting in bottom-up methods.
We shall see instances of both styles of learning in this book. Compare this
view of
top-down
3.4 The Candidate Elimination Method
versus
bottom-up
with the
divide-and-
conquer and
The candidate elimination method, is an incremental method for computing the covering
the boundary sets. Quoting from Hirsh, 1994, page 6]: (or AQ)
methods of
decision-tree
\The candidate-elimination algorithm manipulates the boundary- induction.
set representation of a version space to create boundary sets that
represent a new version space consistent with all the previous
instances plus the new one. For a positive exmple the algo-
rithm generalizes the elements of the sbs] as little as possible
so that they cover the new instance yet remain consistent with
past data, and removes those elements of the gbs] that do not
cover the new instance. For a negative instance the algorithm
specializes elements of the gbs] so that they no longer cover
the new instance yet remain consistent with past data, and re-
moves from the sbs] those elements that mistakenly cover the
new, negative instance."
The method uses the following de nitions (adapted from
Genesereth & Nilsson, 1987]):
a hypothesis is called sucient if and only if it has value 1 for all
training samples labeled by a 1,
a hypothesis is called necessary if and only if it has value 0 for all
training samples labeled by a 0.
Here is how to think about these de nitions: A hypothesis implements a
sucient condition that a training sample has value 1 if the hypothesis has
value 1 for all of the positive instances a hypothesis implements a necessary
condition that a training sample has value 1 if the hypothesis has value 0 for
all of the negative instances. A hypothesis is consistent with the training
set (and thus is in the version space) if and only if it is both sucient and
necessary.
We start (before receiving any members of the training set) with the
function \0" as the singleton element of the speci c boundary set and with
the function \1" as the singleton element of the general boundary set. Upon
receiving a new labeled input vector, the boundary sets are changed as
follows:
1. If the new vector is labelled with a 1:
The new general boundary set is obtained from the previous one by
excluding any elements in it that are not sucient. (That is, we
exclude any elements that have value 0 for the new vector.)
The new speci c boundary set is obtained from the previous one by
replacing each element, i, in it by all of its least generalizations.
h
The hypothesis g is a least generalization of if and only if: a) is

h h h
more speci c than g , b) g is sucient, c) no function (including )

h h h
that is more speci c than g is sucient, and d) g is more speci c

h h
than some member of the new general boundary set. It might be that
h g = . Also, least generalizations of two dierent functions in the
h
speci c boundary set may be identical.

2. If the new vector is labelled with a 0:
The new speci c boundary set is obtained from the previous one by
excluding any elements in it that are not necessary. (That is, we
exclude any elements that have value 1 for the new vector.)
The new general boundary set is obtained from the previous one by
replacing each element, i, in it by all of its least specializations.
h
The hypothesis s is a least specialization of if and only if: a) is

h h h
more general than s, b) s is necessary, c) no function (including )

h h h
that is more general than s is necessary, and d) s is more general

h h
than some member of the new speci c boundary set. Again, it might
be that s = , and least specializations of two dierent functions in
h h
the general boundary set may be identical.

As an example, suppose we present the vectors in the following order:
vector label
(1, 0, 1) 0
(1, 0, 0) 1
(1, 1, 1) 0
(0, 0, 1) 0
We start with general boundary set, \1", and speci c boundary set,
\0." After seeing the rst sample, (1, 0, 1), labeled with a 0, the speci c
boundary set stays at \0" (it is necessary), and we change the general
boundary set to f 1 2 3g. Each of the functions, 1, 2 , and 3, are
x x x x x x
least specializations of \1" (they are necessary, \1" is not, they are more
general than \0", and there are no functions that are more general than
they and also necessary).
Then, after seeing (1, 0, 0), labeled with a 1, the general boundary set
changes to f 3g (because 1 and 2 are not sucient), and the speci c
x x x
boundary set is changed to f 1 2 3g. This single function is a least gen-

x x x
eralization of \0" (it is sucient, \0" is more speci c than it, no function
(including \0") that is more speci c than it is sucient, and it is more
speci c than some member of the general boundary set.
When we see (1, 1, 1), labeled with a 0, we do not change the speci c
boundary set because its function is still necessary. We do not change the
general boundary set either because 3 is still necessary.
x
Finally, when we see (0, 0, 1), labeled with a 0, we do not change the
speci c boundary set because its function is still necessary. We do not
change the general boundary set either because 3 is still necessary.
x Maybe I'll put
in an example
of a version
graph for
3.5 Bibliographical and Historical Remarks non-Boolean
functions.
The concept of version spaces and their role in learning was rst investigated
by Tom Mitchell Mitchell, 1982]. Although these ideas are not used in
practical machine learning procedures, they do provide insight into the
nature of hypothesis selection. In order to accomodate noisy data, version
spaces have been generalized by Hirsh, 1994] to allow hypotheses that are
not necessarily consistent with the training set. More to be
added.


Chapter 4
Neural Networks
In chapter two we de ned several important subsets of Boolean functions.
Suppose we decide to use one of these subsets as a hypothesis set for su-
pervised function learning. We next have the question of how best to im-
plement the function as a device that gives the outputs prescribed by the
function for arbitrary inputs. In this chapter we describe how networks of
non-linear elements can be used to implement various input-output func-
tions and how they can be trained using supervised learning methods.
Networks of non-linear elements, interconnected through adjustable
weights, play a prominent role in machine learning. They are called neural
networks because the non-linear elements have as their inputs a weighted
sum of the outputs of other elements|much like networks of biological
neurons do. These networks commonly use the threshold element which
we encountered in chapter two in our study of linearly separable Boolean
functions. We begin our treatment of neural nets by studying this thresh-
old element and how it can be used in the simplest of all networks, namely
ones composed of a single threshold element.
4.1 Threshold Logic Units

4.1.1 Denitions and Geometry
Linearly separable (threshold) functions are implemented in a straightfor-
ward way by summing the weighted inputs and comparing this sum to a
threshold value as shown in Fig. 4.1. This structure we call a thresh-
old logic unit (TLU). Its output is 1 or 0 depending on whether or not
39
40 CHAPTER 4. NEURAL NETWORKS
the weighted sum of its inputs is greater than or equal to a threshold
value, . It has also been called an Adaline (for adaptive linear element)
Widrow, 1962, Widrow & Lehr, 1990], an LTU (linear threshold unit), a
perceptron, and a neuron. (Although the word \perceptron" is often used
nowadays to refer to a single TLU, Rosenblatt originally de ned it as a
class of networks of threshold elements Rosenblatt, 1958].)
x1
W threshold θ = 0
w1
x
2
Σ
w2
X x
i wi
f
xn wn
threshold weight
wn+1
xn+1 = 1 n+1
f = thresh( Σ wi xi, 0)
i=1
Figure 4.1: A Threshold Logic Unit (TLU)

The n-dimensional feature or input vector is denoted by X =
(x1 : : : xn). When we want to distinguish among dierent feature vec-
tors, we will attach subscripts, such as Xi . The components of X can be
any real-valued numbers, but we often specialize to the binary numbers 0
and 1. The weights of a TLU are represented by an n-dimensional weight
vector, W = (w1 : : : wn). Its components are real-valued numbers
P (but we
sometimes specialize to integers). The TLU has output 1 if ni=1 xi wi
otherwise it has output 0. The weighted sum that is calculated by the
TLU can be simply represented as a vector dot product, X W. (If the
pattern and weight vectors are thought of as \column" vectors, this dot
product is then sometimes written as XtW, where the \row" vector Xt is
the transpose of X.) Often, the threshold, , of the TLU is xed at 0 in
that case, arbitrary thresholds are achieved by using (n + 1)-dimensional
\augmented" vectors, Y, and V, whose rst n components are the same
as those of X and W, respectively. The (n + 1)-st component, xn+1 , of
the augmented feature vector, Y, always has value 1 the (n+ 1)-st compo-
nent, wn+1, of the augmented weight vector, V, is set equal to the negative
of the desired threshold value. (When we want to emphasize the use of
Introduction to Machine Learning c 1996
Nils J. Nilsson. All rights reserved.
4.1. THRESHOLD LOGIC UNITS 41
augmented vectors, we'll use the Y,V notation however when the context
of the discussion makes it clear about what sort of vectors we are talking
about, we'll lapse back into the more familiar X,W notation.) In the Y,V
notation, the TLU has an output of 1 if Y V 0. Otherwise, the output
is 0.
We can give an intuitively useful geometric description of a TLU. A
TLU divides the input space by a hyperplane as sketched in Fig. 4.2. The
hyperplane is the boundary between patterns for which XW + wn+1 > 0
and patterns for which XW + wn+1 < 0. Thus, the equation of the
hyperplane itself is XW +wn+1 = 0. p The unit vector that is normal to the
hyperplane is n = jW Wj , where jW j = (w12 + : : : + wn2 ) is the length of the
vector W. (The normal form of the hyperplane equation is Xn+ jW Wj = 0.)
w
The distance from the hyperplane to the origin is jWj , and the distance
n+1
from an arbitrary point, X, to the hyperplane is X W +wn+1 . When the

jWj
distance from the hyperplane to the origin is negative (that is, when wn+1 <
0), then the origin is on the negative side of the hyperplane (that is, the
side for which XW + wn+1 < 0).
Adjusting the weight vector, W, changes the orientation of the hyper-
plane adjusting wn+1 changes the position of the hyperplane (relative to
the origin). Thus, training of a TLU can be achieved by adjusting the val-
ues of the weights. In this way the hyperplane can be moved so that the
TLU implements dierent (linearly separable) functions of the input.
4.1.2 Special Cases of Linearly Separable Functions

Terms
Any term of size k can be implemented by a TLU with a weight from each
of those inputs corresponding to variables occurring in the term. A weight
of +1 is used from an input corresponding to a positive literal, and a weight
of ;1 is used from an input corresponding to a negative literal. (Literals
not mentioned in the term have weights of zero|that is, no connection at
all|from their inputs.) The threshold, , is set equal to kp ; 1=2, where
kp is the number of positive literals in the term. Such a TLU implements
a hyperplane boundary that is parallel to a subface of dimension (n ; k)
of the unit hypercube. We show a three-dimensional example in Fig. 4.3.
Thus, linearly separable functions are a superset of terms.
Equations of hyperplane:
X W + wn+1 = 0
wn+1
X n+ =0
|W| X
X.W + w n+1 > 0
on this side
wn+1
X W + wn+1
W W
X.W + wn+1 < 0
on this side
Origin
n= W
|W|
Unit vector normal
to hyperplane
Figure 4.2: TLU Geometry
Clauses
The negation of a clause is a term. For example, the negation of the clause
f = x1 + x2 + x3 is the term f = x1 x2 x3. A hyperplane can be used to
implement this term. If we \invert" the hyperplane, it will implement the
clause instead. Inverting a hyperplane is done by multiplying all of the TLU
weights|even wn+1|by ;1. This process simply changes the orientation
of the hyperplane|ipping it around by 180 degrees and thus changing its
\positive side." Therefore, linearly separable functions are also a superset
of clauses. We show an example in Fig. 4.4.
4.1.3 Error-Correction Training of a TLU

There are several procedures that have been proposed for adjusting the
weights of a TLU. We present next a family of incremental training proce-
dures with parameter c. These methods make adjustments to the weight
vector only when the TLU being trained makes an error on a training pat-
tern they are called error-correction procedures. We use augmented feature
and weight vectors in describing them.
x3
f = x1x2
(1,1,1)
x2
(1,1,0)
x1
Equation of plane is:
x1 + x2 - 3/2 = 0
Figure 4.3: Implementing a Term
1. We start with a nite training set, , of vectors, Yi , and their binary

labels.
2. Compose an in nite training sequence, , of vectors from and their
labels such that each member of occurs in nitely often in . Set
the initial weight values of an TLU to arbitrary values.
3. Repeat forever:
Present the next vector, Yi, in to the TLU and note its response.
(a) If the TLU responds correctly, make no change in the weight
vector.
(b) If Yi is supposed to produce an output of 0 and produces an
output of 1 instead, modify the weight vector as follows:
V ; V ; ciYi
where ci is a positive real number called the learning rate pa-
rameter (whose value is diererent in dierent instances of this
family of procedures and may depend on i).
Note that after this adjustment the new dot product will be
(V ; ci Yi )Yi = VYi ; ci Yi Yi, which is smaller than it was
before the weight adjustment.
f = x1 + x2 + x3
x3 f = x1x2x3
x2
x1
Equation of plane is:
x1 + x2 + x3 − 1/2 = 0
Figure 4.4: Implementing a Clause
(c) If Yi is supposed to produce an output of 1 and produces an

output of 0 instead, modify the weight vector as follows:
V ; V + ciYi
In this case, the new dot product will be (V + ci Yi )Yi =
VYi + ci Yi Yi, which is larger than it was before the weight
adjustment.
Note that all three of these cases can be combined in the following
rule:
V ; V + ci (di ; fi)Yi
where di is the desired response (1 or 0) for Yi , and fi is the actual
response (1 or 0) for Yi .]
Note also that because the weight vector V now includes the wn+1
threshold component, the threshold of the TLU is also changed by
these adjustments.
We identify two versions of this procedure:
1) In the xed-increment procedure, the learning rate parameter, ci, is
the same xed, positive constant for all i. Depending on the value of this
constant, the weight adjustment may or may not correct the response to
an erroneously classi ed feature vector.
2) In the fractional-correction procedure, the parameter ci is set to
Y V
Y i Y , where V is the weight vector before it is changed. Note that
i i
if = 0, no correction takes place at all. If = 1, the correction is just
sucient to make Yi V = 0. If > 1, the error will be corrected.
It can be proved that if there is some weight vector, V, that produces a
correct output for all of the feature vectors in , then after a nite number
of feature vector presentations, the xed-increment procedure will nd such
a weight vector and thus make no more weight changes. The same result
holds for the fractional-correction procedure if 1 < 2.
For additional background, proofs, and examples of error-correction pro-
cedures, see Nilsson, 1990]. See
Maass & Turan, 1994]
for a
4.1.4 Weight Space hyperplane-
nding
procedure that
We can give an intuitive idea about how these procedures work by con- makes no more
than2
sidering what happens to the augmented weight vector in \weight space" O(n log n)
as corrections are made. We use augmented vectors in our discussion here mistakes.
so that the threshold function compares the dot product, Yi V, against
a threshold of 0. A particular weight vector, V, then corresponds to a
point in (n + 1)-dimensional weight space. Now, for any pattern vector,
Yi , consider the locus of all points in weight space corresponding to weight
vectors yielding Yi V = 0. This locus is a hyperplane passing through the
origin of the (n + 1)-dimensional space. Each pattern vector will have such
a hyperplane corresponding to it. Weight points in one of the half-spaces
de ned by this hyperplane will cause the corresponding pattern to yield a
dot product less than 0, and weight points in the other half-space will cause
the corresponding pattern to yield a dot product greater than 0.
We show a schematic representation of such a weight space in Fig. 4.5.
There are four pattern hyperplanes, 1, 2, 3, 4 , corresponding to patterns
Y1 , Y2 , Y3, Y4, respectively, and we indicate by an arrow the half-space
for each in which weight vectors give dot products greater than 0. Suppose
we wanted weight values that would give positive responses for patterns Y1,
Y3 , and Y4, and a negative response for pattern Y2. The weight point, V,
indicated in the gure is one such set of weight values.
The question of whether or not there exists a weight vector that gives
desired responses for a given set of patterns can be given a geometric inter-
pretation. To do so involves reversing the \polarity" of those hyperplanes
corresponding to patterns for which a negative response is desired. If we
do that for our example above, we get the weight space diagram shown in
Fig. 4.6.
3
2
1
V
Figure 4.5: Weight Space
If a weight vector exists that correctly classi es a set of patterns, then

the half-spaces de ned by the correct responses for these patterns will have
a non-empty intersection, called the solution region. The solution region
will be a \hyper-wedge" region whose vertex is at the origin of weight space
and whose cross-section increases with increasing distance from the origin.
This region is shown shaded in Fig. 4.6. (The boxed numbers show, for
later purposes, the number of errors made by weight vectors in each of the
regions.) The xed-increment error-correction procedure changes a weight
vector by moving it normal to any pattern hyperplane for which that weight
vector gives an incorrect response. Suppose in our example that we present
the patterns in the sequence Y1 , Y2 , Y3 , Y4 , and start the process with a
weight point V1, as shown in Fig. 4.7. Starting at V1 , we see that it gives
an incorrect response for pattern Y1, so we move V1 to V2 in a direction
normal to plane 1. (That is what adding Y1 to V1 does.) Y2 gives an
incorrect response for pattern Y2, and so on. Ultimately, the responses
are only incorrect for planes bounding the solution region. Some of the
subsequent corrections may overshoot the solution region, but eventually
we work our way out far enough in the solution region that corrections (for
a xed increment size) take us within it. The proofs for convergence of the
xed-increment rule make this intuitive argument precise.
4.1.5 The Widrow-Ho Procedure

The Widrow-Ho procedure (also called the LMS or the delta procedure)
attempts to nd weights that minimize a squared-error function between the
3
2 2
3 1
1
0
4
V
4
3
1
2
Figure 4.6: Solution Region in Weight Space
pattern labels and the dot product computed by a TLU. For this purpose,
the pattern labels are assumed to be either +1 or ;1 (instead of 1 or 0).
The squared error for a pattern, Xi , with label di (for desired output) is:
nX
+1
"i = (di ; xij wj )2
j =1
where xij is the j-th component of Xi . The total squared error (over all
patterns in a training set, , containing m patterns) is then:
m
X nX
+1
"= (di ; xij wj )2
i=1 j =1
We want to choose the weights wj to minimize this squared error. One
way to nd such a set of weights is to start with an arbitrary weight vector
and move it along the negative gradient of " as a function of the weights.
Since " is quadratic in the wj , we know that it has a global minimum, and
thus this steepest descent procedure is guaranteed to nd the minimum.
Each component of the gradient is the partial derivative of " with respect
to one of the weights. One problem with taking the partial derivative of "
is that " depends on all the input vectors in . Often, it is preferable to use
an incremental procedure in which we try the TLU on just one element, Xi,
3
2
V1
V5
V2 V6 V
V3 V4
4
Figure 4.7: Moving Into the Solution Region
of at a time, compute the gradient of the single-pattern squared error,

"i , make the appropriate adjustment to the weights, and then try another
member of . Of course, the results of the incremental version can only
approximate those of the batch one, but the approximation is usually quite
eective. We will be describing the incremental version here.
The j-th component of the gradient of the single-pattern error is:
@"i = ;2(d ; nX +1
xij wj )xij
@wj i
j =1
An adjustment in the direction of the negative gradient would then change
each weight as follows:
wj ; wj + ci (di ; fi )xij
P
where fi = nj =1+1 x w , and c governs the size of the adjustment. The en-
ij j i
tire weight vector (in augmented, or V, notation) is thus adjusted according
to the following rule:
V ; V + ci(di ; fi )Yi
where, as before, Yi is the i-th augmented pattern vector.
The Widrow-Ho procedure makes adjustments to the weight vector
whenever the dot product itself, Y V, does not equal the speci ed desired
i
target value, di (which is either 1 or ;1). The learning-rate factor, ci,

might decrease with time toward 0 to achieve asymptotic convergence. The
Widrow-Ho formula for changing the weight vector has the same form as
the standard xed-increment error-correction formula. The only dierence
is that fi is the thresholded response of the TLU in the error-correction
case while it is the dot product itself for the Widrow-Ho procedure.
Finding weight values that give the desired dot products corresponds
to solving a set of linear equalities, and the Widrow-Ho procedure can
be interpreted as a descent procedure that attempts to minimize the
mean-squared-error between the actual and desired values of the dot
product. (For more on Widrow-Ho and other related procedures, see
Duda & Hart, 1973, pp. 151].) Examples of
training curves
for TLU's
4.1.6 Training a TLU on Non-Linearly-Separable Train- performance on
training set
ing Sets performance on
test set
cumulative
When the training set is not linearly separable (perhaps because of noise number of
or perhaps inherently), it may still be desired to nd a \best" separating corrections.
hyperplane. Typically, the error-correction procedures will not do well on
non-linearly-separable training sets because they will continue to attempt
to correct inevitable errors, and the hyperplane will never settle into an
acceptable place.
Several methods have been proposed to deal with this case. First, we
might use the Widrow-Ho procedure, which (although it will not converge
to zero error on non-linearly separable problems) will give us a weight vec-
tor that minimizes the mean-squared-error. A mean-squared-error criterion
often gives unsatisfactory results, however, because it prefers many small
errors to a few large ones. As an alternative, error correction with a con-
tinuous decrease toward zero of the value of the learning rate constant, c,
will result in ever decreasing changes to the hyperplane. Duda Duda, 1966]
has suggested keeping track of the average value of the weight vector dur-
ing error correction and using this average to give a separating hyperplane
that performs reasonably well on non-linearly-separable problems. Gal-
lant Gallant, 1986] proposed what he called the \pocket algorithm." As
described in Hertz, Krogh, & Palmer, 1991, p. 160]:
. . . the pocket algorithm . . . consists simply in storing
(or \putting in your pocket") the set of weights which has had
the longest unmodi ed run of successes so far. The algorithm
is stopped after some chosen time t . . .
After stopping, the weights in the pocket are used as a set that should give
a small number of errors on the training set. Error-correction proceeds as
Also see usual with the ordinary set of weights.
methods
proposed by
John, 1995]
and by 4.2 Linear Machines
Marchand & Golea, 1993].
The latter is
outperform the The natural generalization of a (two-category) TLU to an R-category clas-
claimed to
pocket si er is the structure, shown in Fig. 4.8, called a linear machine. Here,
algorithm. to use more familiar notation, the Ws and X are meant to be augmented
vectors (with an (n+1)-st component). Such a structure is also sometimes
called a \competitive" net or a \winner-take-all" net. The output of the
linear machine is one of the numbers, f1 : : : Rg, corresponding to which
dot product is largest. Note that when R = 2, the linear machine reduces
to a TLU with weight vector W = (W1 ; W2 ).
W1.X
W1 Σ
X ... ARGMAX
WR Σ
WR.X
Figure 4.8: A Linear Machine

The diagram in Fig. 4.9 shows the character of the regions in a 2-
dimensional space created by a linear machine for R = 5. In n dimensions,
every pair of regions is either separated by a section of a hyperplane or is
non-adjacent.
To train a linear machine, there is a straightforward generalization of
the 2-category error-correction rule. Assemble the patterns in the training
set into a sequence as before.
1. If the machine classi es a pattern correctly, no change is made to any
of the weight vectors.
2. If the machine mistakenly classi es a category u pattern, Xi , in cat-
egory v (u 6= v), then:
Wu ; Wu + ciXi
4.3. NETWORKS OF TLUS 51
R1
R2 R3
R4
R5
In this region:
X.W4 ≥ X.Wi for i ≠ 4
Figure 4.9: Regions For a Linear Machine
and
Wv ; Wv ; ci Xi
and all other weight vectors are not changed.
This correction increases the value of the u-th dot product and decreases
the value of the v-th dot product. Just as in the 2-category xed increment
procedure, this procedure is guaranteed to terminate, for constant ci , if
there exists weight vectors that make correct separations of the training
set. Note that when R = 2, this procedure reduces to the ordinary TLU
error-correction procedure. A proof that this procedure terminates is given
in Nilsson, 1990, pp. 88-90] and in Duda & Hart, 1973, pp. 174-177].
4.3 Networks of TLUs

4.3.1 Motivation and Examples
Layered Networks
To classify correctly all of the patterns in non-linearly-separable training
sets requires separating surfaces more complex than hyperplanes. One way
to achieve more complex surfaces is with networks of TLUs. Consider, for
example, the 2-dimensional, even parity function, f = x1 x2 + x1 x2 . No
single line through the 2-dimensional square can separate the vertices (1,1)
and (0,0) from the vertices (1,0) and (0,1)|the function is not linearly sep-
arable and thus cannot be implemented by a single TLU. But, the network
of three TLUs shown in Fig. 4.10 does implement this function. In the
gure, we show the weight values along input lines to each TLU and the
threshold value inside the circle representing the TLU.
x1 1 1.5 1
-1 1
0.5 f
x2 -0.5 1
-1
Figure 4.10: A Network for the Even Parity Function

The function implemented by a network of TLUs depends on its topol-
ogy as well as on the weights of the individual TLUs. Feedforward networks
have no cycles in a feedforward network no TLU's input depends (through
zero or more intermediate TLUs) on that TLU's output. (Networks that
are not feedforward are called recurrent networks). If the TLUs of a feedfor-
ward network are arranged in layers, with the elements of layer j receiving
inputs only from TLUs in layer j ; 1, then we say that the network is a
layered, feedforward network. The network shown in Fig. 4.10 is a layered,
feedforward network having two layers (of weights). (Some people count the
layers of TLUs and include the inputs as a layer also they would call this
network a three-layer network.) In general, a feedforward, layered network
has the structure shown in Fig. 4.11. All of the TLUs except the \output"
units are called hidden units (they are \hidden" from the output).
Implementing DNF Functions by Two-Layer Networks
We have already de ned k-term DNF functions|they are DNF functions
having k terms. A k-term DNF function can be implemented by a two-layer
network with k units in the hidden layer|to implement the k terms|and
one output unit to implement the disjunction of these terms. Since any
Boolean function has a DNF form, any Boolean function can be imple-
mented by some two-layer network of TLUs. As an example, consider the
output units
hidden units
Figure 4.11: A Layered, Feedforward Network
function f = x1 x2 +x2x3 +x1x3 . The form of the network that implements

this function is shown in Fig. 4.12. (We leave it to the reader to calculate
appropriate values of weights and thresholds.) The 3-cube representation
of the function is shown in Fig. 4.13. The network of Fig. 4.12 can be
designed so that each hidden unit implements one of the planar boundaries
shown in Fig. 4.13.
TLUs
x
disjunct
disjunction
of terms
conjunctions
conjuncts
of literals
(terms)
A Feedforward, 2-layer Network
Figure 4.12: A Two-Layer Network

To train a two-layer network that implements a k-term DNF function,
we rst note that the output unit implements a disjunction, so the weights
in the nal layer are xed. The weights in the rst layer (except for the
\threshold weights") can all have values of 1, ;1, or 0. Later, we will
f = x1x2 + x2x3 + x1x3

x3
x2
x1
Figure 4.13: Three Planes Implemented by the Hidden Units
Discuss present a training procedure for this rst layer of weights.

half-space
intersections,
half-space Important Comment About Layered Networks
unions,
NP-hardness of
optimal
versions, Adding additional layers cannot compensate for an inadequate rst layer of
single-side- TLUs. The rst layer of TLUs partitions the feature space so that no two
error-
hypeplane dierently labeled vectors are in the same region (that is, so that no two
methods, such vectors yield the same set of outputs of the rst-layer units). If the
relation to
\AQ" methods. rst layer does not partition the feature space in this way, then regardless
of what subsequent layers do, the nal outputs will not be consistent with
Add diagrams the labeled training set.
showing the
non-linear
transformation
performed by a
layered
4.3.2 Madalines
network. Two-Category Networks
An interesting example of a layered, feedforward network is the two-layer
one which has an odd number of hidden units, and a \vote-taking" TLU
as the output unit. Such a network was called a \Madaline" (for many
adalines by Widrow. Typically, the response of the vote taking unit is
de ned to be the response of the majority of the hidden units, although
other output logics are possible. Ridgway Ridgway, 1962] proposed the
following error-correction rule for adjusting the weights of the hidden units
of a Madaline:
If the Madaline correctly classi es a pattern, Xi , no corrections are
made to any of the hidden units' weight vectors,
If the Madaline incorrectly classi es a pattern, Xi , then determine
the minimum number of hidden units whose responses need to be
changed (from 0 to 1 or from 1 to 0|depending on the type of error)
in order that the Madaline would correctly classify Xi . Suppose that
minimum number is ki. Of those hidden units voting incorrectly,
change the weight vectors of those ki of them whose dot products are
closest to 0 by using the error correction rule:
W ; W + ci (di ; fi)Xi
where di is the desired response of the hidden unit (0 or 1) and fi
is the actual response (0 or 1). (We assume augmented vectors here
even though we are using X, W notation.)
That is, we perform error-correction on just enough hidden units to
correct the vote to a majority voting correctly, and we change those that
are easiest to change. There are example problems in which even though
a set of weight values exists for a given Madaline structure such that it
could classify all members of a training set correctly, this procedure will
fail to nd them. Nevertheless, the procedure works eectively in most
experiments with it.
We leave it to the reader to think about how this training procedure
could be modi ed if the output TLU implemented an or function (or an
and function).
R-Category Madalines and Error-Correcting Output Codes

If there are k hidden units (k > 1) in a two-layer network, their responses
correspond to vertices of a k-dimensional hypercube. The ordinary two-
category Madaline identi es two special points in this space, namely the
vertex consisting of k 1's and the vertex consisting of k 0's. The Madaline's
response is 1 if the point in \hidden-unit-space" is closer to the all 1's vertex
than it is to the all 0's vertex. We could design an R-category Madaline by
identifying R vertices in hidden-unit space and then classifying a pattern
according to which of these vertices the hidden-unit response is closest to.
A machine using that idea was implemented in the early 1960s at SRI
Brain, et al., 1962]. It used the fact that the 2p so-called maximal-length
shift-register sequences Peterson, 1961, pp. 147] in a (2p ; 1)-dimensional
Boolean space are mutually equidistant (for any integer p). For similar,
more recent work see Dietterich & Bakiri, 1991].
4.3.3 Piecewise Linear Machines

A two-category training set is linearly separable if there exists a threshold
function that correctly classi es all members of the training set. Similarly,
we can say that an R-category training set is linearly separable if there
exists a linear machine that correctly classi es all members of the training
set. When an R-category problem is not linearly separable, we need a more
powerful classi er. A candidate is a structure called a piecewise linear
(PWL) machine illustrated in Fig. 4.14.
W1
1
Σ
1
... MAX
Σ
W1
N1
ARG
X ... MAX
WR
1
Σ
... R
MAX
Σ
WR
NR
Figure 4.14: A Piecewise Linear Machine

The PWL machine groups its weighted summing units into R banks
corresponding to the R categories. An input vector X is assigned to that
category corresponding to the bank with the largest weighted sum. We can
use an error-correction training algorithm similar to that used for a linear
machine. If a pattern is classi ed incorrectly, we subtract (a constant times)
the pattern vector from the weight vector producing the largest dot product
(it was incorrectly the largest) and add (a constant times) the pattern
vector to that weight vector in the correct bank of weight vectors whose dot
product is locally largest in that bank. (Again, we use augmented vectors
here.) Unfortunately, there are example training sets that are separable by
a given PWL machine structure but for which this error-correction training
method fails to nd a solution. The method does appear to work well in
some situations Duda & Fossum, 1966], although Nilsson, 1965, page 89]
observed that \it is probably not a very eective method for training PWL
machines having more than three weight vectors] in each bank."
4.3.4 Cascade Networks
Another interesting class of feedforward networks is that in which all of the

TLUs are ordered and each TLU receives inputs from all of the pattern
components and from all TLUs lower in the ordering. Such a network is
called a cascade network. An example is shown in Fig. 4.15 in which the
TLUs are labeled by the linearly separable functions (of their inputs) that
they implement. Each TLU in the network implements a set of 2k parallel
hyperplanes, where k is the number of TLUs from which it receives inputs.
(Each of the k preceding TLUs can have an output of 1 or 0 that's 2k
dierent combinations|resulting in 2k dierent positions for the parallel
hyperplanes.) We show a 3-dimensional sketch for a network of two TLUs
in Fig. 4.16. The reader might consider how the n-dimensional parity
function might be implemented by a cascade network having log2 n TLUs.
Cascade networks might be trained by rst training L1 to do as good a

job as possible at separating all the training patterns (perhaps by using the
pocket algorithm, for example), then training L2 (including the weight from
L1 to L2 ) also to do as good a job as possible at separating all the training
patterns, and so on until the resulting network classi es the patterns in the
training set satisfactorily. Also mention
the \cascade-
correlation"
Nils J. Nilsson. All rights reserved. method of
Fahlman & Lebiere, 1990].
L1
L2
output
L3
Figure 4.15: A Cascade Network
L2
L1
L2
Figure 4.16: Planes Implemented by a Cascade Network with Two TLUs
4.4 Training Feedforward Networks by Back-

propagation
4.4.1 Notation
The general problem of training a network of TLUs is dicult. Consider,
for example, the layered, feedforward network of Fig. 4.11. If such a net-
work makes an error on a pattern, there are usually several dierent ways
in which the error can be corrected. It is dicult to assign \blame" for
the error to any particular TLU in the network. Intuitively, one looks for
weight-adjusting procedures that move the network in the correct direc-
tion (relative to the error) by making minimal changes. In this spirit, the
Widrow-Ho method of gradient descent has been generalized to deal with
multilayer networks.
4.4. TRAINING FEEDFORWARD NETWORKS BY BACKPROPAGATION59
In explaining this generalization, we use Fig. 4.17 to introduce some
notation. This network has only one output unit, but, of course, it is
possible to have several TLUs in the output layer|each implementing a
dierent function. Each of the layers of TLUs will have outputs that we take
to be the components of vectors, just as the input features are components
of an input vector. The j-th layer of TLUs (1 j < k) will have as their
outputs the vector X(j ) . The input feature vector is denoted by X(0) , and
the nal output (of the k-th layer TLU) is f. Each TLU in each layer has
a weight vector (connecting it to its inputs) and a threshold the i-th TLU
in the j-th layer has a weight vector denoted by Wi(j ). (We will assume
that the \threshold weight" is the last component of the associated weight
vector we might have used V notation instead to include this threshold
component, but we have chosen here to use the familiar X,W notation,
assuming that these vectors are \augmented" as appropriate.) We denote
the weighted sum input to the i-th threshold unit in the j-th layer by s(ij ).
(That is, s( ) = X( ;1) Wi(j).) The number of TLUs in the j-th layer is
i
j j
given by mj . The vector Wi(j ) has components wl(ji) for l = 1 m(j ;1) + 1.
First Layer j-th Layer (k-1)-th Layer k-th Layer
X(0) X(1) X(j) X(k-1)
W(k)
... ... ...
... Wi(1)
f
Wi(j) Wi(k-1) wl(k)
... wli(j) s(k)

si(1) s (j) s (k-1)
... . .i . . .i .
m1 TLUs mj TLUs m(k-1) TLUs
Figure 4.17: A k-layer Network

4.4.2 The Backpropagation Method
A gradient descent method, similar to that used in the Widrow Ho method,
has been proposed by various authors for training a multi-layer, feedforward
network. As before, we de ne an error function on the nal output of the
network and we adjust each weight in the network so as to minimize the
error. If we have a desired response, di , for the i-th input vector, Xi , in the
training set, , we can compute the squared error over the entire training
set to be:
X
"= (di ; fi )2
Xi
where fi is the actual response of the network for input Xi . To do gradient
descent on this squared error, we adjust each weight in the network by
an amount proportional to the negative of the partial derivative of " with
respect to that weight. Again, we use a single-pattern error function so
that we can use an incremental weight adjustment procedure. The squared
error for a single input vector, X, evoking an output of f when the desired
output is d is:
" = (d ; f)2
It is convenient to take the partial derivatives of " with respect to the
various weights in groups corresponding to the weight vectors. We de ne
a partial derivative of a quantity , say, with respect to a weight vector,
Wi(j), thus:
" #
@ def @ : : : @ : : : @
@ Wi(j ) = @w1(ji) @wli(j ) @wm(jj);1 +1 i
where wli(j ) is the l-th component of Wi(j ). This vector partial derivative of
is called the gradient of with respect to W and is sometimes denoted
by rW .
Since "'s dependence on Wi(j ) is entirely through s(ij ) , we can use the
chain rule to write:
@" = @" @s(ij )
@ Wi(j ) @s(ij ) @ Wi(j )
(j )
Because s( ) = X( ;1) Wi(j), @@s
i
j j
W(j) = X . Substituting yields:
i (j ;1)
i
@" = @" X(j ;1)

@ Wi(j ) @s(ij )
Note that @s@"(j) = ;2(d ; f) @s@f(j) . Thus,
i i
@" = ;2(d ; f) @f X(j ;1)

@ Wi(j ) @s(ij )
The quantity (d ; f) @s@f(j) plays an important role in our calculations we
i
shall denote it by
i(j ) . Each of the
i(j ) 's tells us how sensitive the squared
error of the network output is to changes in the input to each threshold
function. Since we will be changing weight vectors in directions along their
negative gradient, our fundamental rule for weight changes throughout the
network will be:
Wi(j ) Wi(j ) + c(ij )
i(j) X(j;1)
where c(ij ) is the learning rate constant for this weight vector. (Usually, the
learning rate constants for all weight vectors in the network are the same.)
We see that this rule is quite similar to that used in the error correction
procedure for a single TLU. A weight vector is changed by the addition of
a constant times its vector of (unweighted) inputs.
Now, we must turn our attention to the calculation of the
i(j )'s. Using
the de nition, we have:

i(j ) = (d ; f) @f(j )
@si
We have a problem, however, in attempting to carry out the partial deriva-
tives of f with respect to the s's. The network output, f, is not continuously
dierentiable with respect to the s's because of the presence of the thresh-
old functions. Most small changes in these sums do not change f at all,
and when f does change, it changes abruptly from 1 to 0 or vice versa.
A way around this diculty was proposed by Werbos Werbos, 1974]
and (perhaps independently) pursued by several other researchers, for ex-
ample Rumelhart, Hinton, & Williams, 1986]. The trick involves replacing
all the threshold functions by dierentiable functions called sigmoids.1 The
output of a sigmoid function, superimposed on that of a threshold function,
is shown in Fig. 4.18. Usually, the sigmoid function used is f(s) = 1+1e;s ,
where s is the input and f is the output.
f (s)
threshold function
sigmoid
f (s) = 1/[1 + e−s]
s
Figure 4.18: A Sigmoid Function
We show the network containing sigmoid units in place of TLUs in Fig.
4.19. The output of the i-th sigmoid unit in the j-th layer is denoted by
fi(j ) . (That is, fi(j ) = ;1 s(j) .)
1+e i
4.4.3 Computing Weight Changes in the Final Layer

We rst calculate
(k) in order to compute the weight change for the nal
sigmoid unit:
1 Russell & Norvig 1995, page 595] attributes the use of this idea to
Bryson & Ho 1969].

First Layer j-th Layer (k-1)-th Layer k-th Layer
X(0) X(1) X(j) X(k-1)
W(k)
... ... ...
f(k)
... Wi(1) Wi(k-1) (k-1) w (k)
fi(1) Wi(j) fi(j) l
fi
δ(k)
... δi(1) w (j) δi(j) s(k)
li δi(k-1)
si(1) s (j) s (k-1)
... . .i . . .i .
m1 sigmoids mj sigmoids m(k-1) sigmoids
Figure 4.19: A Network with Sigmoid Units
(k)

(k) = (d ; f (k) ) @f
@s(k)
Given the sigmoid function that we are using, namely f(s) = 1
1+e;s , we
have that @f
@s = f(1 ; f). Substituting gives us:

(k) = (d ; f (k) )f (k) (1 ; f (k) )
Rewriting our general rule for weight vector changes, the weight vector
in the nal layer is changed according to the rule:
W(k) W(k) + c(k)
(k)X(k;1)
where
(k) = (d ; f (k) )f (k) (1 ; f (k) )
It is interesting to compare backpropagation to the error-correction rule
and to the Widrow-Ho rule. The backpropagation weight adjustment for
the single element in the nal layer can be written as:
W ; W + c(d ; f)f(1 ; f)X
Written in the same format, the error-correction rule is:
W ; W + c(d ; f)X
and the Widrow-Ho rule is:
W ; W + c(d ; f)X
The only dierence (except for the fact that f is not thresholded in Widrow-
Ho) is the f(1 ; f) term due to the presence of the sigmoid function.
With the sigmoid function, f(1 ; f) can vary in value from 0 to 1. When
f is 0, f(1 ; f) is also 0 when f is 1, f(1 ; f) is 0 f(1 ; f) obtains
its maximum value of 1/4 when f is 1/2 (that is, when the input to the
sigmoid is 0). The sigmoid function can be thought of as implementing a
\fuzzy" hyperplane. For a pattern far away from this fuzzy hyperplane,
f(1 ; f) has value close to 0, and the backpropagation rule makes little
or no change to the weight values regardless of the desired output. (Small
changes in the weights will have little eect on the output for inputs far
from the hyperplane.) Weight changes are only made within the region of
\fuzz" surrounding the hyperplane, and these changes are in the direction of
correcting the error, just as in the error-correction and Widrow-Ho rules.
4.4.4 Computing Changes to the Weights in Interme-

diate Layers
Using our expression for the
's, we can similarly compute how to change
each of the weight vectors in the network. Recall:

i(j ) = (d ; f) @f(j )
@si
Again we use a chain rule. The nal output, f, depends on s(ij ) through
each of the summed inputs to the sigmoids in the (j + 1)-th layer. So:

i(j ) = (d ; f) @f(j )
@si
" #
= (d ; f) @f @s(1j +1) + + @f @s(l j +1) + + @f @s(mj +1)
j +1
(
@s1 j +1) @si ( j ) (
@sl j +1) (
@si j ) @smj+1 @sij )
( j +1) (

mX
=
j +1
(d ; f) @f @s(l j +1) = mXj +1

(j +1)
(j +1) @sl
l
l=1 @s(l j +1) @s(ij ) l=1 @s(ij )
(j+1)
It remains to compute the @s@sl (j) 's. To do that we rst write:
i
sl
(j +1)
= X( ) Wl(j+1)
j
mXj +1
= f(j ) wl(j +1)
=1
And then, since the weights do not depend on the s's:
hP
@ mj +1 f (j ) w(j +1) i mXj +1
@s(l j +1) =
=1 l
=
(j )
wl(j +1) @f(j )
@s(j )
i @s(ij ) =1 @si
(j) @f (j) = f (j ) (1 ;
Now, we note that @f
@si( j ) = 0 unless = i, in which case @s(j)
( j )
f ). Therefore:
@s(l j +1) = w(j +1)f (j ) (1 ; f (j ) )
il i i
@s(ij )
We use this result in our expression for
i(j ) to give:
mXj +1

i(j ) = fi(j ) (1 ; fi(j ) )
l(j +1)wil(j +1)
l=1
The above equation is recursive in the
's. (It is interesting to note that
this expression is independent of the error function the error function ex-
plicitly aects only the computation of
(k).) Having computed the
i(j +1)'s
for layer j + 1, we can use this equation to compute the
i(j ) 's. The base
case is
(k), which we have already computed:

(k) = (d ; f (k) )f (k) (1 ; f (k) )
We use this expression for the
's in our generic weight changing rule,
namely:
Wi(j ) Wi(j) + c(ij)
i(j ) X(j ;1)
Although this rule appears complex, it can be simply implemented
by \backpropagating" the
's through the weights in order to adjust the
amount by which X vectors are added to or subtracted from weight vectors
Add additional (thus, the name backprop for this algorithm).
intuitive
explanation.
4.4.5 Variations on Backprop
To be written: problem of local minima, simulated annealing, momem-
tum (Plaut, et al., 1986, see Hertz, Krogh, & Palmer, 1991]), quickprop,
regularization methods]
Simulated Annealing
To apply simulated annealing, the value of the learning rate constant is
gradually decreased with time. If we fall early into an error-function valley
that is not very deep (a local minimum), it typically will neither be very
broad, and soon a subsequent large correction will jostle us out of it. It
is less likely that we will move out of deep valleys, and at the end of the
process (with very small values of the learning rate constant), we descend
to its deepest point. The process gets its name by analogy with annealing
in metallurgy, in which a material's temperature is gradually decreased
allowing its crystalline structure to reach a minimal energy state.
4.4.6 An Application: Steering a Van

A neural network system called ALVINN (Autonomous Land Vehicle in
a Neural Network) has been trained to steer a Chevy van successfully
on ordinary roads and highways at speeds of 55 mph Pomerleau, 1991,
Pomerleau, 1993]. The input to the network is derived from a low-resolution
(30 x 32) television image. The TV camera is mounted on the van and looks
at the road straight ahead. This image is sampled and produces a stream
of 960-dimensional input vectors to the neural network. The network is
shown in Fig. 4.20.
The network has ve hidden units in its rst layer and 30 output units
in the second layer all are sigmoid units. The output units are arranged in
sharp left
centroid
960 inputs of outputs
30 x 32 retina ... steers
vehicle
straight ahead
...
5 hidden
units connected
to all 960 inputs
sharp right
30 output units
connected to all
hidden units
Figure 4.20: The ALVINN Network
a linear order and control the van's steering angle. If a unit near the top of
the array of output units has a higher output than most of the other units,
the van is steered to the left if a unit near the bottom of the array has a
high output, the van is steered to the right. The \centroid" of the responses
of all of the output units is computed, and the van's steering angle is set
at a corresponding value between hard left and hard right.
The system is trained by a modi ed on-line training regime. A driver
drives the van, and his actual steering angles are taken as the correct labels
for the corresponding inputs. The network is trained incrementally by
backprop to produce the driver-speci ed steering angles in response to each
visual pattern as it occurs in real time while driving.
This simple procedure has been augmented to avoid two potential prob-
lems. First, since the driver is usually driving well, the network would never
get any experience with far-from-center vehicle positions and/or incorrect
vehicle orientations. Also, on long, straight stretches of road, the network
would be trained for a long time only to produce straight-ahead steering
angles this training would swamp out earlier training to follow a curved
road. We wouldn't want to try to avoid these problems by instructing the
driver to drive erratically occasionally, because the system would learn to
mimic this erratic behavior.
Instead, each original image is shifted and rotated in software to create
14 additional images in which the vehicle appears to be situated dierently
relative to the road. Using a model that tells the system what steering angle
ought to be used for each of these shifted images, given the driver-speci ed
steering angle for the original image, the system constructs an additional
14 labeled training patterns to add to those encountered during ordinary
driver training.
4.5 Synergies Between Neural Network and

Knowledge-Based Methods
To be written
discuss
rule-generating
procedures
(such as
Towell & Shavlik, 1992])
and how
expert-
provided rules
can aid neural
net training
and vice-versa
Towell, Shavlik, & Noordweier, 1990].
To be added.

Chapter 6
Decision Trees
6.1 Denitions
A decision tree (generally dened) is a tree whose internal nodes are tests
(on input patterns) and whose leaf nodes are categories (of patterns). We
show an example in Fig. 6.1. A decision tree assigns a class number (or
output) to an input pattern by ltering the pattern down through the tests
in the tree. Each test has mutually exclusive and exhaustive outcomes. For
example, test T2 in the tree of Fig. 6.1 has three outcomes the left-most
one assigns the input pattern to class 3, the middle one sends the input
pattern down to test T4 , and the right-most one assigns the pattern to
class 1. We follow the usual convention of depicting the leaf nodes by the
class number.1 Note that in discussing decision trees we are not limited to
implementing Boolean functions|they are useful for general, categorically
valued functions.
There are several dimensions along which decision trees might dier:
1. The tests might be multivariate (testing on several features of the
input at once) or univariate (testing on only one of the features).
2. The tests might have two outcomes or more than two. (If all of the
tests have two outcomes, we have a binary decision tree.)
1 One of the researchers who has done a lot of work on learning decision trees is Ross
Quinlan. Quinlan distinguishes between classes and categories. He calls the subsets of
patterns that lter down to each tip categories and subsets of patterns having the same
label classes. In Quinlan's terminology, our example tree has nine categories and three
classes. We will not make this distinction, however, but will use the words \category"
and \class" interchangeably to refer to what Quinlan calls \class."
81
82 CHAPTER 6. DECISION TREES
T1
T2 T4 T3
3 T4
1 2 3
1
T4
2
1
3 2
Figure 6.1: A Decision Tree
3. The features or attributes might be categorical or numeric. (Binary-

valued ones can be regarded as either.)
4. We might have two classes or more than two. If we have two classes
and binary inputs, the tree implements a Boolean function, and is
called a Boolean decision tree.
It is straightforward to represent the function implemented by a uni-

variate Boolean decision tree in DNF form. The DNF form implemented
by such a tree can be obtained by tracing down each path leading to a tip
node corresponding to an output value of 1, forming the conjunction of the
tests along this path, and then taking the disjunction of these conjunctions.
We show an example in Fig. 6.2. In drawing univariate decision trees, each
non-leaf node is depicted by a single attribute. If the attribute has value 0
in the input pattern, we branch left if it has value 1, we branch right.
The k-DL class of Boolean functions can be implemented by a multi-
variate decision tree having the (highly unbalanced) form shown in Fig.
6.3. Each test, ci, is a term of size k or less. The vi all have values of 0 or
1.
6.2. SUPERVISED LEARNING OF UNIVARIATE DECISION TREES83
x3
0 1
x2 x4
0 0 1
1
0
0
x3x4 x1
x3x2
1 0 1
x3x2
1 0
x3x4x1 x3x4x1
f = x3x2 + x3x4x1
Figure 6.2: A Decision Tree Implementing a DNF Function
6.2 Supervised Learning of Univariate Deci-

sion Trees
Several systems for learning decision trees have been proposed. Promi-
nent among these are ID3 and its new version, C4.5 Quinlan, 1986,
Quinlan, 1993], and CART Breiman, et al., 1984] We discuss here only
batch methods, although incremental ones have also been proposed
Utgo, 1989].
6.2.1 Selecting the Type of Test

As usual, we have n features or attributes. If the attributes are binary, the
tests are simply whether the attribute's value is 0 or 1. If the attributes
are categorical, but non-binary, the tests might be formed by dividing the
attribute values into mutually exclusive and exhaustive subsets. A decision
tree with such tests is shown in Fig. 6.4. If the attributes are numeric, the
tests might involve \interval tests," for example 7 xi 13.2.
cq
vn
cq-1
vn-1
ci
vi
v1
Figure 6.3: A Decision Tree Implementing a Decision List
6.2.2 Using Uncertainty Reduction to Select Tests

The main problem in learning decision trees for the binary-attribute case
is selecting the order of the tests. For categorical and numeric attributes,
we must also decide what the tests should be (besides selecting the order).
Several techniques have been tried the most popular one is at each stage
to select that test that maximally reduces an entropy-like measure.
We show how this technique works for the simple case of tests with
binary outcomes. Extension to multiple-outcome tests is straightforward
computationally but gives poor results because entropy is always decreased
by having more outcomes.
The entropy or uncertainty still remaining about the class of a pattern|
knowing that it is in some set, , of patterns is dened as:
H() = ;
X p(ij) log p(ij)2
i
where p(ij) is the probability that a pattern drawn at random from
belongs to class i, and the summation is over all of the classes. We want to
select tests at each node such that as we travel down the decision tree, the
uncertainty about the class of a pattern becomes less and less.
x3 = a, b, c, or d
{a, c} {d}
{b}
2
x1 = e, b, or d x4 = a, e, f, or g
{e,b}
{d}
{a, g} {e, f}
1
1 2
x2 = a, or g
{a} {g}
2 1
Figure 6.4: A Decision Tree with Categorical Attributes
Since we do not in general have the probabilities p(ij), we estimate

them by sample statistics. Although these estimates might be errorful,
they are nevertheless useful in estimating uncertainties. Let p^(ij) be the
number of patterns in belonging to class i divided by the total number
of patterns in . Then an estimate of the uncertainty is:
^ =;
H()
X p^(ij) log p^(ij)
2
i
For simplicity, from now on we'll drop the \hats" and use sample statistics
as if they were real probabilities.
If we perform a test, T, having k possible outcomes on the patterns in
, we will create k subsets, 1 2 : : : k . Suppose that ni of the patterns
in are in i for i = 1 ::: k. (Some ni may be 0.) If we knew that T
applied to a pattern in resulted in the j-th outcome (that is, we knew
that the pattern was in j ), the uncertainty about its class would be:
H(j ) = ;
X p(ijj) log p(ijj)
2
i
and the reduction in uncertainty (beyond knowing only that the pattern
was in ) would be:
H() ; H(j )
Of course we cannot say that the test T is guaranteed always to produce
that amount of reduction in uncertainty because we don't know that the
result of the test will be the j-th outcome. But we can estimate the average
uncertainty over all the j , by:
EHT ()] =
X p(j)H(j)
j
where by HT () we mean the average uncertainty after performing test T
on the patterns in , p(j ) is the probability that the test has outcome j,
and the sum is taken from 1 to k. Again, we don't know the probabilities
p(j ), but we can use sample values. The estimate p^(j ) of p(j ) is just
the number of those patterns in that have outcome j divided by the total
number of patterns in . The average reduction in uncertainty achieved by
test T (applied to patterns in ) is then:
RT () = H() ; EHT ()]
An important family of decision tree learning algorithms selects for the
root of the tree that test that gives maximum reduction of uncertainty, and
then applies this criterion recursively until some termination condition is
met (which we shall discuss in more detail later). The uncertainty calcu-
lations are particularly simple when the tests have binary outcomes and
when the attributes have binary values. We'll give a simple example to
illustrate how the test selection mechanism works in that case.
Suppose we want to use the uncertainty-reduction method to build a
decision tree to classify the following patterns:
pattern class
(0, 0, 0) 0
(0, 0, 1) 0
(0, 1, 0) 0
(0, 1, 1) 0
(1, 0, 0) 0
(1, 0, 1) 1
(1, 1, 0) 0
(1, 1, 1) 1
x3
x2
The test x1
x1
Figure 6.5: Eight Patterns to be Classied by a Decision Tree
What single test, x1, x2, or x3, should be performed rst? The illustration
in Fig. 6.5 gives geometric intuition about the problem.
The initial uncertainty for the set, , containing all eight points is:
H() = ;(6=8) log2(6=8) ; (2=8) log2(2=8) = 0:81
Next, we calculate the uncertainty reduction if we perform x1 rst. The
left-hand branch has only patterns belonging to class 0 (we call them the
set l ), and the right-hand-branch (r ) has two patterns in each class. So,
the uncertainty of the left-hand branch is:
Hx1 (l ) = ;(4=4) log2 (4=4) ; (0=4) log2 (0=4) = 0
And the uncertainty of the right-hand branch is:
Hx1 (r ) = ;(2=4) log2 (2=4) ; (2=4) log2 (2=4) = 1
Half of the patterns \go left" and half \go right" on test x1. Thus, the
average uncertainty after performing the x1 test is:
1=2Hx1 (l ) + 1=2Hx1 (r ) = 0:5
Therefore the uncertainty reduction on achieved by x1 is:
Rx1 () = 0:81 ; 0:5 = 0:31

By similar calculations, we see that the test x3 achieves exactly the same
uncertainty reduction, but x2 achieves no reduction whatsoever. Thus, our
\greedy" algorithm for selecting a rst test would select either x1 or x3.
Suppose x1 is selected. The uncertainty-reduction procedure would select
x3 as the next test. The decision tree that this procedure creates thus
See implements the Boolean function: f = x1x3.
Quinlan, 1986,
sect. 4] for
another
example. 6.2.3 Non-Binary Attributes
If the attributes are non-binary, we can still use the uncertainty-reduction
technique to select tests. But now, in addition to selecting an attribute, we
must select a test on that attribute. Suppose for example that the value
of an attribute is a real number and that the test to be performed is to
set a threshold and to test to see if the number is greater than or less
than that threshold. In principle, given a set of labeled patterns, we can
measure the uncertainty reduction for each test that is achieved by every
possible threshold (there are only a nite number of thresholds that give
dierent test results if there are only a nite number of training patterns).
Similarly, if an attribute is categorical (with a nite number of categories),
there are only a nite number of mutually exclusive and exhaustive subsets
into which the values of the attribute can be split. We can calculate the
uncertainty reduction for each split.
6.3 Networks Equivalent to Decision Trees

Since univariate Boolean decision trees are implementations of DNF func-
tions, they are also equivalent to two-layer, feedforward neural networks.
We show an example in Fig. 6.6. The decision tree at the left of the gure
implements the same function as the network at the right of the gure. Of
course, when implemented as a network, all of the features are evaluated
in parallel for any input pattern, whereas when implemented as a decision
tree only those features on the branch traveled down by the input pattern
need to be evaluated. The decision-tree induction methods discussed in this
chapter can thus be thought of as particular ways to establish the structure
and the weight values for networks.
6.4. OVERFITTING AND EVALUATION 89
x3 f = x3x2 + x3x4x1
x3x4x1
0 1
x1 -1 1.5
x2 x4 +1
x2
0 0 1 +1
1 X x3 f
0
0 x1 disjunction
x3x4 x4
+1
0 1 -1
x3x2 1
x3x2 0.5
terms
1
0 x3x2
x3x4x1
x3x4x1
Figure 6.6: A Univariate Decision Tree and its Equivalent Network
Multivariate decision trees with linearly separable functions at each

node can also be implemented by feedforward networks|in this case three-
layer ones. We show an example in Fig. 6.7 in which the linearly separable
functions, each implemented by a TLU, are indicated by L1 L2 L3, and
L4 . Again, the nal layer has xed weights, but the weights in the rst two
layers must be trained. Dierent approaches to training procedures have
been discussed by Brent, 1990], by John, 1995], and (for a special case)
by Marchand & Golea, 1993].
6.4 Overtting and Evaluation

6.4.1 Overtting
In supervised learning, we must choose a function to t the training set from
among a set of hypotheses. We have already showed that generalization is
impossible without bias. When we know a priori that the function we are
trying to guess belongs to a small subset of all possible functions, then,
even with an incomplete set of training samples, it is possible to reduce the
subset of functions that are consistent with the training set suciently to
make useful guesses about the value of the function for inputs not in the
training set. And, the larger the training set, the more likely it is that even
a randomly selected consistent function will have appropriate outputs for
L1L2
L1
L1
−
0 1
L2
+
L2 L3
0 X
0 1 L3 + f
1 +
0
0 L4 L4 disjunction
1 0 1 −
1 L1 L3 L4
0
conjunctions
Figure 6.7: A Multivariate Decision Tree and its Equivalent Network
patterns not yet seen.

However, even with bias, if the training set is not suciently large com-
pared with the size of the hypothesis space, there will still be too many
consistent functions for us to make useful guesses, and generalization per-
formance will be poor. When there are too many hypotheses that are
consistent with the training set, we say that we are overtting the train-
ing data. Overtting is a problem that we must address for all learning
methods.
Since a decision tree of sucient size can implement any Boolean func-
tion there is a danger of overtting|especially if the training set is small.
That is, even if the decision tree is synthesized to classify all the members
of the training set correctly, it might perform poorly on new patterns that
were not used to build the decision tree. Several techniques have been pro-
posed to avoid overtting, and we shall examine some of them here. They
make use of methods for estimating how well a given decision tree might
generalize|methods we shall describe next.
6.4.2 Validation Methods

The most straightforward way to estimate how well a hypothesized function
(such as a decision tree) performs on a test set is to test it on the test
set! But, if we are comparing several learning systems (for example, if
we are comparing dierent decision trees) so that we can select the one
that performs the best on the test set, then such a comparison amounts to
\training on the test data." True, training on the test data enlarges the
training set, with a consequent expected improvement in generalization, but
there is still the danger of overtting if we are comparing several dierent
learning systems. Another technique is to split the training set|using (say)
two-thirds for training and the other third for estimating generalization
performance. But splitting reduces the size of the training set and thereby
increases the possibility of overtting. We next describe some validation
techniques that attempt to avoid these problems.
Cross-Validation
In cross-validation, we divide the training set into K mutually exclusive

and exhaustive equal-sized subsets: 1 : : : K . For each subset, i, train
on the union of all of the other subsets, and empirically determine the error
rate, "i , on i. (The error rate is the number of classication errors made
on i divided by the number of patterns in i.) An estimate of the error
rate that can be expected on new patterns of a classier trained on all the
patterns in is then the average of the "i .
Leave-one-out Validation
Leave-one-out validation is the same as cross validation for the special case
in which K equals the number of patterns in , and each i consists of a
single pattern. When testing on each i, we simply note whether or not
a mistake was made. We count the total number of mistakes and divide
by K to get the estimated error rate. This type of validation is, of course,
more expensive computationally, but useful when a more accurate estimate
of the error rate for a classier is needed. Describe
\bootstrap-
ping" also
6.4.3 Avoiding Overtting in Decision Trees Efron, 1982].
Near the tips of a decision tree there may be only a few patterns per node.
For these nodes, we are selecting a test based on a very small sample, and
thus we are likely to be overtting. This problem can be dealt with by ter-
minating the test-generating procedure before all patterns are perfectly split
into their separate categories. That is, a leaf node may contain patterns
of more than one class, but we can decide in favor of the most numerous
class. This procedure will result in a few errors but often accepting a small
number of errors on the training set results in fewer errors on a testing set.
This behavior is illustrated in Fig. 6.8.
Error Rate Iris Data Decision Tree

1.0
0.8
training errors
0.6
validation errors
0.4
0.2
0
0 1 2 3 4 5 6 7 8 9
Number of Terminal
Nodes
(From Weiss, S., and Kulikowski, C., Computer Systems that Learn,
Morgan Kaufmann, 1991)
Figure 6.8: Determining When Overtting Begins
One can use cross-validation techniques to determine when to stop split-

ting nodes. If the cross validation error increases as a consequence of a node
split, then don't split. One has to be careful about when to stop, though,
because undertting usually leads to more errors on test sets than does
overtting. There is a general rule that the lowest error-rate attainable by
a sub-tree of a fully expanded tree can be no less than 1/2 of the error rate
of the fully expanded tree Weiss & Kulikowski, 1991, page 126].
Rather than stopping the growth of a decision tree, one might grow
it to its full size and then prune away leaf nodes and their ances-
tors until cross-validation accuracy no longer increases. This technique
is called post-pruning. Various techniques for pruning are discussed in
Weiss & Kulikowski, 1991].
6.4.4 Minimum-Description Length Methods

An important tree-growing and pruning technique is based on the
minimum-description-length (MDL) principle. (MDL is an important idea
that extends beyond decision-tree methods Rissanen, 1978].) The idea is
that the simplest decision tree that can predict the classes of the training
patterns is the best one. Consider the problem of transmitting just the
labels of a training set of patternsassuming that the receiver of this infor-
mation already has the ordered set of patterns. If there are m patterns,
each labeled by one of R classes, one could transmit a list of m R-valued
numbers. Assuming equally probable classes, this transmission would re-
quire m log2 R bits. Or, one could transmit a decision tree that correctly
labelled all of the patterns. The number of bits that this transmission would
require depends on the technique for encoding decision trees and on the size
of the tree. If the tree is small and accurately classies all of the patterns, it
might be more economical to transmit the tree than to transmit the labels
directly. In between these extremes, we might transmit a tree plus a list of
labels of all the patterns that the tree misclassies.
In general, the number of bits (or description length of the binary en-
coded message) is t + d, where t is the length of the message required to
transmit the tree, and d is the length of the message required to transmit
the labels of the patterns misclassied by the tree. In a sense, that tree as-
sociated with the smallest value of t+d is the best or most economical tree.
The MDL method is one way of adhering to the Occam's razor principle.
Quinlan and Rivest Quinlan & Rivest, 1989] have proposed techniques
for encoding decision trees and lists of exception labels and for calculating
the description length (t + d) of these trees and labels. They then use the
description length as a measure of quality of a tree in two ways:
1. In growing a tree, they use the reduction in description length to

select tests (instead of reduction in uncertainty).
2. In pruning a tree after it has been grown to zero error, they prune
away those nodes (starting at the tips) that achieve a decrease in the
description length.
These techniques compare favorably with the uncertainty-reduction method,

although they are quite sensitive to the coding schemes used.
6.4.5 Noise in Data

Noise in the data means that one must inevitably accept some number
of errors|depending on the noise level. Refusal to tolerate errors on the
training set when there is noise leads to the problem of \tting the noise."
Dealing with noise, then, requires accepting some errors at the leaf nodes
just as does the fact that there are a small number of patterns at leaf nodes.
6.5 The Problem of Replicated Subtrees
Decision trees are not the most economical means of implementing some
Boolean functions. Consider, for example, the function f = x1x2 + x3x4.
A decision tree for this function is shown in Fig. 6.9. Notice the replicated
subtrees shown circled. The DNF-form equivalent to the function imple-
mented by this decision tree is f = x1 x2 + x1 x2x3x4 + x1x3 x4. This DNF
form is non-minimal (in the number of disjunctions) and is equivalent to
f = x1x2 + x3x4.
x1
x3 x2
x3 1
0
x4
0
x4
0 1
0 1
Figure 6.9: A Decision Tree with Subtree Replication

The need for replication means that it takes longer to learn the tree
and that subtrees replicated further down the tree must be learned using a
smaller training subset. This problem is sometimes called the fragmentation
problem.
Several approaches might be suggested for dealing with fragmenta-
tion. One is to attempt to build a decision graph instead of a tree
Oliver, Dowe, & Wallace, 1992, Kohavi, 1994]. A decision graph that im-
plements the same decisions as that of the decision tree of Fig. 6.9 is shown
in Fig. 6.10.
Another approach is to use multivariate (rather than univariate tests at
each node). In our example of learning f = x1 x2 + x3 x4, if we had a test
6.5. THE PROBLEM OF REPLICATED SUBTREES 95
x1
x2
x3 1
0
x4
0 1
Figure 6.10: A Decision Graph
for x1x2 and a test for x3x4, the decision tree could be much simplied,
as shown in Fig. 6.11. Several researchers have proposed techniques for
learning decision trees in which the tests at each node are linearly separable
functions. John, 1995] gives a nice overview (with several citations) of
learning such linear discriminant trees and presents a method based on
\soft entropy."
A third method for dealing with the replicated subtree problem involves
extracting propositional \rules" from the decision tree. The rules will have
as antecedents the conjunctions that lead down to the leaf nodes, and as
consequents the name of the class at the corresponding leaf node. An ex-
ample rule from the tree with the repeating subtree of our example would
be: x1 ^ :x2 ^ x3 ^ x4 1. Quinlan Quinlan, 1987] discusses methods
for reducing a set of rules to a simpler set by 1) eliminating from the an-
tecedent of each rule any \unnecessary" conjuncts, and then 2) eliminating
\unnecessary" rules. A conjunct or rule is determined to be unnecessary
if its elimination has little eect on classication accuracy|as determined
by a chi-square test, for example. After a rule set is processed, it might be
the case that more than one rule is \active" for any given pattern, and care
must be taken that the active rules do not con"ict in their decision about
the class of a pattern.
x1x2
1
x3x4
1
0
Figure 6.11: A Multivariate Decision Tree
6.6 The Problem of Missing Attributes

To be added.
6.7 Comparisons
Several experimenters have compared decision-tree, neural-net, and
nearest-neighbor classiers on a wide variety of problems. For a
comparison of neural nets versus decision trees, for example, see
Dietterich, et al., 1990, Shavlik, Mooney, & Towell, 1991, Quinlan, 1994].
In their StatLog project, Taylor, Michie, & Spiegalhalter, 1994] give thor-
ough comparisons of several machine learning algorithms on several dier-
ent types of problems. There seems to be no single type of classier that is
best for all problems. And, there do not seem to be any general conclusions
that would enable one to say which classier method is best for which sorts
of classication problems, although Quinlan, 1994] does provide some in-
tuition about properties of problems that might render them ill suited for
decision trees, on the one hand, or backpropagation, on the other.

To be added.

Chapter 7
Inductive Logic
Programming
There are many di erent representational forms for functions of input vari-
ables. So far, we have seen (Boolean) algebraic expressions, decision trees,
and neural networks, plus other computational mechanisms such as tech-
niques for computing nearest neighbors. Of course, the representation most
important in computer science is a computer program. For example, a Lisp
predicate of binary-valued inputs computes a Boolean function of those in-
puts. Similarly, a logic program (whose ordinary application is to compute
bindings for variables) can also be used simply to decide whether or not
a predicate has value True (T) or False (F). For example, the Boolean
exclusive-or (odd parity) function of two variables can be computed by the
following logic program:
Parity(x,y) :- True(x), : True(y)

:- True(y), : True(x)
We follow Prolog syntax (see, for example, Mueller & Page, 1988]), except
that our convention is to write variables as strings beginning with lower-
case letters and predicates as strings beginning with upper-case letters. The
unary function \True" returns T if and only if the value of its argument is
T . (We now think of Boolean functions and arguments as having values of
T and F instead of 0 and 1.) Programs will be written in \typewriter"
font.
97
98 CHAPTER 7. INDUCTIVE LOGIC PROGRAMMING
In this chapter, we consider the matter of learning logic programs given
a set of variable values for which the logic program should return T (the
positive instances) and a set of variable values for which it should return F
(the negative instances). The subspecialty of machine learning that deals
with learning logic programs is called inductive logic programming (ILP)
Lavrac & Dzeroski, 1994]. As with any learning problem, this one can be
quite complex and intractably dicult unless we constrain it with biases of
some sort. In ILP, there are a variety of possible biases (called language bi-
ases). One might restrict the program to Horn clauses, not allow recursion,
not allow functions, and so on.
As an example of an ILP problem, suppose we are trying to induce a
function Nonstop(x,y), that is to have value T for pairs of cities connected
by a non-stop air ight and F for all other pairs of cities. We are given a
training set consisting of positive and negative examples. As positive ex-
amples, we might have (A,B), (A, A1), and some other pairs as negative
examples, we might have (A1, A2), and some other pairs. In ILP, we usu-
ally have additional information about the examples, called \background
knowledge." In our air-ight problem, the background information might
be such ground facts as Hub(A), Hub(B), Satellite(A1,A), plus others.
(Hub(A) is intended to mean that the city denoted by A is a hub city, and
Satellite(A1,A) is intended to mean that the city denoted by A1 is a
satellite of the city denoted by A.) From these training facts, we want to
induce a program Nonstop(x,y), written in terms of the background re-
lations Hub and Satellite, that has value T for all the positive instances
and has value F for all the negative instances. Depending on the exact set
of examples, we might induce the program:
Nonstop(x,y) :- Hub(x), Hub(y)
:- Satellite(x,y)
:- Satellite(y,x)
which would have value T if both of the two cities were hub cities or if
one were a satellite of the other. As with other learning problems, we
want the induced program to generalize well that is, if presented with
arguments not represented in the training set (but for which we have the
needed background knowledge), we would like the function to guess well.
7.1. NOTATION AND DEFINITIONS 99
7.1 Notation and Denitions
In evaluating logic programs in ILP, we implicitly append the background
facts to the program and adopt the usual convention that a program has
value T for a set of inputs if and only if the program interpreter returns
T when actually running the program (with background facts appended)
on those inputs otherwise it has value F . Using the given background
facts, the program above would return T for input (A, A1), for example.
If a logic program, , returns T for a set of arguments X, we say that
the program covers the arguments and write covers( X). Following our
terminology introduced in connection with version spaces, we will say that
a program is sucient if it covers all of the positive instances and that
it is necessary if it does not cover any of the negative instances. (That
is, a program implements a sucient condition that a training instance
is positive if it covers all of the positive training instances it implements
a necessary condition if it covers none of the negative instances.) In the
noiseless case, we want to induce a program that is both sucient and
necessary, in which case we will call it consistent. With imperfect (noisy)
training sets, we might relax this criterion and settle for a program that
covers all but some fraction of the positive instances while allowing it to
cover some fraction of the negative instances. We illustrate these denitions
schematically in Fig. 7.1.
π1 is a necessary program
A positive instance
covered by π2 and π3
−
+
+ + + − −
+ − −
+ −
+ −
+ −
+ +
−
π2 is a sufficient program
π3 is a consistent program
Figure 7.1: Sucient, Necessary, and Consistent Programs

As in version spaces, if a program is sucient but not necessary it can be
made to cover fewer examples by specializing it. Conversely, if it is necessary
but not sucient, it can be made to cover more examples by generalizing
it. Suppose we are attempting to induce a logic program to compute the
relation . The most general logic program, which is certainly sucient,
is the one that has value T for all inputs, namely a single clause with an
empty body, :- ], which is called a fact in Prolog. The most special
logic program, which is certainly necessary, is the one that has value F for
all inputs, namely :- F ]. Two of the many di erent ways to search for
a consistent logic program are: 1) start with :- ] and specialize until
the program is consistent, or 2) start with :- F ] and generalize until
the program is consistent. We will be discussing a method that starts with
:- ], specializes until the program is necessary (but might no longer be
sucient), then reachieves suciency in stages by generalizing|ensuring
within each stage that the program remains necessary (by specializing).
7.2 A Generic ILP Algorithm

Since the primary operators in our search for a consistent program are
specialization and generalization, we must next discuss those operations.
There are three major ways in which a logic program might be generalized:
1. Replace some terms in a program clause by variables. (Readers fa-
miliar with substitutions in the predicate calculus will note that this
process is the inverse of substitution.)
2. Remove literals from the body of a clause.
3. Add a clause to the program
Analogously, there are three ways in which a logic program might be spe-
cialized:
1. Replace some variables in a program clause by terms (a substitution).
2. Add literals to the body of a clause.
3. Remove a clause from the program
We will be presenting an ILP learning method that adds clauses to a pro-
gram when generalizing and that adds literals to the body of a clause when
specializing. When we add a clause, we will always add the clause :- ]
7.2. A GENERIC ILP ALGORITHM 101
and then specialize it by adding literals to the body. Thus, we need only
describe the process for adding literals.
Clauses can be partially ordered by the specialization relation. In gen-
eral, clause c1 is more special than clause c2 if c2 j= c1. A special case,
which is what we use here, is that a clause c1 is more special than a clause
c2 if the set of literals in the body of c2 is a subset of those in c1 . This
ordering relation can be used in a structure of partially ordered clauses,
called the renement graph, that is similar to a version space. Clause c1 is
an immediate successor of clause c2 in this graph if and only if clause c1
can be obtained from clause c2 by adding a literal to the body of c2. A
renement graph then tells us the ways in which we can specialize a clause
by adding a literal to it.
Of course there are unlimited possible literals we might add to the body
of a clause. Practical ILP systems restrict the literals in various ways.
Typical allowed additions are:
1. Literals used in the background knowledge.
2. Literals whose arguments are a subset of those in the head of the
clause.
3. Literals that introduce a new distinct variable di erent from those in
the head of the clause.
4. A literal that equates a variable in the head of the clause with another
such variable or with a term mentioned in the background knowledge.
(This possibility is equivalent to forming a specialization by making
a substitution.)
5. A literal that is the same (except for its arguments) as that in the
head of the clause. (This possibility admits recursive programs, which
are disallowed in some systems.)
We can illustrate these possibilities using our air-ight example. We
start with the program Nonstop(x,y) :- ]. The literals used in the
background knowledge are Hub and Satellite. Thus the literals that we
might consider adding are:
Hub(x)
Hub(y)
Hub(z)
Satellite(x,y)

Satellite(y,x)
Satellite(x,z)
Satellite(z,y)
(x = y)
(If recursive programs are allowed, we could also add the literals
Nonstop(x,z) and Nonstop(z,y).) These possibilities are among those il-
lustrated in the renement graph shown in Fig. 7.2. Whatever restrictions
on additional literals are imposed, they are all syntactic ones from which
the successors in the renement graph are easily computed. ILP programs
that follow the approach we are discussing (of specializing clauses by adding
a literal) thus have well dened methods of computing the possible literals
to add to a clause.
Nonstop(x,y) :-
. . .
Nonstop(x,y) :- . . .
Hub(x)
Nonstop(x,y) :-
Satellite(x,y)
. . .
Nonstop(x,y) :-
(x = y)
. . . . . .
. . .
. . .
Figure 7.2: Part of a Renement Graph

Now we are ready to write down a simple generic algorithm for inducing
a logic program, for inducing a relation . We are given a training set,
of argument sets some known to be in the relation and some not in
7.3. AN EXAMPLE 103
+ are the positive instances, and ; are the negative instances. The
algorithm has an outer loop in which it successively adds clauses to make
more and more sucient. It has an inner loop for constructing a clause,
c, that is more and more necessary and in which it refers only to a subset,
cur , of the training instances. (The positive instances in cur will be
denoted by +cur , and the negative ones by ;cur .) The algorithm is also
given background relations and the means for adding literals to a clause.
It uses a logic program interpreter to compute whether or not the program
it is inducing covers training instances. The algorithm can be written as
follows:
Generic ILP Algorithm

(Adapted from Lavrac & Dzeroski, 1994, p. 60].)
Initialize cur := .
Initialize := empty set of clauses.
repeat The outer loop works to make sucient.]
Initialize c := : ; .
repeat The inner loop makes c necessary.]
Select a literal l to add to c. This is a nondeterministic choice point.]
Assign c := c l.
until c is necessary. That is, until c covers no negative instances in cur.]
Assign := c. We add the clause c to the program.]
Assign cur := cur ; (the positive instances in cur covered by ).
until is sucient.
(The termination tests for the inner and outer loops can be relaxed as
appropriate for the case of noisy instances.)
7.3 An Example
We illustrate how the algorithm works by returning to our example of airline
ights. Consider the portion of an airline route map, shown in Fig. 7.3.
Cities A, B, and C are \hub" cities, and we know that there are nonstop
ights between all hub cities (even those not shown on this portion of the
route map). The other cities are \satellites" of one of the hubs, and we know
that there are nonstop ights between each satellite city and its hub. The
learning program is given a set of positive instances, + , of pairs of cities
between which there are nonstop ights and a set of negative instances, ;,
of pairs of cities between which there are not nonstop ights. + contains
just the pairs:
f< A B > < A C > < C A > < C B >
< A A1 > < A A2 > < A1 A > < A2 A > 
 < C C 1 > < C C 2 > < C 1 C > < C 2 C >g
For our example, we will assume that ; contains all those pairs of cities
shown in Fig. 7.3 that are not in + (a type of closed-world assumption).
These are:
f< A B 1 > < A B 2 > < A C 1 > < A C 2 > 
 < C A1 > < C A2 > < C B 1 > < C B 2 >
 < C 1 A > < C 2 A > < C 1 B > < C 2 B >
< A1 B > < A2 B > < A1 C > < A2 C > g
There may be other cities not shown on this map, so the training set does
not necessarily exhaust all the cities.
B1 B2
A1 C1
A C
A2 C2
Figure 7.3: Part of an Airline Route Map

7.3. AN EXAMPLE 105
We want the learning program to induce a program for computing the
value of the relation Nonstop. The training set, , can be thought of as a
partial description of this relation in extensional form|it explicitly names
some pairs in the relation and some pairs not in the relation. We desire to
learn the Nonstop relation as a logic program in terms of the background
relations, Hub and Satellite, which are also given in extensional form.
Doing so will give us a more compact, intensional, description of the rela-
tion, and this description could well generalize usefully to other cities not
mentioned in the map.
We assume the learning program has the following extensional deni-
tions of the relations Hub and Satellite:
Hub
f< A > < C >g
All other cities mentioned in the map are assumed not in the relation Hub.
We will use the notation Hub(x) to express that the city named x is in the
relation Hub.
Satellite
f< A1 A > < A2 A > < C 1 C > < C 2 C >g
All other pairs of cities mentioned in the map are not in the relation
Satellite. We will use the notation Satellite(x,y) to express that the
pair < x y > is in the relation Satellite.
Knowing that the predicate Nonstop is a two-place predicate, the inner
loop of our algorithm initializes the rst clause to Nonstop(x,y) :- .
This clause is not necessary because it covers all the negative examples
(since it covers all examples). So we must add a literal to its (empty) body.
Suppose (selecting a literal from the renement graph) the algorithm adds
Hub(x). The following positive instances in are covered by Nonstop(x,y)
:- Hub(x):

f< A B > < A C > < C A > < C B >
< A A1 > < A A2 > < C C 1 > < C C 2 >g
To compute this covering, we interpret the logic program Nonstop(x,y) :-

Hub(x) for all pairs of cities in
, using the pairs given in the background
relation Hub as ground facts. The following negative instances are also
covered:
f< A B 1 > < A B 2 > < A C 1 > < A C 2 > < C A1 > < C A2 >
< C B 1 > < C B 2 > g
Thus, the clause is not yet necessary and another literal must be added.
Suppose we next add Hub(y). The following positive instances are covered
by Nonstop(x,y) :- Hub(x), Hub(y):
f< A B > < A C > < C A > < C B >g
There are no longer any negative instances in covered so the clause

Nonstop(x,y) :- Hub(x), Hub(y) is necessary, and we can terminate the
rst pass through the inner loop.
But the program, , consisting of just this clause is not sucient. These
positive instances are not covered by the clause:
f< A A1 > < A A2 > < A1 A > < A2 A > 
 < C C 1 > < C C 2 > < C 1 C > < C 2 C >g
The positive instances that were covered by Nonstop(x,y) :- Hub(x),

Hub(y) are removed from to form the cur to be used in the next pass
through the inner loop. cur consists of all the negative instances in plus
the positive instances (listed above) that are not yet covered. In order to
attempt to cover them, the inner loop creates another clause c, initially set
to Nonstop(x,y) :- . This clause covers all the negative instances, and
so we must add literals to make it necessary. Suppose we add the literal
Satellite(x,y). The clause Nonstop(x,y) :- Satellite(x,y) covers
no negative instances, so it is necessary. It does cover the following positive
instances in cur :

7.4. INDUCING RECURSIVE PROGRAMS 107
f< A1 A > < A2 A > < C 1 C > < C 2 C >g
These instances are removed from cur for the next pass through the inner
loop. The program now contains two clauses:

:- Satellite(x,y)
This program is not yet sucient since it does not cover the following
positive instances:
f< A A1 > < A A2 > < C C 1 > < C C 2 >g
During the next pass through the inner loop, we add the clause
Nonstop(x,y) :- Satellite(y,x). This clause is necessary, and since
the program containing all three clauses is now sucient, the procedure
terminates with:

:- Satellite(x,y)
:- Satellite(y,x)
Since each clause is necessary, and the whole program is sucient, the
program is also consistent with all instances of the training set. Note that
this program can be applied (perhaps with good generalization) to other
cities besides those in our partial map|so long as we can evaluate the
relations Hub and Satellite for these other cities. In the next section, we
show how the technique can be extended to use recursion on the relation
we are inducing. With that extension, the method can be used to induce
more general logic programs.
7.4 Inducing Recursive Programs

To induce a recursive program, we allow the addition of a literal having the
same predicate letter as that in the head of the clause. Various mechanisms
must be used to ensure that such a program will terminate one such is to
make sure that the new literal has di erent variables than those in the
head literal. The process is best illustrated with another example. Our
example continues the one using the airline map, but we make the map
somewhat simpler in order to reduce the size of the extensional relations
used. Consider the map shown in Fig. 7.4. Again, B and C are hub cities,
B1 and B2 are satellites of B, C1 and C2 are satellites of C. We have
introduced two new cities, B3 and C3. No ights exist between these cities
and any other cities|perhaps there are only bus routes as shown by the
grey lines in the map.
B3
B1
B2
C1
C
C3
C2
Figure 7.4: Another Airline Route Map

We now seek to learn a program for Canfly(x,y) that covers only those
pairs of cities that can be reached by one or more nonstop ights. The
relation Canfly is satised by the following pairs of postive instances:
f 
 < C B 1 > < C 1 B 1 > < C 2 B 1 >
 
< C B 2 > < C 1 B 2 > < C 2 B 2 > 
 < C B > < C 1 B > < C 2 B > < C C 1 >
< C C 2 > < C 1 C > < C 2 C > < C 1 C 2 > < C 2 C 1 >g

7.4. INDUCING RECURSIVE PROGRAMS 109
Using a closed-world assumption on our map, we take the negative instances
of Canfly to be:
f 
 
< C B 3 > < C 1 B 3 > < C 2 B 3 > < C 3 B 3 > < C 3 B 2 >
< C 3 B > < C 3 B 1 > < C 3 C > < C 3 C 1 > < C 3 C 2 >
 < C C 3 > < C 1 C 3 >
< C 2 C 3 >g
We will induce Canfly(x,y) using the extensionally dened background
relation Nonstop given earlier (modied as required for our reduced airline
map) and Canfly itself (recursively).
As before, we start with the empty program and proceed to the inner
loop to construct a clause that is necessary. Suppose that the inner loop
adds the background literal Nonstop(x,y). The clause Canfly(x,y) :-
Nonstop(x,y) is necessary it covers no negative instances. But it is not
sucient because it does not cover the following positive instances:
f 
< C B 1 > < C 1 B 1 > < C 2 B 1 > 
 < C B 2 > < C 1 B 2 > < C 2 B 2 > 
 < C 1 B > < C 2 B > < C 1 C 2 > < C 2 C 1 >g
Thus, we must add another clause to the program. In the inner loop, we rst
create the clause Canfly(x,y) :- Nonstop(x,z) which introduces the new
variable z . We digress briey to describe how a program containing a clause
with unbound variables in its body is interpreted. Suppose we try to inter-
pret it for the positive instance Canfly(B1,B2). The interpreter attempts
to establish Nonstop(B1,z) for some z . Since Nonstop(B1, B), for exam-
ple, is a background fact, the interpreter returns T |which means that the
instance is covered. Suppose now, we attempt to interpret the
clause for the negative instance Canfly(B3,B). The interpreter attempts to
establish Nonstop(B3,z) for some z . There are no background facts that
match, so the clause does not cover . Using the interpreter, we
see that the clause Canfly(x,y) :- Nonstop(x,z) covers all of the pos-
itive instances not already covered by the rst clause, but it also covers
many negative instances such as , and . So the inner
loop must add another literal. This time, suppose it adds Canfly(y,z)
to yield the clause Canfly(x,y) :- Nonstop(x,z), Canfly(y,z). This
clause is necessary no negative instances are covered. The program is now
sucient and consistent it is:
Canfly(x,y) :- Nonstop(x,y)
:- Nonstop(x,z), Canfly(y,z)
7.5 Choosing Literals to Add

One of the rst practical ILP systems was Quinlan's FOIL Quinlan, 1990].
A major problem involves deciding how to select a literal to add in the
inner loop (from among the literals that are allowed). In FOIL, Quinlan
suggested that candidate literals can be compared using an information-
like measure|similar to the measures used in inducing decision trees. A
measure that gives the same comparison as does Quinlan's is based on the
amount by which adding a literal increases the odds that an instance drawn
at random from those covered by the new clause is a positive instance
beyond what these odds were before adding the literal.
Let p be an estimate of the probability that an instance drawn at ran-
dom from those covered by a clause before adding the literal is a posi-
tive instance. That is, p =(number of positive instances covered by the
clause)/(total number of instances covered by the clause). It is convenient
to express this probability in \odds form." The odds, o, that a covered in-
stance is positive is dened to be o = p=(1 ; p). Expressing the probability
in terms of the odds, we obtain p = o=(1 + o).
After selecting a literal, l, to add to a clause, some of the instances
previously covered are still covered some of these are positive and some are
negative. Let pl denote the probability that an instance drawn at random
from the instances covered by the new clause (with l added) is positive.
The odds will be denoted by ol . We want to select a literal, l, that gives
maximal increase in these odds. That is, if we dene l = ol =o, we want a
literal that gives a high value of l . Specializing the clause in such a way
that it fails to cover many of the negative instances previously covered but
still covers most of the positive instances previously covered will result in
a high value of l . (It turns out that the value of Quinlan's information
theoretic measure increases monotonically with l , so we could just as well
use the latter instead.)
7.6. RELATIONSHIPS BETWEEN ILP AND DECISION TREE INDUCTION111
Besides nding a literal with a high value of l , Quinlan's FOIL system
also restricts the choice to literals that:
a) contain at least one variable that has already been used,
b) place further restrictions on the variables if the literal selected has
the same predicate letter as the literal being induced (in order to prevent
innite recursion), and
c) survive a pruning test based on the values of l for those literals
selected so far.
We refer the reader to Quinlan's paper for further discussion of these points.
Quinlan also discusses post-processing pruning methods and presents ex-
perimental results of the method applied to learning recursive relations on
lists, on learning rules for chess endgames and for the card game Eleu-
sis, and for some other standard tasks mentioned in the machine learning
literature.
The reader should also refer to Pazzani & Kibler, 1992,
Lavrac & Dzeroski, 1994, Muggleton, 1991, Muggleton, 1992]. Discuss
preprocessing,
postprocessing,
7.6 Relationships Between ILP and Deci- bottom-up
methods, and
LINUS.
sion Tree Induction
The generic ILP algorithm can also be understood as a type of decision tree
induction. Recall the problem of inducing decision trees when the values of
attributes are categorical. When splitting on a single variable, the split at
each node involves asking to which of several mutuallyexclusive and exhaus-
tive subsets the value of a variable belongs. For example, if a node tested
the variable xi , and if xi could have values drawn from fA B C D E F g,
then one possible split (among many) might be according to whether the
value of xi had as value one of fA B C g or one of fD E F g.
It is also possible to make a multi-variate split|testing the values of
two or more variables at a time. With categorical variables, an n-variable
split would be based on which of several n-ary relations the values of the
variables satised. For example, if a node tested the variables xi and xj ,
and if xi and xj both could have values drawn from fA B C D E F g, then
one possible binary split (among many) might be according to whether or
not < xi xj > satised the relation f< A C > < C D >g. (Note that
our subset method of forming single-variable splits could equivalently have
been framed using 1-ary relations|which are usually called properties.)
In this framework, the ILP problem is as follows: We are given a training
set, , of positively and negatively labeled patterns whose components are
drawn from a set of variables fx y z : : :g. The positively labeled patterns
in form an extensional denition of a relation, R. We are also given back-
ground relations, R1 : : : Rk , on various subsets of these variables. (That
is, we are given sets of tuples that are in these relations.) We desire to
construct an intensional denition of R in terms of the R1 : : : Rk, such
that all of the positively labeled patterns in are satised by R and none
of the negatively labeled patterns are. The intensional denition will be
in terms of a logic program in which the relation R is the head of a set of
clauses whose bodies involve the background relations.
The generic ILP algorithm can be understood as decision tree induction,
where each node of the decision tree is itself a sub-decision tree, and each
sub-decision tree consists of nodes that make binary splits on several vari-
ables using the background relations, Ri. Thus we will speak of a top-level
decision tree and various sub-decision trees. (Actually, our decision trees
will be decision lists|a special case of decision trees, but we will refer to
them as trees in our discussions.)
In broad outline, the method for inducing an intensional version of the
relation R is illustrated by considering the decision tree shown in Fig. 7.5.
In this diagram, the patterns in are rst ltered through the decision
tree in top-level node 1. The background relation R1 is satised by some
of these patterns these are ltered to the right (to relation R2), and the
rest are ltered to the left (more on what happens to these later). Right-
going patterns are ltered through a sequence of relational tests until only
positively labeled patterns satisfy the last relation|in this case R3. That is,
the subset of patterns satisfying all the relations, R1, R2, and R3 contains
only positive instances from . (We might say that this combination of
tests is necessary. They correspond to the clause created in the rst pass
through the inner loop of the generic ILP algorithm.) Let us call the subset
of patterns satisfying these relations, 1 these satisfy Node 1 at the top
level. All other patterns, that is f ; 1 g = 2 are ltered to the left by
Node 1.
2 is then ltered by top-level Node 2 in much the same manner, so that
Node 2 is satised only by the positively labeled samples in 2. We continue
ltering through top-level nodes until only the negatively labeled patterns
fail to satisfy a top node. In our example, 4 contains only negatively
labeled patterns and the union of 1 and 3 contains all the positively
labeled patterns. The relation, R, that distinguishes positive from negative
patterns in is then given in terms of the following logic program:
R :- R1, R2, R3
:- R4, R5

7.6. RELATIONSHIPS BETWEEN ILP AND DECISION TREE INDUCTION113
R1
Node 1 T
F
R2
T
F
R3
(only positive
T instances
F
satisfy all three
Node 2 F
tests)
Ξ2 = Ξ − Ξ 1
T
R4 Ξ1
T
F
R5
T (only positivel
F instances satisfy
these two tests)
F T
(only negative Ξ3
Ξ4= Ξ2 − Ξ3 instances)
Figure 7.5: A Decision Tree for ILP
If we apply this sort of decision-tree induction procedure to the problem

of generating a logic program for the relation Nonstop (refer to Fig. 7.3),
we obtain the decision tree shown in Fig. 7.6. The logic program resulting
from this decision tree is the same as that produced by the generic ILP
algorithm.
In setting up the problem, the training set, can be expressed as a
set of 2-dimensional vectors with components x and y. The values of these
components range over the cities fA B C A1 A2 B 1 B 2C 1C 2g except
(for simplicity) we do not allow patterns in which x and y have the same
value. As before, the relation, Nonstop, contains the following pairs of
cities, which are the positive instances:
f< A B > < A C > < C A > < C B >
< A A1 > < A A2 > < A1 A > < A2 A > 
 < C C 1 > < C C 2 > < C 1 C > < C 2 C >g

All other pairs of cities named in the map of Fig. 7.3 (using the closed
world assumption) are not in the relation Nonstop and thus are negative
instances.
Because the values of x and y are categorical, decision-tree induction
would be a very dicult task|involving as it does the need to invent rela-
tions on x and y to be used as tests. But with the background relations, Ri
(in this case Hub and Satellite), the problem is made much easier. We
select these relations in the same way that we select literals from among
the available tests, we make a selection based on which leads to the largest
value of R .
i

To be added.

Ξ
Node 1
(top level)
Hub(x) T
F
Hub(y)
T
F
T
{<A,B>, <A,C>,
Node 2 F <B,C>, <B,A>,
(top level) <C,A>, <C,B>}
(Only positive instances)
Satellite(x,y)
F T
T
{<A1,A>, <A2,A>, <B1,B>,
<B2,B>, <C1,C>, <C2,C>}
F
(Only positive instances)
Node 3
(top level)
Satellite(y,x)
F T
T
{<A,A1>, <A,A2>,<B,B1>,
<B,B2>, <C,C1>, <C,C2>}
F (Only positive instances)
{Only negative instances}
Figure 7.6: A Decision Tree for the Airline Route Problem


Chapter 8
Computational Learning
Theory
In chapter one we posed the problem of guessing a function given a set of
sample inputs and their values. We gave some intuitive arguments to sup-
port the claim that after seeing only a small fraction of the possible inputs
(and their values) that we could guess almost correctly the values of most
subsequent inputs|if we knew that the function we were trying to guess
belonged to an appropriately restricted subset of functions. That is, a given
training set of sample patterns might be adequate to allow us to select a
function, consistent with the labeled samples, from among a restricted set
of hypotheses such that with high probability the function we select will
be approximately correct (small probability of error) on subsequent sam-
ples drawn at random according to the same distribution from which the
labeled samples were drawn. This insight led to the theory of probably ap-
proximately correct (PAC) learning|initially developed by Leslie Valiant
Valiant, 1984]. We present here a brief description of the theory for the case
of Boolean functions. Dietterich, 1990, Haussler, 1988, Haussler, 1990]
give nice surveys of the important results. Other
overviews?
8.1 Notation and Assumptions for PAC

Learning Theory
We assume a training set of n-dimensional vectors, Xi , i = 1 : : : m, each
labeled (by 1 or 0) according to a target function, f, which is unknown to
117
118 CHAPTER 8. COMPUTATIONAL LEARNING THEORY
the learner. The probability of any given vector X being in , or later being
presented to the learner, is P(X). The probability distribution, P , can be
arbitrary. (In the literature of PAC learning theory, the target function is
usually called the target concept and is denoted by c, but to be consistent
with our previous notation we will continue to denote it by f.) Our problem
is to guess a function, h(X), based on the labeled samples in . In PAC
theory such a guessed function is called the hypothesis. We assume that the
target function is some element of a set of functions, C . We also assume that
the hypothesis, h, is an element of a set, H, of hypotheses, which includes
the set, C , of target functions. H is called the hypothesis space.
In general, h won't be identical to f, but we can strive to have the
value of h(X) = the value of f(X) for most X's. That is, we want h to
be approximately correct. To quantify this notion, we dene the error of h,
"h , as the probability that an X drawn randomly according to P will be
misclassied:
X
"h = P(X)
X:h(X)6=f (X)]
Boldface
symbols need
to be smaller We say that h is approximately (except for " ) correct if "h ", where " is
when they are the accuracy parameter.
subscripts in
math Suppose we are able to nd an h that classies all m randomly drawn
environments. training samples correctly that is, h is consistent with this randomly se-
lected training set, . If m is large enough, will such an h be approximately
correct (and for what value of ")? On some training occasions, using m
randomly drawn training samples, such an h might turn out to be approx-
imately correct (for a given value of "), and on others it might not. We say
that h is probably (except for ) approximately correct (PAC) if the proba-
bility that it is approximately correct is greater than 1 ; , where is the
condence parameter. We shall show that if m is greater than some bound
whose value depends on " and , such an h is guaranteed to be probably
approximately correct.
In general, we say that a learning algorithm PAC-learns functions from
C in terms of H i for every function f C , it outputs a hypothesis h H,
such that with probability at least (1 ; ), "h ". Such a hypothesis is
called probably (except for ) approximately (except for ") correct.
We want learning algorithms that are tractable, so we want an algorithm
that PAC-learns functions in polynomial time. This can only be done for
certain classes of functions. If there are a nite number of hypotheses in
a hypothesis set (as there are for many of the hypothesis sets we have
considered), we could always produce a consistent hypothesis from this
8.2. PAC LEARNING 119
set by testing all of them against the training data. But if there are an
exponential number of hypotheses, that would take exponential time. We
seek training methods that produce consistent hypotheses in less time. The
time complexities for various hypothesis sets have been determined, and
these are summarized in a table to be presented later.
A class, C , is polynomially PAC learnable in terms of H provided there
exists a polynomial-time learning algorithm (polynomial in the number of
samples needed, m, in the dimension, n, in 1=", and in 1=) that PAC-learns
functions in C in terms of H.
Initial work on PAC assumed H = C , but it was later shown that some
functions cannot be polynomially PAC-learned under such an assumption
(assuming P 6= NP)|but can be polynomially PAC-learned if H is a strict
superset of C ! Also our denition does not specify the distribution, P , from
which patterns are drawn nor does it say anything about the properties of
the learning algorithm. Since C and H do not have to be identical, we have
the further restrictive denition:
A properly PAC-learnable class is a class C for which there exists an
algorithm that polynomially PAC-learns functions from C in terms of C .
8.2 PAC Learning

8.2.1 The Fundamental Theorem
Suppose our learning algorithm selects some h randomly from among those
that are consistent with the values of f on the m training patterns. The
probability that the error of this randomly selected h is greater than some
", with h consistent with the values of f(X) for m instances of X (drawn
according to arbitrary P ), is less than or equal to jHje;"m , where jHj is
the number of hypotheses in H. We state this result as a theorem Blumer,
et al., 1987]:
Theorem 8.1 (Blumer, et al.) Let H be any set of hypotheses, be a
set of m 1 training examples drawn independently according to some
distribution P , f be any classication function in H, and " > 0. Then, the
probability that there exists a hypothesis h consistent with f for the members
of but with error greater than " is at most jHje;"m .
Proof:
Consider the set of all hypotheses, fh1 h2 : : : hi : : : hS g, in H, where
S = jHj. The error for hi is "hi = the probability that hi will classify a pat-
tern in error (that is, dierently than f would classify it). The probability
that hi will classify a pattern correctly is (1 ; "hi ). A subset, HB , of H will
have error greater than ". We will call the hypotheses in this subset bad.
The probability that any particular one of these bad hypotheses, say hb,
would classify a pattern correctly is (1 ; "hb ). Since "hb > ", the probability
that hb (or any other bad hypothesis) would classify a pattern correctly is
less than (1 ; "). The probability that it would classify all m independently
drawn patterns correctly is then less than (1 ; ")m .
That is,
probhb classies all m patterns correctly jhb HB ] (1 ; ")m .
probsome h HB classies all m patterns correctly]

P
= hb HB probhb classies all m patterns correctly jhb HB ]
K(1 ; ")m , where K = jHB j.
That is,
probthere is a bad hypothesis that classies all m patterns correctly]
K(1 ; ")m .
Since K jHj and (1 ; ")m e;"m , we have:

probthere is a bad hypothesis that classies all m patterns correctly]
= probthere is a hypothesis with error > " and that classies all m
patterns correctly] jHje;"m .
2
A corollary of this theorem is:
Corollary 8.2 Given m (1=")(ln jHj + ln(1=)) independent samples,
the probability that there exists a hypothesis in H that is consistent with f
on these samples and has error greater than " is at most .
Proof: We are to nd a bound on m that guarantees that
probthere is a hypothesis with error > " and that classies all m pat-
terns correctly] . Thus, using the result of the theorem, we must show
that jHje;"m . Taking the natural logarithm of both sides yields:
ln jHj ; "m ln
or
m (1=")(ln jHj + ln(1=))

2
This corollary is important for two reasons. First it clearly states that
we can select any hypothesis consistent with the m samples and be assured
that with probability (1 ; ) its error will be less than ". Also, it shows
that in order for m to increase
k)
no more than polynomially with n, jHj can
be no larger than 2 O ( n . No class larger than that can be guaranteed to
be properly PAC learnable.
Here is a possible point of confusion: The bound given in the corollary is
an upper bound on the value of m needed to guarantee polynomial probably
approximately correct learning. Values of m greater than that bound are
sucient (but might not be necessary). We will present a lower (necessary)
bound later in the chapter.
8.2.2 Examples
Terms
Let H be the set of terms (conjunctions of literals). Then, jHj = 3n, and
m (1=")(ln(3n ) + ln(1=))
(1=")(1:1n + ln(1=))
Note that the bound on m increases only polynomially with n, 1=", and
1=.
For n = 50, " = 0:01 and = 0:01, m 5 961 guarantees PAC learn-
ability.
In order to show that terms are properly PAC learnable, we additionally
have to show that one can nd in time polynomial in m and n a hypothesis
h consistent with a set of m patterns labeled by the value of a term. The
following procedure for nding such a consistent hypothesis requires O(nm)
steps (adapted from Dietterich, 1990, page 268]):
We are given a training sequence, , of m examples. Find the rst
pattern, say X1 , in that list that is labeled with a 1. Initialize a Boolean
function, h, to the conjunction of the n literals corresponding to the values
of the n components of X1 . (Components with value 1 will have corre-
sponding positive literals components with value 0 will have corresponding
negative literals.) If there are no patterns labeled by a 1, we exit with the
null concept (h 0 for all patterns). Then, for each additional pattern, Xi,
that is labeled with a 1, we delete from h any Boolean variables appearing
in Xi with a sign dierent from their sign in h. After processing all the
patterns labeled with a 1, we check all of the patterns labeled with a 0 to
make sure that none of them is assigned value 1 by h. If, at any stage of
the algorithm, any patterns labeled with a 0 are assigned a 1 by h, then
there exists no term that consistently classies the patterns in , and we
Change this exit with failure. Otherwise, we exit with h.
paragraph if
this algorithm As an example, consider the following patterns, all labeled with a 1
was presented
in Chapter (from Dietterich, 1990]):
Three. (0 1 1 0)
(1 1 1 0)
(1 1 0 0)
After processing the rst pattern, we have h = x1x2 x3x4 after processing
the second pattern, we have h = x2x3x4 nally, after the third pattern, we
have h = x2x4 .
Linearly Separable Functions
Let H be the set of all linearly separable functions. Then, jHj 2n2 , and
;
m (1=") n2 ln 2 + ln(1=)
Again, note that the bound on m increases only polynomially with n, 1=",
and 1=.
For n = 50, " = 0:01 and = 0:01, m 173 748 guarantees PAC
learnability.
To show that linearly separable functions are properly PAC learnable,
we would have additionally to show that one can nd in time polynomial in
m and n a hypothesis h consistent with a set of m labeled linearly separable
Linear patterns.
programming
is polynomial.
8.2.3 Some Properly PAC-Learnable Classes
Some properly PAC-learnable classes of functions are given in the following
table. (Adapted from Dietterich, 1990, pages 262 and 268] which also
gives references to proofs of some of the time complexities.)
H jHj Time Complexity P. Learnable?

terms 3n polynomial yes
k-term DNF 2O(kn) NP-hard no
(k disjunctive terms)
k-DNF 2O(nk ) polynomial yes
(a disjunction of k-sized terms)
k-CNF 2O(nk ) polynomial yes
(a conjunction of k-sized clauses)
k-DL 2O(nkk lg n) polynomial yes
(decision lists with k-sized terms)
lin. sep. 2O(n2 ) polynomial yes
lin. sep. with (0,1) weights ? NP-hard no
k-2NN ? NP-hard no
DNF 22n polynomial no
(all Boolean functions)
(Members of the class k-2NN are two-layer, feedforward neural networks

with exactly k hidden units and one output unit.)
Summary: In order to show that a class of functions is Properly PAC-
Learnable :
1. Show that there is an algorithm that produces a consistent hypothesis
on m n-dimensional samples in time polynomial in m and n.
2. Show that the sample size, m, needed to ensure PAC learnability is
polynomial (or better) in (1="), (1=), and n by showing that ln jHj
is polynomial or better in the number of dimensions.
As hinted earlier, sometimes enlarging the class of hypotheses makes
learning easier. For example, the table above shows that k-CNF is PAC
learnable, but k-term-DNF is not. And yet, k-term-DNF is a subclass of
k-CNF! So, even if the target function were in k-term-DNF, one would be
able to nd a hypothesis in k-CNF that is probably approximately correct
for the target function. Similarly, linearly separable functions implemented
by TLUs whose weight values are restricted to 0 and 1 are not properly
PAC learnable, whereas unrestricted linearly separable functions are. It
is possible that enlarging the space of hypotheses makes nding one that
is consistent with the training examples easier. An interesting question
is whether or not the class of functions in k-2NN is polynomially PAC
learnable if the hypotheses are drawn from k0-2NN with k0 > k. (At the
time of writing, this matter is still undecided.)
Although PAC learning theory is a powerful analytic tool, it (like com-
plexity theory) deals mainly with worst-case results. The fact that the class
of two-layer, feedforward neural networks is not polynomially PAC learn-
able is more an attack on the theory than it is on the networks, which have
had many successful applications. As Baum, 1994, page 416-17] says: \
: : : humans are capable of learning in the natural world. Therefore, a proof
within some model of learning that learning is not feasible is an indictment
of the model. We should examine the model to see what constraints can be
relaxed and made more realistic."
8.3 The Vapnik-Chervonenkis Dimension

8.3.1 Linear Dichotomies
Consider a set, H, of functions, and a set, , of (unlabeled) patterns. One
measure of the expressive power of a set of hypotheses, relative to , is its
ability to make arbitrary classications of the patterns in .1 If there are m
patterns in , there are 2m dierent ways to divide these patterns into two
disjoint and exhaustive subsets. We say there are 2m dierent dichotomies
of . Ifn were to include all of the 2n Boolean patterns, for example, there
are 22 ways to dichotomize them, and (of course) the set of all possible
Boolean functions dichotomizes them in all of these ways. But a subset, H,
of the Boolean functions might not be able to dichotomize an arbitrary set,
, of m Boolean patterns in all 2m ways. In general (that is, even in the
non-Boolean case), we say that if a subset, H, of functions can dichotomize
a set, , of m patterns in all 2m ways, then H shatters .
As an example, consider a set of m patterns in the n-dimensional
space, Rn . (That is, the n components of these patterns are real numbers.)
We dene a linear dichotomy as one implemented by an (n ; 1)-dimensional
hyperplane in the n-dimensional space. How many linear dichotomies of m
patterns in n dimensions are there? For example, as shown in Fig. 8.1,
there are 14 dichotomies of four points in two dimensions (each separating
line yields two dichotomies depending on whether the points on one side
of the line are classied as 1 or 0). (Note that even though there are an
innite number of hyperplanes, there are, nevertheless, only a nite number
1 And, of course, if a hypothesis drawn from a set that could make arbitrary classi-
cations of a set of training patterns, there is little likelihood that such a hypothesis will
generalize well beyond the training set.

8.3. THE VAPNIK-CHERVONENKIS DIMENSION 125
of ways in which hyperplanes can dichotomize a nite number of patterns.
Small movements of a hyperplane typically do not change the classications
of any patterns.)
7
6
2 1
3
14 dichotomies of 4 points in 2 dimensions
Figure 8.1: Dichotomizing Points in Two Dimensions

The number of dichotomies achievable by hyperplanes depends on how
the patterns are disposed. For the maximum number of linear dichotomies,
the points must be in what is called general position. For m > n, we
say that a set of m points is in general position in an n-dimensional space
if and only if no subset of (n + 1) points lies on an (n ; 1)-dimensional
hyperplane. When m n, a set of m points is in general position if no
(m ; 2)-dimensional hyperplane contains the set. Thus, for example, a set
of m 4 points is in general position in a three-dimensional space if no
four of them lie on a (two-dimensional) plane. We will denote the number
of linear dichotomies of m points in general position in an n-dimensional
space by the expression L (m n).
It is not too dicult to verify that: Include the
derivation.
n
X
L (m n) = 2 C(m ; 1 i) for m > n and
i=0
= 2m for m n
where C(m ; 1 i) is the binomial coecient (m(m;1;;1)!i)!i! .
The table below shows some values for L (m n).
m n
(no. of patterns) (dimension)
1 2 3 4 5
1 2 2 2 2 2
2 4 4 4 4 4
3 6 8 8 8 8
4 8 14 16 16 16
5 10 22 30 32 32
6 12 32 52 62 64
7 14 44 84 114 126
8 16 58 128 198 240
Note that the class of linear dichotomies shatters the m patterns if m

n + 1. The bold-face entries in the table correspond to the highest values
of m for which linear dichotomies shatter m patterns in n dimensions.
8.3.2 Capacity
Let Pmn = L2(mn )
m = the probability that a randomly selected dichotomy
m
(out of the 2 possible dichotomies of m patterns in n dimensions) will be
linearly separable. In Fig. 8.2 we plot P(n+1)n versus and n, where
= m=(n + 1).
Note that for large n (say n > 30) how quickly Pmn falls from 1 to
0 as m goes above 2(n + 1). For m < 2(n + 1), any dichotomy of the
m points is almost certainly linearly separable. But for m > 2(n + 1), a
randomly selected dichotomy of the m points is almost certainly not linearly
separable. For this reason m = 2(n + 1) is called the capacity of a TLU
Cover, 1965]. Unless the number of training patterns exceeds the capacity,
the fact that a TLU separates those training patterns according to their
labels means nothing in terms of how well that TLU will generalize to new
patterns. There is nothing special about a separation found for m < 2(n+1)
patterns|almost any dichotomy of those patterns would have been linearly
separable. To make sure that the separation found is forced by the training
set and thus generalizes well, it has to be the case that there are very few
linearly separable functions that would separate the m training patterns.
8.3. THE VAPNIK-CHERVONENKIS DIMENSION 127
Pλ(n + 1), n
1
0.75
75 50
0.5
.5
0.25
25 40
0 30
0
1 20
2
10
3
4
λ
Figure 8.2: Probability that a Random Dichotomy is Linearly Separable
Analogous results about the generalizing abilities of neural networks

have been developed by Baum & Haussler, 1989] and given intuitive and
experimental justication in Baum, 1994, page 438]:
\The results seemed to indicate the following heuristic rule
holds. If M examples can be correctly classied by] a net with
W weights (for M >> W ), the net will make a fraction " of
errors on new examples chosen from the same uniform] distri-
bution where " = W=M."
8.3.3 A More General Capacity Result

Corollary 7.2 gave us an expression for the number of training patterns
sucient to guarantee a required level of generalization|assuming that
the function we were guessing was a function belonging to a class of known
and nite cardinality. The capacity result just presented applies to linearly
separable functions for non-binary patterns. We can extend these ideas to
general dichotomies of non-binary patterns.
In general, let us denote the maximum number of dichotomies of any set
of m n-dimensional patterns by hypotheses in H as H (m n). The number
of dichotomies will, of course, depend on the disposition of the m points
in the n-dimensional space we take H (m n) to be the maximum over all
possible arrangements of the m points. (In the case of the class of linearly
separable functions, the maximum number is achieved when the m points
are in general position.) For each class, H, there will be some maximum
value of m for which H (m n) = 2m , that is, for which H shatters the m
patterns. This maximum number is called the Vapnik-Chervonenkis (VC)
dimension and is denoted by VCdim(H) Vapnik & Chervonenkis, 1971].
We saw that for the class of linear dichotomies, VCdim(Linear) = (n+
1). As another example, let us calculate the VC dimension of the hypothesis
space of single intervals on the real line|used to classify points on the real
line. We show an example of how points on the line might be dichotomized
by a single interval in Fig. 8.3. The set could be, for example, f0.5, 2.5,
- 2.3, 3.14g, and one of the hypotheses in our set would be 1, 4.5]. This
hypothesis would label the points 2.5 and 3.14 with a 1 and the points - 2.3
and 0.5 with a 0. This set of hypotheses (single intervals on the real line)
can arbitrarily classify any two points. But no single interval can classify
three points such that the outer two are classied as 1 and the inner one as
0. Therefore the VC dimension of single intervals on the real line is 2. As
soon as we have many more than 2 training patterns on the real line and
provided we know that the classication function we are trying to guess is
a single interval, then we begin to have good generalization.
Figure 8.3: Dichotomizing Points by an Interval
The VC dimension is a useful measure of the expressive power of a

hypothesis set. Since any dichotomy of VCdim(H) or fewer patterns in
general position in n dimensions can be achieved by some hypothesis in
H, we must have many more than VCdim(H) patterns in the training set
in order that a hypothesis consistent with the training set is suciently
constrained to imply good generalization. Our examples have shown that
the concept of VC dimension is not restricted to Boolean functions.
8.4. VC DIMENSION AND PAC LEARNING 129
8.3.4 Some Facts and Speculations About the VC Di-
mension
If there are a nite number, jHj, of hypotheses in H, then:
VCdim(H) log(jHj)
The VC dimension of terms in n dimensions is n.
Suppose we generalize our example that used a hypothesis set of single
intervals on the real line. Now let us consider an n-dimensional feature
space and tests of the form Li xi Hi. We allow only one such
test per dimension. A hypothesis space consisting of conjunctions of
these tests (called axis-parallel hyper-rectangles) has VC dimension
bounded by:
n VCdim 2n
As we have already seen, TLUs with n inputs have a VC dimension
of n + 1.
Baum, 1994, page 438] gives experimental evidence for the proposi-
tion that \ : : : multilayer neural] nets have a VC dimension roughly
equal to their total number of adjustable] weights."
8.4 VC Dimension and PAC Learning

There are two theorems that connect the idea of VC dimension with PAC
learning Blumer, et al., 1990]. We state these here without proof.
Theorem 8.3 (Blumer, et al.) A hypothesis space H is PAC learnable i
it has nite VC dimension.
Theorem 8.4 A set of hypotheses, H, is properly PAC learnable if:
1. m (1=") max4 lg(2=) 8 VCdim lg(13=")], and
2. if there is an algorithm that outputs a hypothesis h H consistent
with the training set in polynomial (in m and n) time.
The second of these two theorems improves the bound on the number
of training patterns needed for linearly separable functions to one that is
linear in n. In our previous example of how many training patterns were
needed to ensure PAC learnability of a linearly separable function if n = 50,
" = 0:01, and = 0:01, we obtained m 173 748. Using the Blumer, et
al. result we would get m 52 756.
As another example of the second theorem, let us take H to be the
set of closed intervals on the real line. The VC dimension is 2 (as shown
previously). With n = 50, " = 0:01, and = 0:01, m 16 551 ensures
PAC learnability.
There is also a theorem that gives a lower (necessary) bound on the num-
ber of training patterns required for PAC learning Ehrenfeucht, et al., 1988]:
Theorem 8.5 Any PAC learning algorithm must examine at least
!(1=" lg(1=) + VCdim(H)) training patterns.
The dierence between the lower and upper bounds is
O(log(1=")VCdim(H)=").

To be added.

Chapter 9
Unsupervised Learning
9.1 What is Unsupervised Learning?
Consider the various sets of points in a two-dimensional space illustrated
in Fig. 9.1. The rst set (a) seems naturally partitionable into two classes,
while the second (b) seems dicult to partition at all, and the third (c) is
problematic. Unsupervised learning uses procedures that attempt to nd
natural partitions of patterns. There are two stages:
Form an R-way partition of a set
of unlabeled training patterns
(where the value of R, itself, may need to be induced from the pat-
terns). The partition separates
into R mutually exclusive and ex-
haustive subsets,
1 : : :
R, called clusters.
Design a classier based on the labels assigned to the training patterns
by the partition.
We will explain shortly various methods for deciding how many clusters
there should be and for separating a set of patterns into that many clusters.
We can base some of these methods, and their motivation, on minimum-
description-length (MDL) principles. In that setting, we assume that we
want to encode a description of a set of points,
, into a message of mini-
mal length. One encoding involves a description of each point separately
other, perhaps shorter, encodings might involve a description of clusters of
points together with how each point in a cluster can be described given
the cluster it belongs to. The specic techniques described in this chapter
do not explicitly make use of MDL principles, but the MDL method has
131
132 CHAPTER 9. UNSUPERVISED LEARNING
a) two clusters
b) one cluster
c) ?
Figure 9.1: Unlabeled Patterns
been applied with success. One of the MDL-based methods, Autoclass II

Cheeseman, et al., 1988] discovered a new classication of stars based on
the properties of infrared sources.
Another type of unsupervised learning involves nding hierarchies of

partitionings or clusters of clusters. A hierarchical partition is one in which
is divided into mutually exclusive and exhaustive subsets,

1 : : :
R
each set,
i , (i = 1 : : : R) is divided into mutually exclusive and exhaustive
subsets, and so on. We show an example of such a hierarchical partition
in Fig. 9.2. The hierarchical form is best displayed as a tree, as shown
in Fig. 9.3. The tip nodes of the tree can further be expanded into their
individual pattern elements. One application of such hierarchical partitions
is in organizing individuals into taxonomic hierarchies such as those used
in botany and zoology.
9.2. CLUSTERING METHODS 133
Ξ21 ∪ Ξ22 ∪ Ξ23 = Ξ2

Ξ21
Ξ22
Ξ11
Ξ11 ∪ Ξ12 = Ξ1
Ξ23
Ξ12
Ξ1 ∪ Ξ2 ∪ Ξ3 = Ξ
Ξ31 ∪ Ξ32 = Ξ3
Ξ31
Ξ32
Figure 9.2: A Hierarchy of Clusters
9.2 Clustering Methods

9.2.1 A Method Based on Euclidean Distance
Most of the unsupervised learning methods use a measure of similarity be-
tween patterns in order to group them into clusters. The simplest of these
involves dening a distance between patterns. For patterns whose features
are numeric, the distance measure can be ordinary Euclidean distance be-
tween two points in an n-dimensional space.
There is a simple, iterative clustering method based on distance. It
can be described as follows. Suppose we have R randomly chosen cluster
seekers, C1 : : : CR . These are points in an n-dimensional space that we
want to adjust so that they each move toward the center of one of the
clusters of patterns. We present the (unlabeled) patterns in the training
Ξ1 Ξ3 Ξ2
Ξ11 Ξ12 Ξ31 Ξ32 Ξ21 Ξ22 Ξ23
Figure 9.3: Displaying a Hierarchy as a Tree
set,
, to the algorithm one-by-one. For each pattern, Xi , presented, we
nd that cluster seeker, Cj , that is closest to Xi and move it closer to Xi:
Cj ; (1 ; j )Cj + j Xi
where j is a learning rate parameter for the j-th cluster seeker it deter-
mines how far Cj is moved toward Xi .
Renements on this procedure make the cluster seekers move less far
as training proceeds. Suppose each cluster seeker, Cj , has a mass, mj ,
equal to the number of times that it has moved. As a cluster seeker's mass
increases it moves less far towards a pattern. For example, we might set
j = 1=(1+mj ) and use the above rule together with mj ; mj +1. With
this adjustment rule, a cluster seeker is always at the center of gravity
(sample mean) of the set of patterns toward which it has so far moved.
Intuitively, if a cluster seeker ever gets within some reasonably well clustered
set of patterns (and if that cluster seeker is the only one so located), it will
converge to the center of gravity of that cluster.
Once the cluster seekers have converged, the classier implied by the
now-labeled patterns in
can be based on a Voronoi partitioning of the
space (based on distances to the various cluster seekers). This kind of
classication, an example of which is shown in Fig. 9.4, can be implemented
by a linear machine. Georgy
Fedoseevich
Voronoi, was a
Russian
mathematician
who lived from
Separating boundaries 1868 to 1909.
C2
C1
C3
Figure 9.4: Minimum-Distance Classication

When basing partitioning on distance, we seek clusters whose patterns
are as close together as possible. We can measure the badness, V , of a
cluster of patterns, fXi g, by computing its sample variance dened by:
V = (1=K)
X(Xi ; M)2
i
where M is the sample mean of the cluster, which is dened to be:
M = (1=K)
X Xi
i
and K is the number of points in the cluster.
We would like to partition a set of patterns into clusters such that the
sum of the sample variances (badnesses) of these clusters is small. Of course
if we have one cluster for each pattern, the sample variances will all be zero,
so we must arrange that our measure of the badness of a partition must
increase with the number of clusters. In this way, we can seek a trade-o
between the variances of the clusters and the number of them in a way
somewhat similar to the principle of minimal description length discussed
earlier.
Elaborations of our basic cluster-seeking procedure allow the number
of cluster seekers to vary depending on the distances between them and
depending on the sample variances of the clusters. For example, if the
distance, dij , between two cluster seekers, Ci and Cj , ever falls below some
threshold ", then we can replace them both by a single cluster seeker placed
at their center of gravity (taking into account their respective masses). In
this way we can decrease the overall badness of a partition by reducing the
number of clusters for comparatively little penalty in increased variance.
On the other hand, if any of the cluster seekers, say Ci , denes a cluster
whose sample variance is larger than some amount , then we can place a
new cluster seeker, Cj , at some random location somewhat adjacent to Ci
and reset the masses of both Ci and Cj to zero. In this way the badness
of the partition might ultimately decrease by decreasing the total sample
variance with comparatively little penalty for the additional cluster seeker.
The values of the parameters " and are set depending on the relative
weights given to sample variances and numbers of clusters.
In distance-based methods, it is important to scale the components of
the pattern vectors. The variation of values along some dimensions of the
pattern vector may be much dierent than that of other dimensions. One
commonly used technique is to compute the standard deviation (i.e., the
square root of the variance) of each of the components over the entire train-
ing set and normalize the values of the components so that their adjusted
standard deviations are equal.
9.2.2 A Method Based on Probabilities

Suppose we have a partition of the training set,
, into R mutually exclu-
sive and exhaustive clusters, C1 : : : CR. We can decide to which of these
clusters some arbitrary pattern, X, should be assigned by selecting the Ci
for which the probability, p(Ci jX), is largest, providing p(Ci jX) is larger
than some xed threshold, . As we saw earlier, we can use Bayes rule
and base our decision on maximizing p(XjCi )p(Ci). Assuming conditional
independence of the pattern components, xi, the quantity to be maximized
is:
S(X Ci ) = p(x1 jCi)p(x2jCi) p(xnjCi)p(Ci )
The p(xj jCi) can be estimated from the sample statistics of the patterns in
the clusters and then used in the above expression. (Recall the linear form
that this formula took in the case of binary-valued components.)
We call S(X Ci ) the similarity of X to a cluster, Ci, of patterns. Thus,
we assign X to the cluster to which it is most similar, providing the simi-
larity is larger than .
X Xj
Just as before, we can dene the sample mean of a cluster, Ci, to be:
Mi = (1=Ki)
X j Ci
where Ki is the number of patterns in Ci .
We can base an iterative clustering algorithm on this measure of simi-
larity Mahadevan & Connell, 1992]. It can be described as follows:
1. Begin with a set of unlabeled patterns
and an empty list, L, of
clusters.
2. For the next pattern, X, in
, compute S(X Ci ) for each cluster, Ci.
(Initially, these similarities are all zero.) Suppose the largest of these
similarities is S(X Cmax ).
(a) If S(X Cmax ) > , assign X to Cmax . That is,
Cmax ; Cmax fXg
Update the sample statistics p(x1jCmax ) p(x2jCmax) : : : p(xnjCmax ),
and p(Cmax ) to take the new pattern into account. Go to 3.
(b) If S(X Cmax ) , create a new cluster, Cnew = fXg and add
Cnew to L. Go to 3.
3. Merge any existing clusters, Ci and Cj if (Mi ; Mj )2 < ". Compute
new sample statistics p(x1jCmerge ) p(x2jCmerge ) : : : p(xnjCmerge),
and p(Cmerge ) for the merged cluster, Cmerge = Ci Cj .
4. If the sample statistics of the clusters have not changed during an
entire iteration through
, then terminate with the clusters in L
otherwise go to 2.
The value of the parameter controls the number of clusters. If is
high, there will be a large number of clusters with few patterns in each
cluster. For small values of , there will be a small number of clusters
with many patterns in each cluster. Similarly, the larger the value of ", the
smaller the number clusters that will be found.
Designing a classier based on the patterns labeled by the partitioning is
straightforward. We assign any pattern, X, to that category that maximizes
S(X Ci ). Mention
\k-means and
\EM"
Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved. methods.
9.3 Hierarchical Clustering Methods
9.3.1 A Method Based on Euclidean Distance
Suppose we have a set,
, of unlabeled training patterns. We can form
a hierarchical classication of the patterns in
by a simple agglomerative
method. (The description of this algorithm is based on an unpublished
manuscript by Pat Langley.) Our description here gives the general idea
we leave it to the reader to generate a precise algorithm.
We rst compute the Euclidean distance between all pairs of patterns
in
. (Again, appropriate scaling of the dimensions is assumed.) Suppose
the smallest distance is between patterns Xi and Xj . We collect Xi and
Xj into a cluster, C, eliminate Xi and Xj from
and replace them by a
cluster vector, C, equal to the average of Xi and Xj . Next we compute the
Euclidean distance again between all pairs of points in
. If the smallest
distance is between pairs of patterns, we form a new cluster, C, as before
and replace the pair of patterns in
by their average. If the shortest
distance is between a pattern, Xi , and a cluster vector, Cj (representing a
cluster, Cj ), we form a new cluster, C, consisting of the union of Cj and
fXi g. In this case, we replace Cj and Xi in
by their (appropriately
weighted) average and continue. If the shortest distance is between two
cluster vectors, Ci and Cj , we form a new cluster, C, consisting of the union
of Ci and Cj . In this case, we replace Ci and Cj by their (appropriately
weighted) average and continue. Since we reduce the number of points in
by one each time, we ultimately terminate with a tree of clusters rooted

in the cluster containing all of the points in the original training set.
An example of how this method aggregates a set of two dimensional
patterns is shown in Fig. 9.5. The numbers associated with each clus-
ter indicate the order in which they were formed. These clusters can be
organized hierarchically in a binary tree with cluster 9 as root, clusters 7
and 8 as the two descendants of the root, and so on. A ternary tree could
be formed instead if one searches for the three points in
whose triangle
dened by those patterns has minimal area.
9.3.2 A Method Based on Probabilities

A probabilistic quality measure for partitions
We can develop a measure of the goodness of a partitioning based on how
accurately we can guess a pattern given only what partition it is in. Suppose
we are given a partitioning of
into R classes, C1 : : : CR . As before, we
9.3. HIERARCHICAL CLUSTERING METHODS 139
1 7
3
2
9
5
6
Figure 9.5: Agglommerative Clustering
can compute the sample statistics p(xi jCk) which give probability values for
each component given the class assigned to it by the partitioning. Suppose
each component xi of X can take on the values vij , where the index j steps
over the domain of that component. We use the notation pi (vij jCk ) =
probability(xi = vij jCk).
Suppose we use the following probabilistic guessing rule about the values
of the components of a vector X given only that it is in class k. Guess that
xi = vij with probability pi(vij jCk ). Then, the probability that we guess
the i-th component correctly is:
X probability(guess is vij )pi(vij Ck) = X pi(vij Ck)]
j j 2
j j
The average number of (the n) components whose values are guessed cor-
rectly by this method is then given by the sum of these probabilities over
all of the components of X:
X X pi(vij Ck)] j 2
i j

Given our partitioning into R classes, the goodness measure, G, of this
partitioning is the average of the above expression over all classes:
G=
X p(Ck) X X pi(vij Ck)] j
2
k i j
where p(Ck ) is the probability that a pattern is in class Ck . In order to
penalize this measure for having a large number of classes, we divide it by
R to get an overall \quality" measure of a partitioning:
Z = (1=R)
X p(Ck) X X pi(vij Ck)] j
2
k i j
We give an example of the use of this measure for a trivially simple
clustering of the four three-dimensional patterns shown in Fig. 9.6. There
are several dierent partitionings. Let's evaluate Z values for the following
ones: P1 = fa b c dg, P2 = ffa bg fc dgg, P3 = ffa cg fb dgg, and
P4 = ffag fbg fcg fdgg. The rst, P1, puts all of the patterns into a
single cluster. The sample probabilities pi (vi1 = 1) and pi (vi0 = 0) are all
equal to 1/2 for each of the three components. Summing over the values of
the components (0 and 1) gives (1=2)2 + (1=2)2 = 1=2. Summing over the
three components gives 3=2. Averaging over all of the clusters (there is just
one) also gives 3=2. Finally, dividing by the number of clusters produces
the nal Z value of this partition, Z(P1 ) = 3=2.
The second partition, P2, gives the following sample probabilities:
p1 (v11 = 1jC1) = 1
p2(v21 = 1jC1) = 1=2
p3 (v31 = 1jC1) = 1
Summing over the values of the components (0 and 1) gives (1)2 + (0)2 = 1
for component 1, (1=2)2 +(1=2)2 = 1=2 for component 2, and (1)2 +(0)2 = 1
for component 3. Summing over the three components gives 2 1=2 for class
1. A similar calculation also gives 2 1=2 for class 2. Averaging over the
two clusters also gives 2 1=2. Finally, dividing by the number of clusters
produces the nal Z value of this partition, Z(P2 ) = 1 1=4, not quite as
high as Z(P1 ).
Similar calculations yield Z(P3 ) = 1 and Z(P4 ) = 3=4, so this method
of evaluating partitions would favor placing all patterns in a single cluster.
9.3. HIERARCHICAL CLUSTERING METHODS 141
x3
d c
b
a x2
x1
Figure 9.6: Patterns in 3-Dimensional Space
An iterative method for hierarchical clustering

Evaluating all partitionings of m patterns and then selecting the best would
be computationally intractable. The following iterative method is based on
a hierarchical clustering procedure called COBWEB Fisher, 1987]. The
procedure grows a tree each node of which is labeled by a set of patterns.
At the end of the process, the root node contains all of the patterns in
. The successors of the root node will contain mutually exclusive and
exhaustive subsets of
. In general, the successors of a node, , are labeled
by mutually exclusive and exhaustive subsets of the pattern set labelling
node . The tips of the tree will contain singleton sets. The method uses Z
values to place patterns at the various nodes sample statistics are used to
update the Z values whenever a pattern is placed at a node. The algorithm
is as follows:
1. We start with a tree whose root node contains all of the patterns in

and a single empty successor node. We arrange that at all times
during the process every non-empty node in the tree has (besides any
other successors) exactly one empty successor.
2. Select a pattern Xi in
(if there are no more patterns to select,
terminate).
3. Set to the root node.
4. For each of the successors of (including the empty successor!), cal-
culate the best host for Xi . A best host is determined by tentatively
placing Xi in one of the successors and calculating the resulting Z
value for each one of these ways of accomodating Xi . The best host
corresponds to the assignment with the highest Z value.
5. If the best host is an empty node, , we place Xi in , generate an
empty successor node of , generate an empty sibling node of , and
go to 2.
6. If the best host is a non-empty, singleton (tip) node, , we place Xi in
, create one successor node of containing the singleton pattern that
was in , create another successor node of containing Xi , create an
empty successor node of , create empty successor nodes of the new
non-empty successors of , and go to 2.
7. If the best host is a non-empty, non-singleton node, , we place Xi in
, set to , and go to 4.
This process is rather sensitive to the order in which patterns are pre-
sented. To make the nal classication tree less order dependent, the COB-
WEB procedure incorporates node merging and splitting.
Node merging:
It may happen that two nodes having the same parent could be merged
with an overall increase in the quality of the resulting classication per-
formed by the successors of that parent. Rather than try all pairs to merge,
a good heuristic is to attempt to merge the two best hosts. When such a
merging improves the Z value, a new node containing the union of the pat-
terns in the merged nodes replaces the merged nodes, and the two nodes
that were merged are installed as successors of the new node.
Node splitting:
A heuristic for node splitting is to consider replacing the best host
among a group of siblings by that host's successors. This operation is
performed only if it increases the Z value of the classication performed by
a group of siblings.
Example results from COBWEB

We mention two experiments with COBWEB. In the rst, the program
attempted to nd two categories (we will call them Class 1 and Class 2) of
United States Senators based on their votes (yes or no) on six issues. After
the clusters were established, the majority vote in each class was computed.
These are shown in the table below.
Issue Class 1 Class 2
Toxic Waste yes no
Budget Cuts yes no
SDI Reduction no yes
Contra Aid yes no
Line-Item Veto yes no
MX Production yes no
In the second experiment, the program attempted to classify soybean
diseases based on various characteristics. COBWEB grouped the diseases
in the taxonomy shown in Fig. 9.7.
N0
soybean
diseases
N1 N2
Diaporthe N3
Charcoal
Stem Canker Rot
N31 N32
Rhizoctonia Phytophthora
Rot Rot
Figure 9.7: Taxonomy Induced for Soybean Diseases

To be added.


Chapter 10
Temporal-Dierence
Learning
10.1 Temporal Patterns and Prediction Prob-
lems
In this chapter, we consider problems in which we wish to learn to pre-
dict the future value of some quantity, say z, from an n-dimensional input
pattern, X. In many of these problems, the patterns occur in temporal
sequence, X1 , X2 , . . ., Xi , Xi+1 , : : :, Xm , and are generated by a dynam-
ical process. The components of Xi are features whose values are available
at time, t = i. We distinguish two kinds of prediction problems. In one,
we desire to predict the value of z at time t = i + 1 based on input Xi for
every i. For example, we might wish to predict some aspects of tomorrow's
weather based on a set of measurements made today. In the other kind
of prediction problem, we desire to make a sequence of predictions about
the value of z at some xed time, say t = m + 1, based on each of the Xi,
i = 1 : : : m. For example, we might wish to make a series of predictions
about some aspect of the weather on next New Year's Day, based on mea-
surements taken every day before New Year's. Sutton Sutton, 1988] has
called this latter problem, multi-step prediction, and that is the problem we
consider here. In multi-step prediction, we might expect that the prediction
accuracy should get better and better as i increases toward m.
145
146 CHAPTER 10. TEMPORAL-DIFFERENCE LEARNING
10.2 Supervised and Temporal-Dierence Meth-
ods
A training method that naturally suggests itself is to use the actual value of
z at time m +1 (once it is known) in a supervised learning procedure using
a sequence of training patterns, fX1, X2 , : : :, Xi , Xi+1 , : : :, Xm g. That
is, we seek to learn a function, f, such that f(Xi ) is as close as possible
to z for each i. Typically, we would need a training set, , consisting of
several such sequences. We will show that a method that is better than
supervised learning for some important problems is to base learning on the
dierence between f(Xi+1 ) and f(Xi ) rather than on the dierence between
z and f(Xi ). Such methods involve what is called temporal-dierence (TD)
learning.
We assume that our prediction, f(X), depends on a vector of modiable
weights, W. To make that dependence explicit, we write f(X W). For
supervised learning, we consider procedures of the following type: For each
Xi , the prediction f(Xi W) is computed and compared to z, and the
learning rule (whatever it is) computes the change, (Wi ), to be made
to W. Then, taking into account the weight changes for each pattern in a
sequence all at once after having made all of the predictions with the old
weight vector, we change W as follows:
; W + X(W)i
m
W
i=1
Whenever we are attempting to minimize the squared error between
z and f(Xi W) by gradient descent, the weight-changing rule for each
pattern is:
(W)i = c(z ; fi ) @@fWi
where c is a learning rate parameter, fi is our prediction of z, f(Xi W),
at time t = i, and @@fW is, by denition, the vector of partial derivatives
i
@f : : : @f : : : @f ) in which the w are the individual components of

( @w i i
@w
i
@w i
W. (The expression @@fW is sometimes written rW fi .) The reader will
1 i n
i
recall that we used an equivalent expression for (W)i in deriving the

backpropagation formulas used in training multi-layer neural networks.
The Widrow-Ho rule results when f(X W) = X W. Then:
(W)i = c(z ; fi )Xi
10.2. SUPERVISED AND TEMPORAL-DIFFERENCE METHODS 147
An interesting form for (W)i can be developed if we note that
(z ; fi ) =
X
m
(fk ; fk )
+1
k=i
where we dene fm+1 = z. Substituting in our formula for (W)i yields:
(W)i = c(z ; fi ) @@fWi
Xm
= c @@fWi (fk+1 ; fk )
k=i
In this form, instead of using the dierence between a prediction and the
value of z, we use the dierences between successive predictions|thus the
phrase temporal-dierence (TD) learning.
In the case when f(X W) = X W, the temporal dierence form of
the Widrow-Ho rule is:
(W)i = cXi
X
m
(fk ; fk )
+1
k=i
One reason for writing (W)i in temporal-dierence form is to permit
an interesting generalization as follows:
(W)i = c @@fWi
X
m
(k;i)(fk+1 ; fk )
k=i
where 0 < 1. Here, the term gives exponentially decreasing weight
to dierences later in time than t = i. When = 1, we have the same
rule with which we began|weighting all dierences equally, but as ! 0,
we weight only the (fi+1 ; fi ) dierence. With the term, the method is
called TD().
It is interesting to compare the two extreme cases:
For TD(0):
(W)i = c(fi+1 ; fi ) @@fWi

For TD(1):
(W)i = c(z ; fi ) @@fWi
Both extremes can be handled by the same learning mechanism only the
error term is dierent. In TD(0), the error is the dierence between succes-
sive predictions, and in TD(1), the error is the dierence between the nally
revealed value of z and the prediction. Intermediate values of take into
account dierently weighted dierences between future pairs of successive
predictions.
Only TD(1) can be considered a pure supervised learning procedure,
sensitive to the nal value of z provided by the teacher. For < 1, we have
various degrees of unsupervised learning, in which the prediction function
strives to make each prediction more like successive ones (whatever they
might be). We shall soon see that these unsupervised procedures result
in better learning than do the supervised ones for an important class of
problems.
10.3 Incremental Computation of the (W)i

We can rewrite our formula for (W)i , namely
@f Xm
(W)i = c @ W (k;i)(fk+1 ; fk )
i
k=i
to allow a type of incremental computation. First we write the expression
for the weight change rule that takes into account all of the (W)i :
; W + X c @@fWi X (k;i)(fk+1 ; fk )
m m
W
i=1 k=i
Interchanging the order of the summations yields:
; W + X c X (k;i)(fk+1 ; fk ) @@fWi
m k
W
k=1 i=1
=W+
X
m Xk
c(fk ; fk ) k;i @@fWi ( )
+1
k=1 i=1
10.3. INCREMENTAL COMPUTATION OF THE (W)I 149
Interchanging the indices k and i nally yields:
; W + X c(fi+1 ; fi) X (i;k) @@fWk
m i
W
i=1 k=1
Pmi=1as(earlier,
If, we want to use an expression of the form W ; W+
W)i, we see that we can write:
(W)i = c(fi+1 ; fi )
Xi i;k @fk
( )
k=1 @W
P
Now, if we let ei = ik=1 (i;k) @@fW , we can develop a computationally
k
ecient recurrence equation for ei+1 as follows:

X
i+1
ei+1 = (i+1;k) @@fWk
k=1
= @f i+1 Xi i( +1;k) @fk

@W + k=1 @W
= @f i+1
@ W + ei
Rewriting (W)i in these terms, we obtain:
(W)i = c(fi+1 ; fi )ei
where:
e1 = @@fW1
e2 = @@fW2 + e1
etc.
Quoting Sutton Sutton, 1988, page 15] (about a dierent equation, but the
quote applies equally well to this one):
\: : : this equation can be computed incrementally, because each
(W)i depends only on a pair of successive predictions and
on the weighted] sum of all past values for @@fW . This saves
i
substantially on memory, because it is no longer necessary to

individually remember all past values of @@fW ." i

10.4 An Experiment with TD Methods
TD prediction methods especially TD(0)] are well suited to situations in
which the patterns are generated by a dynamic process. In that case, se-
quences of temporally presented patterns contain important information
that is ignored by a conventional supervised method such as the Widrow-
Ho rule. Sutton Sutton, 1988, page 19] gives an interesting example in-
volving a random walk, which we repeat here. In Fig. 10.1, sequences of
vectors, X, are generated as follows: We start with vector XD the next
vector in the sequence is equally likely to be one of the adjacent vectors in
the diagram. If the next vector is XC (or XE ), the next one after that is
equally likely to be one of the vectors adjacent to XC (or XE ). When XB
is in the sequence, it is equally likely that the sequence terminates with
z = 0 or that the next vector is XC . Similarly, when XF is in the sequence,
it is equally likely that the sequence terminates with z = 1 or that the next
vector is XE . Thus the sequences are random, but they always start with
XD . Some sample sequences are shown in the gure. This random walk is
an example of a Markov process transitions from state i to state j occur
with probabilities that depend only on i and j.
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
XB XC XD XE XF
z=0 z=1
Typical Sequences:
XDXCXDXEXF 1
XDXCXBXCXDXEXDXEXF 1
XDXEXDXCXB 0
Figure 10.1: A Markov Process

Given a set of sequences generated by this process as a training set, we
want to be able to predict the value of z for each X in a test sequence. We
10.4. AN EXPERIMENT WITH TD METHODS 151
assume that the learning system does not know the transition probabilities.
For his experiments with this process, Sutton used a linear predictor,
that is f(X W) = X W. The learning problem is to nd a weight vector,
W, that minimizes the mean-squared error between z and the predicted
value of z. Given the ve dierent values that X can take on, we have the
following predictions: f(XB ) = w1, f(XC ) = w2, f(XD ) = w3, f(XE ) =
w4, f(XF ) = w5 , where wi is the i-th component of the weight vector.
(Note that the values of the predictions are not limited to 1 or 0|even
though z can only have one of those values|because we are minimizing
mean-squared error.) After training, these predictions will be compared
with the optimal ones|given the transition probabilities.
The experimental setup was as follows: ten random sequences were
generated using the transition probabilities. Each of these sequences was
presented in turn to a TD() method for various values of . Weight
vector increments, (W)i , were computed after each pattern presentation
but no weight changes were made until all ten sequences were presented.
The weight vector increments were summed after all ten sequences were
presented, and this sum was used to change the weight vector to be used
for the next pass through the ten sequences. This process was repeated
over and over (using the same training sequences) until (quoting Sutton)
\the procedure no longer produced any signicant changes in the weight
vector. For small c, the weight vector always converged in this way, and
always to the same nal value for 100 dierent training sets of ten random
sequences], independent of its initial value." (Even though, for xed, small
c, the weight vector always converged to the same vector, it might converge
to a somewhat dierent vector for dierent values of c.)
After convergence, the predictions made by the nal weight vector are
compared with the optimal predictions made using the transition probabil-
ities. These optimal predictions are simply p(z = 1jX). We can compute
these probabilities to be 1/6, 1/3, 1/2, 2/3, and 5/6 for XB , XC , XD ,
XE , XF , respectively. The root-mean-squared dierences between the best
learned predictions (over all c) and these optimal ones are plotted in Fig.
10.2 for seven dierent values of . (For each data point, the standard error
is approximately = 0:01.)
Notice that the Widrow-Ho procedure does not perform as well as
other versions of TD() for < 1! Quoting Sutton, 1988, page 21]:
\This result contradicts conventional wisdom. It is well known
that, under repeated presentations, the Widrow-Ho procedure
minimizes the RMS error between its predictions and the ac-
tual outcomes in the training set (Widrow & Stearns, 1985]).
Error using
best c
Widrow-Hoff
0.20 TD(1)
0.18
0.16
0.14
TD(0)
0.12
0.10
0.0 0.1 0.3 0.5 0.7 0.9 1.0
λ
(Adapted from Sutton, p. 20, 1988)
Figure 10.2: Prediction Errors for TD()
How can it be that this optimal method peformed worse than

all the TD methods for < 1? The answer is that the Widrow-
Ho procedure only minimizes error on the training set it does
not necessarily minimize error for future experience. Later] we
prove that in fact it is linear TD(0) that converges to what
can be considered the optimal estimates for matching future
experience|those consistent with the maximum-likelihood es-
timate of the underlying Markov process."
10.5 Theoretical Results

It is possible to analyze the performance of the linear-prediction TD()
methods on Markov processes. We state some theorems here without proof.
Theorem 10.1 (Sutton, page 24, 1988) For any absorbing Markov chain,
and for any linearly independent set of observation vectors fXi g for the
non-terminal states, there exists an " > 0 such that for all positive c < "
and for any initial weight vector, the predictions of linear TD(0) (with
10.6. INTRA-SEQUENCE WEIGHT UPDATING 153
weight updates after each sequence) converge in expected value to the opti-
mal (maximum likelihood) predictions of the true process.
Even though the expected values of the predictions converge, the pre-
dictions themselves do not converge but vary around their expected values
depending on their most recent experience. Sutton conjectures that if c is
made to approach 0 as training progresses, the variance of the predictions
will approach 0 also.
Dayan Dayan, 1992] has extended the result of Theorem 9.1 to TD()
for arbitrary between 0 and 1. (Also see Dayan & Sejnowski, 1994].)
10.6 Intra-Sequence Weight Updating

Our standard weight updating rule for TD() methods is:
W
X
m X
i
; W + c(fi ; fi) i;k @@fWk ( )
+1
i=1 k=1
where the weight update occurs after an entire sequence is observed. To
make the method truly incremental (in analogy with weight updating rules
for neural nets), it would be desirable to change the weight vector after
every pattern presentation. The obvious extension is:
; Wi + c(fi+1 ; fi ) X (i;k) @@fWk

i
Wi+1
k=1
where fi+1 is computed before making the weight change that is, fi+1 =
f(Xi+1 Wi). But that would make fi = f(Xi Wi;1), and such a rule
would make the prediction dierence, namely (fi+1 ; fi ), sensitive both to
changes in X and changes in W and could lead to instabilities. Instead, we
modify the rule so that, for every pair of predictions, fi+1 = f(Xi+1 Wi )
and fi = f(Xi Wi ). This version of the rule has been used in practice with
excellent results.
For TD(0) and linear predictors, the rule is:
Wi+1 = Wi + c(fi+1 ; fi)Xi
The rule is implemented as follows:
1. Initialize the weight vector, W, arbitrarily.
2. For i = 1 ::: m, do:
(a) fi ; Xi W
(We compute fi anew each time through rather than use the
value of fi+1 the previous time through.)
(b) fi+1 ; Xi+1 W
(c) di+1 ; fi+1 ; fi
(d) W ; W + c di+1Xi
(If fi were computed again with this changed weight vector, its
value would be closer to fi+1 as desired.)
The linear TD(0) method can be regarded as a technique for training a
very simple network consisting of a single dot product unit (and no thresh-
old or sigmoid function). TD methods can also be used in combination
with backpropagation to train neural networks. For TD(0) we change the
network weights according to the expression:
Wi+1 = Wi + c(fi+1 ; fi ) @@fWi
The only change that must be made to the standard backpropagation
weight-changing rule is that the dierence term between the desired output
and the output of the unit in the nal (k-th) layer, namely (d ; f (k) ),
must be replaced by a dierence term between successive outputs, (fi+1 ;
fi ). This change has a direct eect only on the expression for (k) which
becomes:
(k) = 2(f 0(k) ; f (k) )f (k) (1 ; f (k) )
where f 0(k) and f (k) are two successive outputs of the network.
The weight changing rule for the i-th weight vector in the j-th layer of
weights has the same form as before, namely:
Wi(j) ; Wi(j) + ci(j)X(j;1)
where the i(j ) are given recursively by:
i(j ) = fi(j ) (1 ; fi(j ) )

Xj
mj+1
( +1) (j +1)
wil
l
l=1
10.7. AN EXAMPLE APPLICATION: TD-GAMMON 155
and wil(j +1) is the l-th component of the i-th weight vector in the (j + 1)-th
layer of weights. Of course, here also it is assumed that f 0(k) and f (k) are
computed using the same weights and then the weights are changed. In the
next section we shall see an interesting example of this application of TD
learning.
10.7 An Example Application: TD-gammon

A program called TD-gammon Tesauro, 1992] learns to play backgammon
by training a neural network via temporal-dierence methods. The struc-
ture of the neural net, and its coding is as shown in Fig. 10.3. The network
is trained to minimize the error between actual payo and estimated payo,
where the actual payo is dened to be df = p1 +2p2 ; p3 ; 2p4 , and the pi
are the actual probabilities of the various outcomes as dened in the gure.
TD-gammon learned by using the network to select that move that
results in the best predicted payo. That is, at any stage of the game some
nite set of moves is possible and these lead to the set, fXg, of new board
positions. Each member of this set is evaluated by the network, and the
one with the largest predicted payo is selected if it is white's move (and
the smallest if it is black's). The move is made, and the network weights
are adjusted to make the predicted payo from the original position closer
to that of the resulting position.
The weight adjustment procedure combines temporal-dierence (TD())
learning with backpropagation. If dt is the network's estimate of the payo
at time t (before a move is made), and dt+1 is the estimate at time t + 1
(after a move is made), the weight adjustment rule is:
Wt = c(dt+1 ; dt )
Xt t;k @dk
k=1 @W
where Wt is a vector of all weights in the network at time t, and @@dW is the
k
gradient of dk in this weight space. (For a layered, feedforward network,

such as that of TD-gammon, the weight changes for the weight vectors in
each layer can be expressed in the usual manner.)
To make the special cases clear, recall that for TD(0), the network would
be trained so that, for all t, its output, dt , for input Xt tended toward its
expected output, dt+1, for input Xt+1 . For TD(1), the network would be
trained so that, for all t, its output, dt, for input Xt tended toward the
expected nal payo, df , given that input. The latter case is the same as
the Widrow-Ho rule.
After about 200,000 games the following results were obtained. TD-
gammon (with 40 hidden units, = 0:7, and c = 0:1) won 66.2% of 10,000
games against SUN Microsystems Gammontool and 55% of 10,000 games
against a neural network trained using expert moves. Commenting on
a later version of TD-gammon, incorporating special features as inputs,
Tesauro said: \It appears to be the strongest program ever seen by this
author."

To be added.

1 estimated payoff:
d = p1 + 2p2 − p3 − 2p4
2
no. of white
on cell 1
3
estimated probabilities:
#>3 p1 = pr(white wins)
p2 = pr(white gammons)
2 x 24 ... p3 = pr(black wins)

cells
...
p4 = pr(black gammons)
4 output units
no. on bar,
off board,
and who
moves
up to 40 hidden units
198 inputs
hidden and output units are sigmoids

learning rate: c = 0.1; initial weights chosen
randomly between −0.5 and +0.5.
Figure 10.3: The TD-gammon Network


Chapter 11
Delayed-Reinforcement
Learning
11.1 The General Problem
Imagine a robot that exists in an environment in which it can sense and
act. Suppose (as an extreme case) that it has no idea about the eects
of its actions. That is, it doesn't know how acting will change its sensory
inputs. Along with its sensory inputs are \rewards," which it occasionally
receives. How should it choose its actions so as to maximize its rewards
over the long run? To maximize rewards, it will need to be able to predict
how actions change inputs, and in particular, how actions lead to rewards.
We formalize the problem in the following way: The robot exists in an
environment consisting of a set, S , of states. We assume that the robot's
sensory apparatus constructs an input vector, X, from the environment,
which informs the robot about which state the environment is in. For
the moment, we will assume that the mapping from states to vectors is
one-to-one, and, in fact, will use the notation X to refer to the state of
the environment as well as to the input vector. When presented with an
input vector, the robot decides which action from a set, A, of actions to
perform. Performing the action produces an eect on the environment|
moving it to a new state. The new state results in the robot perceiving
a new input vector, and the cycle repeats. We assume a discrete time
model the input vector at time = is Xi , the action taken at that time
t i
is i , and the expected reward, i , received at = depends on the action

a r t i
taken and on the state, that is i = (Xi i). The learner's goal is to
r r a
159
160 CHAPTER 11. DELAYED-REINFORCEMENT LEARNING
nd a policy, (X), that maps input vectors to actions in such a way that

maximizes rewards accumulated over time. This type of learning is called

reinforcement learning. The learner must nd the policy by trial and error
it has no initial knowledge of the eects of its actions. The situation is as
shown in Fig. 11.1.
(state)
Xi
(action)
Learner
ri
(reward)
Environment ai
Figure 11.1: Reinforcement Learning
11.2 An Example
A \grid world," such as the one shown in Fig. 11.2 is ofEleven used to
illustrate reinforcement learning. Imagine a robot initially in cell (2,3). The
robot receives input vector ( 1 2) telling it what cell it is in it is capable
x x
of four actions, n e s w moving the robot one cell up, right, down, or left,
respectively. It is rewarded one negative unit whenever it bumps into the
wall or into the blocked cells. For example, if the input to the robot is (1,3),
and the robot chooses action , the next input to the robot is still (1,3)
w
and it receives a reward of ;1. If the robot lands in the cell marked (for G
goal), it receives a reward of +10. Let's suppose that whenever the robot
lands in the goal cell and gets its reward, it is immediately transported out
to some random cell, and the quest for reward continues.
A policy for our robot is a specication of what action to take for every
one of its inputs, that is, for every one of the cells in the grid. For example,
11.3. TEMPORAL DISCOUNTING AND OPTIMAL POLICIES 161
8
7 G
6
5
4
3 R
2
1
1 2 3 4 5 6 7
Figure 11.2: A Grid World
a component of such a policy would be \when in cell (3,1), move right."

An optimal policy is a policy that maximizes long-term reward. One way
of displaying a policy for our grid-world robot is by an arrow in each cell
indicating the direction the robot should move when in that cell. In Fig.
11.3, we show an optimal policy displayed in this manner. In this chapter we
will describe methods for learning optimal policies based on reward values
received by the learner.
11.3 Temporal Discounting and Optimal Poli-

cies
In delayed reinforcement learning, one often assumes that rewards in the
distant future are not as valuable as are more immediate rewards. This
preference can be accomodated by a temporal discount factor, 0 1. <
The present value of a reward, i, occuring time units in the future, is

r i
taken to be i i. Suppose we have a policy (X) that maps input vectors

r
into actions, and let i (X) be the reward that will be received on the -th
r i
time step after one begins executing policy starting in state X. Then the

total reward accumulated over all time steps by policy beginning in state

X is:
1
X (X)
V (X) = i
ri
i=0

8
7 G
6
5
4
3 R
2
1
1 2 3 4 5 6 7
Figure 11.3: An Optimal Policy in the Grid World
One reason for using a temporal discount factor is so that the above sum
will be nite. An optimal policy is one that maximizes (X) for all inputs,
V
X.
In general, we want to consider the case in which the rewards, i , are r
random variables and in which the eects of actions on environmental states

are random. In Markovian environments, for example, the probability that
action in state Xi will lead to state Xj is given by a transition probability
a
pXj jXi ]. Then, we will want to maximize expected future reward and
a
would dene (X) as:

V
"1 #
X i ( ) X
V (X) = E r
i
i=0
In either case, we call (X) the value of policy for input X.

V
If the action prescribed by taken in state X leads to state X0 (ran-

domly according to the transition probabilities), then we can write (X)

in terms of (X0 ) as follows:
V
X
V (X) = X (X)] +
r p X0jX (X)] (X0 )
V
X 0
where (in summary):

11.3. TEMPORAL DISCOUNTING AND OPTIMAL POLICIES 163
= the discount factor,
V (X) = the value of state X under policy ,
rX (X)] = the expected immediate reward received when we execute

the action prescribed by in state X, and
X0jX (X)] = the probability that the environment transitions to

state X0 when we execute the action prescribed by in state X.
p
In other words, the value of state X under policy is the expected value
of the immediate reward received when executing the action recommended

by plus the average value (under ) of all of the states accessible from X.

For an optimal policy, (and no others!), we have the famous \opti-

mality equation:"
2 3
X

(X) = max 4 (X ) + X0jX ] (X 0 ) 5

V r a p a V
a
X0
The theory of dynamic programming (DP) Bellman, 1957, Ross, 1983]

assures us that there is at least one optimal policy, , that satises
this equation. DP also provides methods for calculating (X) and at
least one , assuming that we know the average rewards and the tran-
V
sition probabilities. If we knew the transition probabilities, the average

rewards, and (X) for all X and , then it would be easy to imple-
V

a
ment an optimal policy. We would simply select that that maximizes

(X ) + X X0jX ] (X0 ). That is,
P
a
r a 0 p a V
2 3
(X) = arg max 4 (X ) + X
X0jX ] (X0 )5

r a p a V
a
X 0
But, of course, we are assuming that we do not know these average rewards
nor the transition probabilities, so we have to nd a method that eectively
learns them.
If we had a model of actions, that is, if we knew for every state, X, and
action , which state, X0 resulted, then we could use a method called value
a
iteration to nd an optimal policy. Value iteration works as follows: We

begin by assigning, randomly, an estimated value ^ (X) to every state, X. V
On the -th step of the process, suppose we are at state Xi (that is, our input
i
on the -th step is Xi ), and that the estimated value of state Xi on the -th
i i
step is î (Xi ). We then select that action that maximizes the estimated
V a
value of the predicted subsequent state. Suppose this subsequent state

having the highest estimated value is X0i . Then we update the estimated
value, î (Xi ), of state Xi as follows:
V
h i
^ (X) = (1 ; i) î;1 (X) +
Vi c V ci ri + î;1 (X0i )
V
if X = Xi ,
= î;1 (X)
V
otherwise.
We see that this adjustment
h moves the ^
i value of i (Xi ) an increment V
(depending on i ) closer to i + i (Xi ) . Assuming that î (X0i ) is a good

c r ^ 0 V V
estimate for i (X0i ), then this adjustment helps to make the two estimates
V
more consistent. Providing that 0 i 1 and that we visit each state

< c <
innitely often, this process of value iteration will converge to the optimal
Discuss values.
synchronous
dynamic
programming,
asynchronous
dynamic
programming,
11.4 Q-Learning
and policy
iteration. Watkins Watkins, 1989] has proposed a technique that he calls incremental
dynamic programming. Let a stand for the policy that chooses action a
once, and thereafter chooses actions according to policy . We dene:
Q (X ) = a V
a (X)
Then the optimal value from state X is given by:
(X) = max (X )

V Q a
a
This equation holds only for an optimal policy, . The optimal policy is
given by:

(X) = arg max Q

(X )
a
a
Note that if an action makes (X ) larger than (X), then we can

a Q a V
improve by changing it so that (X) = . Making such a change is the

a
basis for a powerful learning rule that we shall describe shortly.

11.4. Q -LEARNING 165
Suppose action in state X leads to state X0 . Then using the denitions
a
of and , it is easy to show that:

Q V
Q (X ) = (X ) +
a r a E V (X0 )]
where (X ) is the average value of the immediate reward received when
r a
we execute action in state X. For an optimal policy (and no others), we

a
have another version of the optimality equation in terms of values: Q
h h ii
Q

(X ) = max
a
a
(X ) + r a E Q

(X0 )a
for all actions, , and states, X. Now, if we had the optimal values (for all
a Q
and X), then we could implement an optimal

policy
simply by selecting
(X0 ) .
a
that action that maximized (X ) + r a E Q

a
That is,

(X) = arg max h (X ) + r a E Q
h
(X0 )a
ii
a
Watkins' proposal amounts to a TD(0) method of learning the values. Q
We quote (with minor notational changes) from Watkins & Dayan, 1992,
page 281]:
\In -Learning, the agent's experience consists of a sequence of
Q
distinct stages or episodes. In the -th episode, the agent: i
observes its current state Xi ,

selects using the method described below] and performs
an action i , a
observes the subsequent state X0i ,

receives an immediate reward i, and r
adjusts its i;1 values using a learning factor i , according

Q c
to:
(X ) = (1 ; i ) i;1(X ) + i i +
Qi a c Q a c r Vi ;1 (X0i )]
if X = Xi and = i , a a
= i;1(X )
Q a
otherwise,
where
Vi ;1 (X0 ) = max
b
i;1(X0 )] Q b
is the best the agent thinks it can do from state X0 . :::
The initial values, 0 (X ), for all states and actions

Q Q a
are assumed given."

Using the current values, i (X ), the agent always selects that ac-
Q Q a
tion that maximizes i (X ). Note that only the value corresponding

Q a Q
to the state just exited and the action just taken is adjusted. And that
Q value is adjusted so that it is closer (by an amount determined by i ) c
to the sum of the immediate reward plus the discounted maximum (over
all actions) of the values of the state just entered. If we imagine the
Q
Q values to be predictions of ultimate (innite horizon) total reward, then

the learning procedure described above is exactly a TD(0) method of learn-
ing how to predict these values. learning strengthens the usual TD
Q Q
methods, however, because TD (applied to reinforcement problems using

value iteration) requires a one-step lookahead, using a model of the eects
of actions, whereas learning does not.
Q
A convenient notation (proposed by Schwartz, 1993]) for representing

the change in value is:
Q
Q (X ) ;

+
a r V (X0 )
where (X ) is the new value for input X and action , is the imme-
diate reward when action is taken in response to input X, (X0 ) is the
Q a Q a r
a V
maximum (over all actions) of the value of the state next reached when
Q
action is taken from state X, and is the fraction of the way toward
which the new value, (X ), is adjusted to equal + (X0 ).
a
Q Q a r V
Watkins and Dayan Watkins & Dayan, 1992] prove that, under certain
conditions, the values computed by this learning procedure converge to
Q
optimal ones (that is, to ones on which an optimal policy can be based).
We dene i(X ) as the index (episode number) of the -th time that
n a i
action is tried in state X. Then, we have:

a
Theorem 11.1 (Watkins and Dayan) For Markov problems with states
fXg and actions f g, and given bounded rewards j nj , learning rates
a r R
0 n 1, and
c <

11.5. DISCUSSION, LIMITATIONS, AND EXTENSIONS OF Q-LEARNING167
1
X 1h
X i2
c i
n (Xa) = 1 c i
n (Xa) < 1
i=0 i=0
for all X and a, then

Qn (X a) ! Qn (X a) as n ! 1, for all X and a, with probability 1, where

Qn (X a) corresponds to the Q values of an optimal policy.
Again, we quote from Watkins & Dayan, 1992, page 281]:

\The most important condition implicit in the convergence the-
orem is that the sequence of episodes that forms the basis
:::
of learning must include an innite number of episodes for each

starting state and action. This may be considered a strong con-
dition on the way states and actions are selected|however, un-
der the stochastic conditions of the theorem, no method could be
guaranteed to nd an optimal policy under weaker conditions.
Note, however, that the episodes need not form a continuous
sequence|that is the X0 of one episode need not be the X of
the next episode."
The relationships among learning, dynamic programming, and control
Q
are very well described in Barto, Bradtke, & Singh, 1994]. learning is Q
best thought of as a stochastic approximation method for calculating the

Q values. Although the denition of the optimal values for any state
Q
depends recursively on expected values of the values for subsequent states

Q
(and on the expected values of rewards), no expected values are explicitly

computed by the procedure. Instead, these values are approximated by
iterative sampling using the actual stochastic mechanism that produces
successor states.
11.5 Discussion, Limitations, and Exten-

sions of Q-Learning
11.5.1 An Illustrative Example
The Q-learning procedure requires that we maintain a table of (X ) Q a
values for all state-action pairs. In the grid world that we described earlier,
such a table would not be excessively large. We might start with random
entries in the table a portion of such an intial table might be as follows:
X a Q (X )
a r (X ) a
(2,3) w 7 0
(2,3) n 4 0
(2,3) e 3 0
(2,3) s 6 0
(1,3) w 4 -1
(1,3) n 5 0
(1,3) e 2 0
(1,3) s 4 0
Suppose the robot is in cell (2,3). The maximum value occurs for = ,
Q a w
so the robot moves west to cell (1,3)|receiving no immediate reward. The

maximum value in cell (1,3) is 5, and the learning mechanism attempts
Q
to make the value of ((2 3) ) closer to the discounted value of 5 plus

Q w
the immediate reward (which was 0 in this case). With a learning rate
parameter = 0 5 and = 0 9, the value of ((2 3) ) is adjusted from
c : : Q Q w
7 to 5.75. No other changes are made to the table at this episode. The
reader might try this learning procedure on the grid world with a simple
computer program. Notice that an optimal policy might not be discovered
if some cells are not visited nor some actions not tried frequently enough.
The learning problem faced by the agent is to associate specic actions
with specic input patterns. learning gradually reinforces those actions
Q
that contribute to positive rewards by increasing the associated values. Q
Typically, as in this example, rewards occur somewhat after the actions

that lead to them|hence the phrase delayed-reinforcement learning. One
can imagine that better and better approximations to the optimal values Q
gradually propagate back from states producing rewards toward all of the
other states that the agent frequently visits. With random values to Q
begin, the agent's actions amount to a random walk through its space of
states. Only when this random walk happens to stumble into rewarding
states does learning begin to produce values that are useful, and, even
Q Q
then, the values have to work their way outward from these rewarding
Q
states. The general problem of associating rewards with state-action pairs

is called the temporal credit assignment problem|how should credit for a
reward be apportioned to the actions leading up to it? learning is, to date,
Q
the most successful technique for temporal credit assignment, although a

related method, called the bucket brigade algorithm, has been proposed by
Holland, 1986].
Learning problems similar to that faced by the agent in our grid world
have been thoroughly studied by Sutton who has proposed an architecture,
called DYNA, for solving them Sutton, 1990]. DYNA combines reinforce-
ment learning with planning. Sutton characterizes planning as learning
in a simulated world that models the world that the agent inhabits. The
agent's model of the world is obtained by learning in its actual world,
Q
and planning is accomplished by learning in its model of the world.

Q
We should note that the learning problem faced by our grid-world robot
could be modied to have several places in the grid that give positive re-
wards. This possibility presents an interesting way to generalize the clas-
sical notion of a \goal" in AI planning systems|even in those that do no
learning. Instead of representing a goal as a condition to be achieved, we
represent a \goal structure" as a set of rewards to be given for achiev-
ing various conditions. Then, the generalized \goal" becomes maximizing
discounted future reward instead of simply achieving some particular con-
dition. This generalization can be made to encompass so-called goals of
maintenance and goals of avoidance. The example presented above in-
cluded avoiding bumping into the grid-world boundary. A goal of mainte-
nance, of a particular state, could be expressed in terms of a reward that
was earned whenever the agent was in that state and performed an action
that transitioned back to that state in one step.
11.5.2 Using Random Actions

When the next pattern presentation in a sequence of patterns is the one
caused by the agent's own action in response to the last pattern, we have
what is called an on-line learning method. In Watkins and Dayan's ter-
minology, in on-line learning the episodes form a continous sequence. As
already mentioned, the convergence theorem for learning does not require
Q
on-line learning indeed, special precautions must be taken to ensure that

on-line learning meets the conditions of the theorem. If on-line learning
discovers some good paths to rewards, the agent may xate on these and
never discover a policy that leads to a possibly greater long-term reward.
In reinforcement learning phraseology, this problem is referred to as the
problem of exploitation (of already learned behavior) versus exploration (of
possibly better behavior).
One way to force exploration is to perform occasional random actions
(instead of that single action prescribed by the current values). For
Q
example, in the grid-world problem, one could imagine selecting an action

randomly according to a probability distribution over the actions ( n e s
and ). This distribution, in turn, could depend on the values. For

w Q
example, we might rst nd that action prescribed by the values and
Q

then choose that action with probability 1/2, choose the two orthogonal
actions with probability 3/16 each, and choose the opposite action with
probability 1/8. This policy might be modied by \simulated annealing"
which would gradually increase the probability of the action prescribed by
the values more and more as time goes on. This strategy would favor
Q
exploration at the beginning of learning and exploitation later.

Other methods, also, have been proposed for dealing with exploration,
including making unvisited states intrinsically rewarding and using an \in-
terval estimate," which is related to the uncertainty in the estimate of a
state's value Kaelbling, 1993].
11.5.3 Generalizing Over Inputs

For large problems it would be impractical to maintain a table like that used
in our grid-world example. Various researchers have suggested mechanisms
for computing values, given pattern inputs and actions. One method
Q
that suggests itself is to use a neural network. For example, consider the
simple linear machine shown in Fig. 11.4.
trainable weights
Σ Q(a1, X)
Σ Q(a2, X)
...
X Wi
Σ Q(ai, X) = X . Wi
...
Σ Q(aR, X)
R dot product units
Figure 11.4: A Net that Computes Q Values

Such a neural net could be used by an agent that has actions to select
R
from. The values (as a function of the input pattern X and the action
Q
a i ) are computed as dot products of weight vectors (one for each action)
and the input vector. Weight adjustments are made according to a TD(0)
procedure to bring the value for the action last selected closer to the sum
Q
of the immediate reward (if any) and the (discounted) maximum value Q
for the next input pattern.

If the optimum values for the problem (whatever they might be)
Q
are more complex than those that can be computed by a linear machine, a
layered neural network might be used. Sigmoid units in the nal layer would
compute values in the range 0 to 1. The TD(0) method for updating
Q Q
values would then have to be combined with a multi-layer weight-changing

rule, such as backpropagation.
Networks of this sort are able to aggregate dierent input vectors into
regions for which the same action should be performed. This kind of aggre-
gation is an example of what has been called structural credit assignment.
Combining TD( ) and backpropagation is a method for dealing with both

the temporal and the structural credit assignment problems.

Interesting examples of delayed-reinforcement training of simulated and
actual robots requiring structural credit assignment have been reported by
Lin, 1992, Mahadevan & Connell, 1992].
11.5.4 Partially Observable States

So far, we have identied the input vector, X, with the actual state of the
environment. When the input vector results from an agent's perceptual
apparatus (as we assume it does), there is no reason to suppose that it
uniquely identies the environmental state. Because of inevitable percep-
tual limitations, several dierent environmental states might give rise to
the same input vector. This phenomenon has been referred to as percep-
tual aliasing. With perceptual aliasing, we can no longer guarantee that
Q learning will result in even useful action policies, let alone optimal ones.
Several researchers have attempted to deal with this problem using a va-
riety of methods including attempting to model \hidden" states by using
internal memory Lin, 1993]. That is, if some aspect of the environment
cannot be sensed currently, perhaps it was sensed once and can be remem-
bered by the agent. When such is the case, we no longer have a Markov
problem that is, the next X vector, given any action, may depend on a
sequence of previous ones rather than just the immediately preceding one.
It might be possible to reinstate a Markov framework (over the X's) if X
includes not only current sensory precepts but information from the agent's
memory.
11.5.5 Scaling Problems

Several di culties have so far prohibited wide application of reinforcement
learning to large problems. (The TD-gammon program, mentioned in the
last chapter, is probably unique in terms of success on a high-dimensional
problem.) We have already touched on some di culties these and others
are summarized below with references to attempts to overcome them.
1. Exploration versus exploitation.
use random actions
favor states not visited recently
separate the learning phase from the use phase
employ a teacher to guide exploration
2. Slow time to convergence
combine learning with prior knowledge use estimates of values
Q
(rather than random values) initially

use a hierarchy of actions learn primitive actions rst and freeze
the useful sequences into macros and then learn how to use the
macros
employ a teacher use graded \lessons"|starting near the re-
wards and then backing away, and use examples of good behavior
Lin, 1992]
use more e cient computations e.g. do several updates per
episode Moore & Atkeson, 1993]
3. Large state spaces
use hand-coded features
use neural networks
use nearest-neighbor methods Moore, 1990]
4. Temporal discounting problems. Using small can make the learner

too greedy for present rewards and indierent to the future but using
large slows down learning.


use a learning method based on average rewards Schwartz, 1993]
5. No \transfer" of learning . What is learned depends on the reward
structure if the rewards change, learning has to start over.
Separate the learning into two parts: learn an \action model"
which predicts how actions change states (and is constant over
all problems), and then learn the \values" of states by reinforce-
ment learning for each dierent set of rewards. Sometimes the
reinforcement learning part can be replaced by a \planner" that
uses the action model to produce plans to achieve goals.
Also see other articles in the special issue on reinforcement learning:
Machine Learning, 8, May, 1992.

To be added.


Chapter 12
Explanation-Based
Learning
12.1 Deductive Learning

In the learning methods studied so far, typically the training set does not
exhaust the version space. Using logical terminology, we could say that
the classier's output does not logically follow from the training set. In
this sense, these methods are inductive. In logic, a deductive system is one
whose conclusions logically follow from a set of input facts, if the system is
sound.1
To contrast inductive with deductive systems in a logical setting, sup-
pose we have a set of facts (the training set) that includes the following
formulas:
fRound(Obj1) Round(Obj2) Round(Obj3) Round(Obj4)
Ball(Obj1) Ball(Obj2) Ball(Obj3) Ball(Obj4)g
A learning system that forms the conclusion (8x)Ball(x) Round(x)]
is inductive. This conclusion may be useful (if there are no facts of the
form Ball() ^ :Round()), but it does not logically follow from the facts.
On the other hand, if we had the facts Green(Obj5) and Green(Obj5)
1 Logical reasoningsystems that are not sound, for examplethose using non-monotonic
reasoning, themselves might produce inductive conclusions that do not logically follow
from the input facts.
175
176 CHAPTER 12. EXPLANATION-BASED LEARNING
Round(Obj5), then we could logically conclude Round(Obj5). Making this
conclusion and saving it is an instance of deductive learning|a topic we
study in this chapter.
Suppose that some logical proposition, , logically follows from some
set of facts, . Under what circumstances might we say that the process
of deducing from results in our learning ? In a sense, we implicitly
knew all along, since it was inherent in knowing . Yet, might not be
obvious given , and the deduction process to establish might have been
arduous. Rather than have to deduce again, we might want to save it,
perhaps along with its deduction, in case it is needed later. Shouldn't that
process count as learning? Dietterich Dietterich, 1990] has called this type
of learning speed-up learning.
Strictly speaking, speed-up learning does not result in a system being
able to make decisions that, in principle, could not have been made before
the learning took place. Speed-up learning simply makes it possible to make
those decisions more eciently. But, in practice, this type of learning might
make possible certain decisions that might otherwise have been infeasible.
To take an extreme case, a chess player can be said to learn chess even
though optimal play is inherent in the rules of chess. On the surface, there
seems to be no real dierence between the experience-based hypotheses
that a chess player makes about what constitutes good play and the kind
of learning we have been studying so far.
As another example, suppose we are given some theorems about geom-
etry and are asked to prove that the sum of the angles of a right triangle
is 180 degrees. Let us further suppose that the proof we constructed did
not depend on the given triangle being a right triangle in that case we
can learn a more general fact. The learning technique that we are going to
study next is related to this example. It is called explanation-based learning
(EBL). EBL can be thought of as a process in which implicit knowledge is
converted into explicit knowledge.
In EBL, we specialize parts of a domain theory to explain a particular
example, then we generalize the explanation to produce another element of
the domain theory that will be useful on similar examples. This process is
illustrated in Fig. 12.1.
12.2 Domain Theories

Two types of information were present in the inductive methods we have
studied: the information inherent in the training samples and the infor-
mation about the domain that is implied by the \bias" (for example, the
12.2. DOMAIN THEORIES 177
Domain
Theory
specialize
Example Prove: X is P
(X is P)
Complex Proof
Process
Explanation
(Proof)
generalize
A New Domain Rule:

Things "like" X are P
Y is like X Trivial Proof
Y is P
Figure 12.1: The EBL Process
hypothesis set from which we choose functions). The learning methods are
successful only if the hypothesis set is appropriate for the problem. Typi-
cally, the smaller the hypothesis set (that is, the more a priori information
we have about the function being sought), the less dependent we are on
information being supplied by a training set (that is, fewer samples). A
priori information about a problem can be expressed in several ways. The
methods we have studied so far restrict the hypotheses in a rather direct
way. A less direct method involves making assertions in a logical language
about the property we are trying to learn. A set of such assertions is usually
called a \domain theory."
Suppose, for example, that we wanted to classify people according to
whether or not they were good credit risks. We might represent a person
by a set of properties (income, marital status, type of employment, etc.),
assemble such data about people who are known to be good and bad credit
risks and train a classier to make decisions. Or, we might go to a loan
ocer of a bank, ask him or her what sorts of things s/he looks for in
making a decision about a loan, encode this knowledge into a set of rules
for an expert system, and then use the expert system to make decisions.
The knowledge used by the loan ocer might have originated as a set
of \policies" (the domain theory), but perhaps the application of these
policies were specialized and made more ecient through experience with
the special cases of loans made in his or her district.
12.3 An Example
To make our discussion more concrete, let's consider the following fanciful
example. We want to nd a way to classify robots as \robust" or not. The
attributes that we use to represent a robot might include some that are
relevant to this decision and some that are not.
Suppose we have a domain theory of logical sentences that taken to-
gether, help to dene whether or not a robot can be classied as robust.
(The same domain theory may be useful for several other purposes also,
but among other things, it describes the concept \robust.")
In this example, let's suppose that our domain theory includes the sen-
tences:
Fixes(u u) Robust(u)
(An individual that can x itself is robust.)
Sees(x y) ^ Habile(x) F ixes(x y)
(A habile individual that can see another entity can x that entity.)
Robot(w) Sees(w w)
(All robots can see themselves.)
R2D2(x) Habile(x)
(R2D2-class individuals are habile.)
C3PO(x) Habile(x)
12.3. AN EXAMPLE 179
(C3PO-class individuals are habile.)
:::
(By convention, variables are assumed to be universally quantied.) We
could use theorem-proving methods operating on this domain theory to
conclude whether certain robots are robust. These methods might be com-
putationally quite expensive because extensive search may have to be per-
formed to derive a conclusion. But after having found a proof for some
particular robot, we might be able to derive some new sentence whose use
allows a much faster conclusion.
We next show how such a new rule might be derived in this example.
Suppose we are given a number of facts about Num5, such as:
Robot(Num5)
R2D2(Num5)
Age(Num5 5)
Manufacturer(Num5 GR)
:::
We are also told that Robust(Num5) is true, but we nevertheless attempt
to nd a proof of that assertion using these facts about Num5 and the
domain theory. The facts about Num5 correspond to the features that we
might use to represent Num5. In this example, not all of them are relevant
to a decision about Robust(Num5). The relevant ones are those used or
needed in proving Robust(Num5) using the domain theory. The proof tree
in Fig. 12.2 is one that a typical theorem-proving system might produce.
In the language of EBL, this proof is an explanation for the fact
Robust(Num5). We see from this explanation that the only facts about
Num5 that were used were Robot(Num5) and R2D2(Num5). In fact, we
could construct the following rule from this explanation:
Robot(Num5) ^ R2D2(Num5) Robust(Num5)
The explanation has allowed us to prune some attributes about Num5 that
are irrelevant (at least for deciding Robust(Num5)). This type of pruning is
the rst sense in which an explanation is used to generalize the classication
problem. (DeJong & Mooney, 1986] call this aspect of explanation-based
Robust(Num5)
Fixes(u, u) => Robust(u)
Fixes(Num5, Num5)
Sees(x,y) & Habile(x)

=> Fixes(x,y)
Sees(Num5,Num5) Habile(Num5)
R2D2(x)
Robot(w)
=> Habile(x)
=> Sees(w,w)
Robot(Num5) R2D2(Num5)
Figure 12.2: A Proof Tree
learning feature elimination.) But the rule we extracted from the expla-
nation applies only to Num5. There might be little value in learning that
rule since it is so specic. Can it be generalized so that it can be applied
to other individuals as well?
Examination of the proof shows that the same proof structure, using
the same sentences from the domain theory, could be used independently
of whether we are talking about Num5 or some other individual. We can
generalize the proof by a process that replaces constants in the tip nodes
of the proof tree with variables and works upward|using unication to
constrain the values of variables as needed to obtain a proof.
In this example, we replace Robot(Num5) by Robot(r) and R2D2(Num5)
by R2D2(s) and redo the proof|using the explanation proof as a template.
Note that we use dierent values for the two dierent occurrences of Num5
at the tip nodes. Doing so sometimes results in more general, but never-
theless valid rules. We now apply the rules used in the proof in the forward
direction, keeping track of the substitutions imposed by the most general
12.3. AN EXAMPLE 181
uniers used in the proof. (Note that we always substitute terms that are
already in the tree for variables in rules.) This process results in the gener-
alized proof tree shown in Fig. 12.3. Note that the occurrence of Sees(r r)
as a node in the tree forces the unication of x with y in the domain rule,
Sees(x y) ^ Habile(y) Fixes(x y). The substitutions are then applied
to the variables in the tip nodes and the root node to yield the general rule:
Robot(r) ^ R2D2(r) Robust(r).
Robust(r)
Fixes(u, u) => Robust(u)

{r/u}
Fixes(r, r)
Sees(x,y) & Habile(x)

=> Fixes(x,y)
{r/x, r/y, r/s}
Sees(r,r) Habile(s)
{s/x} R2D2(x)
{r/w} Robot(w)
=> Habile(x)
=> Sees(w,w)
Robot(r) R2D2(s)
becomes R2D2(r) after

applying {r/s}
Figure 12.3: A Generalized Proof Tree

This rule is the end result of EBL for this example. The process
by which Num5 in this example was generalized to a variable is what
DeJong & Mooney, 1986] call identity elimination (the precise identity of
Num5 turned out to be irrelevant). (The generalization process described
in this example is based on that of DeJong & Mooney, 1986] and dif-
fers from that of Mitchell, et al., 1986]. It is also similar to that used
in Fikes, et al., 1972].) Clearly, under certain assumptions, this general
rule is more easily used to conclude Robust about an individual than the
original proof process was.
It is important to note that we could have derived the general rule from
the domain theory without using the example. (In the literature, doing so is
called static analysis Etzioni, 1991].) In fact, the example told us nothing
new other than what it told us about Num5. The sole role of the example
in this instance of EBL was to provide a template for a proof to help guide
the generalization process. Basing the generalization process on examples
helps to insure that we learn rules matched to the distribution of problems
that occur.
There are a number of qualications and elaborations about EBL that
need to be mentioned.
12.4 Evaluable Predicates

The domain theory includes a number of predicates other than the one
occuring in the formula we are trying to prove and other than those that
might customarily be used to describe an individual. One might note, for
example, that if we used Habile(Num5) to describe Num5, the proof would
have been shorter. Why didn't we? The situation is analogous to that of
using a data base augmented by logical rules. In the latter application, the
formulas in the actual data base are \extensional," and those in the logical
rules are \intensional." This usage reects the fact that the predicates in
the data base part are dened by their extension|we explicitly list all the
tuples sastisfying a relation. The logical rules serve to connect the data base
predicates with higher level abstractions that are described (if not dened)
by the rules. We typically cannot look up the truth values of formulas
containing these intensional predicates they have to be derived using the
rules and the database.
The EBL process assumes something similar. The domain theory is use-
ful for connecting formulas that we might want to prove with those whose
truth values can be \looked up" or otherwise evaluated. In the EBL lit-
erature, such formulas satisfy what is called the operationality criterion.
Perhaps another analogy might be to neural networks. The evaluable pred-
icates correspond to the components of the input pattern vector the pred-
icates in the domain theory correspond to the hidden units. Finding the
new rule corresponds to nding a simpler expression for the formula to be
proved in terms only of the evaluable predicates.
12.5. MORE GENERAL PROOFS 183
12.5 More General Proofs
Examining the domain theory for our example reveals that an alternative
rule might have been: Robot(u) ^ C3PO(u) Robust(u). Such a rule
might have resulted if we were given fC3PO(Num6) Robot(Num6) : : :g
and proved Robust(Num6). After considering these two examples (Num5
and Num6), the question arises, do we want to generalize the two rules to
something like: Robot(u) ^ C3P O(u) _ R2D2(u)] Robust(u)? Doing so is
an example of what DeJong & Mooney, 1986] call structural generalization
(via disjunctive augmentation ).
Adding disjunctions for every alternative proof can soon become cum-
bersome and destroy any eciency advantage of EBL. In our example,
the eciency might be retrieved if there were another evaluable predicate,
say, Bionic(u) such that the domain theory also contained R2D2(x)
Bionic(x) and C3PO(x) Bionic(x). After seeing a number of sim-
ilar examples, we might be willing to induce the formula Bionic(u)
C3PO(u) _ R2D2(u)] in which case the rule with the disjunction could
be replaced with Robot(u) ^ Bionic(u) Robust(u).
12.6 Utility of EBL

It is well known in theorem proving that the complexity of nding a proof
depends both on the number of formulas in the domain theory and on the
depth of the shortest proof. Adding a new rule decreases the depth of the
shortest proof but it also increases the number of formulas in the domain
theory. In realistic applications, the added rules will be relevant for some
tasks and not for others. Thus, it is unclear whether the overall utility of
the new rules will turn out to be positive. EBL methods have been applied
in several settings, usually with positive utility. (See Minton, 1990] for an
analysis).
12.7 Applications
There have been several applications of EBL methods. We mention two
here, namely the formation of macro-operators in automatic plan generation
and learning how to control search.
12.7.1 Macro-Operators in Planning
In automatic planning systems, eciency can sometimes be enhanced by
chaining together a sequence of operators into macro-operators. We show
an example of a process for creating macro-operators based on techniques
explored by Fikes, et al., 1972].
Referring to Fig. 12.4, consider the problem of nding a plan for a
robot in room R1 to fetch a box, B1, by going to an adjacent room, R2,
and pushing it back to R1. The goal for the robot is INROOM(B1 R1),
and the facts that are true in the initial state are listed in the gure.
R1 R2
D1
B1
D2
R3
Initial State:
INROOM(ROBOT, R1)
INROOM(B1,R2)
CONNECTS(D1,R1,R2)
CONNECTS(D1,R2,R1)
...
Figure 12.4: Initial State of a Robot Problem

We will construct the plan from a set of STRIPS operators that include:
GOTHRU(d r1 r2)
Preconditions: INROOM(ROBOT r1) CONNECT S(d r1 r2)
Delete list: INROOM(ROBOT r1)
Add list: INROOM(ROBOT r2)

12.7. APPLICATIONS 185
PUSHTHRU(b d r1 r2)
Preconditions: INROOM(ROBOT r1) CONNECTS(d r1 r2) INROOM(b r1)
Delete list: INROOM(ROBOT r1) INROOM(b r1)
Add list: INROOM(ROBOT r2) INROOM(b r2)
A backward-reasoning STRIPS system might produce the plan shown

in Fig. 12.5. We show there the main goal and the subgoals along a solution
path. (The conditions in each subgoal that are true in the initial state are
shown underlined.) The preconditions for this plan, true in the initial state,
are:
INROOM(ROBOT R1)
CONNECT S(D1 R1 R2)
CONNECT S(D1 R2 R1)
INROOM(B1 R2)
Saving this specic plan, valid only for the specic constants it mentions,
would not be as useful as would be saving a more general one. We rst
generalize these preconditions by substituting variables for constants. We
then follow the structure of the specic plan to produce the generalized
plan shown in Fig. 12.6 that achieves INROOM(b1 r4). Note that the
generalized plan does not require pushing the box back to the place where
the robot started. The preconditions for the generalized plan are:
INROOM(ROBOT r1)
CONNECTS(d1 r1 r2)
CONNECTS(d2 r2 r4)
INROOM(b r4)
Another related technique that chains together sequences of oper-
ators to form more general ones is the chunking mechanism in Soar
Laird, et al., 1986].
INROOM(B1,R1)
PUSHTHRU(B1,d,r1,R1)
R1 R2
INROOM(ROBOT, r1), INROOM(ROBOT, R2),

D1
CONNECTS(d, r1, R1), CONNECTS(D1, R2, R1),
INROOM(B1, r1) {R2/r1, INROOM(B1, R2)
D1/d} B1
D2
GOTHRU(d2, r3, R2) R3
INROOM(ROBOT, r3),
CONNECTS(d2, r3, R2),
CONNECTS(D1, R2, R1),
PLAN:
INROOM(B1, R2)
{R1/r3, D1/d2}
GOTHRU(D1,R1,R2)
PUSHTHRU(B1,D1,R2,R1)
INROOM(ROBOT, R1),
INROOM(B1, R2)
Figure 12.5: A Plan for the Robot Problem
12.7.2 Learning Search Control Knowledge

Besides their use in creating macro-operators, EBL methods can be used to
improve the eciency of planning in another way also. In his system called
PRODIGY, Minton proposed using EBL to learn eective ways to control
search Minton, 1988]. PRODIGY is a STRIPS-like system that solves
planning problems in the blocks-world, in a simple mobile robot world, and
in job-shop scheduling. PRODIGY has a domain theory involving both the
domain of the problem and a simple (meta) theory about planning. Its
meta theory includes statements about whether a control choice about a
subgoal to work on, an operator to apply, etc. either succeeds or fails. After
producing a plan, it analyzes its successful and its unsuccessful choices and
attempts to explain them in terms of its domain theory. Using an EBL-like
process, it is able to produce useful control rules such as:

INROOM(b1,r4)
PUSHTHRU(b1,d2,r2,r4)
INROOM(ROBOT, r2),
CONNECTS(d1, r1, r2),
INROOM(b1, r4)
GOTHRU(d1, r1, r2)
INROOM(ROBOT, r1),
INROOM(b1, r4)
Figure 12.6: A Generalized Plan
IF (AND (CURRENT ; NODE node)
(CANDIDATE ; GOAL node (ON x y))

(CANDIDATE ; GOAL node (ON y z)))
THEN (PREFER GOAL (ON y z) TO (ON x y))
PRODIGY keeps statistics on how often these learned rules are used, their
savings (in time to nd plans), and their cost of application. It saves
only the rules whose utility, thus measured, is judged to be high. Minton
Minton, 1990] has shown that there is an overall advantage of using these
rules (as against not having any rules and as against hand-coded search
control rules).

To be added.


Bibliography
Acorn & Walden, 1992] Acorn, T., and Walden, S., \SMART: Support
Management Automated Reasoning Technology for COMPAQ Cus-
tomer Service," Proc. Fourth Annual Conf. on Innovative Applica-
tions of Articial Intelligence, Menlo Park, CA: AAAI Press, 1992.
Aha, 1991] Aha, D., Kibler, D., and Albert, M., \Instance-Based Learning
Algorithms," Machine Learning, 6, 37-66, 1991.
Anderson & Bower, 1973] Anderson, J. R., and Bower, G. H., Human As-
sociative Memory, Hillsdale, NJ: Erlbaum, 1973.
Anderson, 1958] Anderson, T. W., An Introduction to Multivariate Statis-
tical Analysis, New York: John Wiley, 1958.
Barto, Bradtke, & Singh, 1994] Barto, A., Bradtke, S., and Singh, S.,
\Learning to Act Using Real-Time Dynamic Programming," to ap-
pear in Articial Intelligence, 1994.
Baum & Haussler, 1989] Baum, E, and Haussler, D., \What Size Net
Gives Valid Generalization?" Neural Computation, 1, pp. 151-160,
1989.
Baum, 1994] Baum, E., \When Are k-Nearest Neighbor and Backprop-
agation Accurate for Feasible-Sized Sets of Examples?" in Han-
son, S., Drastal, G., and Rivest, R., (eds.), Computational Learning
Theory and Natural Learning Systems, Volume 1: Constraints and
Prospects, pp. 415-442, Cambridge, MA: MIT Press, 1994.
Bellman, 1957] Bellman, R. E., Dynamic Programming, Princeton:
Princeton University Press, 1957.
Blumer, et al., 1987] Blumer, A., et al., \Occam's Razor," Info. Process.
Lett., vol 24, pp. 377-80, 1987.
189
190 BIBLIOGRAPHY
Blumer, et al., 1990] Blumer, A., et al., \Learnability and the Vapnik-
Chervonenkis Dimension," JACM, 1990.
Bollinger & Due, 1988] Bollinger, J., and Due, N., Computer Control
of Machines and Processes, Reading, MA: Addison-Wesley, 1988.
Brain, et al., 1962] Brain, A. E., et al., \Graphical Data Processing Re-
search Study and Experimental Investigation," Report No. 8 (pp.
9-13) and No. 9 (pp. 3-10), Contract DA 36-039 SC-78343, SRI In-
ternational, Menlo Park, CA, June 1962 and September 1962.
Breiman, et al., 1984] Breiman, L., Friedman, J., Olshen, R., and Stone,
C., Classication and Regression Trees, Monterey, CA: Wadsworth,
1984.
Brent, 1990] Brent, R. P., \Fast Training Algorithms for Multi-Layer Neu-
ral Nets," Numerical Analysis Project Manuscript NA-90-03, Com-
puter Science Department, Stanford University, Stanford, CA 94305,
March 1990.
Bryson & Ho 1969] Bryson, A., and Ho, Y.-C., Applied Optimal Control,
New York: Blaisdell.
Buchanan & Wilkins, 1993] Buchanan, B. and Wilkins, D., (eds.), Read-
ings in Knowledge Acquisition and Learning, San Francisco: Morgan
Kaufmann, 1993.
Carbonell, 1983] Carbonell, J., \Learning by Analogy," in Machine Learn-
ing: An Articial Intelligence Approach, Michalski, R., Carbonell,
J., and Mitchell, T., (eds.), San Francisco: Morgan Kaufmann, 1983.
Cheeseman, et al., 1988] Cheeseman, P., et al., \AutoClass: A Bayesian
Classication System," Proc. Fifth Intl. Workshop on Machine
Learning, Morgan Kaufmann, San Mateo, CA, 1988. Reprinted in
Shavlik, J. and Dietterich, T., Readings in Machine Learning, Mor-
gan Kaufmann, San Francisco, pp. 296-306, 1990.
Cover & Hart, 1967] Cover, T., and Hart, P., \Nearest Neighbor Pat-
tern Classication," IEEE Trans. on Information Theory, 13, 21-27,
1967.
Cover, 1965] Cover, T., \Geometrical and Statistical Properties of Systems
of Linear Inequalities with Applications in Pattern Recognition,"
IEEE Trans. Elec. Comp., EC-14, 326-334, June, 1965.
BIBLIOGRAPHY 191
Dasarathy, 1991] Dasarathy, B. V., Nearest Neighbor Pattern Classica-
tion Techniques, IEEE Computer Society Press, 1991.
Dayan & Sejnowski, 1994] Dayan, P., and Sejnowski, T., \T D( ) Con-
verges with Probability 1," Machine Learning, 14, pp. 295-301, 1994.
Dayan, 1992] Dayan, P., \The Convergence of TD( ) for General ," Ma-
chine Learning, 8, 341-362, 1992.
DeJong & Mooney, 1986] DeJong, G., and Mooney, R., \Explanation-
Based Learning: An Alternative View," Machine Learning, 1:145-
176, 1986. Reprinted in Shavlik, J. and Dietterich, T., Readings
in Machine Learning, San Francisco: Morgan Kaufmann, 1990, pp
452-467.
Dietterich & Bakiri, 1991] Dietterich, T. G., and Bakiri, G., \Error-
Correcting Output Codes: A General Method for Improving Mul-
ticlass Inductive Learning Programs," Proc. Ninth Nat. Conf. on
A.I., pp. 572-577, AAAI-91, MIT Press, 1991.
Dietterich, et al., 1990] Dietterich, T., Hild, H., and Bakiri, G., \A Com-
parative Study of ID3 and Backpropagation for English Text-to-
Speech Mapping," Proc. Seventh Intl. Conf. Mach. Learning, Porter,
B. and Mooney, R. (eds.), pp. 24-31, San Francisco: Morgan Kauf-
mann, 1990.
Dietterich, 1990] Dietterich, T., \Machine Learning," Annu. Rev. Comput.
Sci., 4:255-306, Palo Alto: Annual Reviews Inc., 1990.
Duda & Fossum, 1966] Duda, R. O., and Fossum, H., \Pattern Classica-
tion by Iteratively Determined Linear and Piecewise Linear Discrim-
inant Functions," IEEE Trans. on Elect. Computers, vol. EC-15, pp.
220-232, April, 1966.
Duda & Hart, 1973] Duda, R. O., and Hart, P.E., Pattern Classication
and Scene Analysis, New York: Wiley, 1973.
Duda, 1966] Duda, R. O., \Training a Linear Machine on Mislabeled
Patterns," SRI Tech. Report prepared for ONR under Contract
3438(00), SRI International, Menlo Park, CA, April 1966.
Efron, 1982] Efron, B., The Jackknife, the Bootstrap and Other Resam-
pling Plans, Philadelphia: SIAM, 1982.
192 BIBLIOGRAPHY
Ehrenfeucht, et al., 1988] Ehrenfeucht, A., et al., \A General Lower
Bound on the Number of Examples Needed for Learning," in Proc.
1988 Workshop on Computational Learning Theory, pp. 110-120,
San Francisco: Morgan Kaufmann, 1988.
Etzioni, 1991] Etzioni, O., \STATIC: A Problem-Space Compiler for
PRODIGY," Proc. of Ninth National Conf. on Articial Intelli-
gence, pp. 533-540, Menlo Park: AAAI Press, 1991.
Etzioni, 1993] Etzioni, O., \A Structural Theory of Explanation-Based
Learning," Articial Intelligence, 60:1, pp. 93-139, March, 1993.
Evans & Fisher, 1992] Evans, B., and Fisher, D., Process Delay Analyses
Using Decision-Tree Induction, Tech. Report CS92-06, Department
of Computer Science, Vanderbilt University, TN, 1992.
Fahlman & Lebiere, 1990] Fahlman, S., and Lebiere, C., \The Cascade-
Correlation Learning Architecture," in Touretzky, D., (ed.), Ad-
vances in Neural Information Processing Systems, 2, pp. 524-532,
Fayyad, et al., 1993] Fayyad, U. M., Weir, N., and Djorgovski, S., \SKI-
CAT: A Machine Learning System for Automated Cataloging of
Large Scale Sky Surveys," in Proc. Tenth Intl. Conf. on Machine
Learning, pp. 112-119, San Francisco: Morgan Kaufmann, 1993.
(For a longer version of this paper see: Fayyad, U. Djorgovski, G.,
and Weir, N., \Automating the Analysis and Cataloging of Sky Sur-
veys," in Fayyad, U., et al.(eds.), Advances in Knowledge Discovery
and Data Mining, Chapter 19, pp. 471., Cambridge: The MIT
Press, March, 1996.)
Feigenbaum, 1961] Feigenbaum, E. A., \The Simulation of Verbal Learn-
ing Behavior," Proceedings of the Western Joint Computer Confer-
ence, 19:121-132, 1961.
Fikes, et al., 1972] Fikes, R., Hart, P., and Nilsson, N., \Learning and
Executing Generalized Robot Plans," Articial Intelligence, pp 251-
288, 1972. Reprinted in Shavlik, J. and Dietterich, T., Readings in
Machine Learning, San Francisco: Morgan Kaufmann, 1990, pp 468-
486.
Fisher, 1987] Fisher, D., \Knowledge Acquisition via Incremental Concep-
tual Clustering," Machine Learning, 2:139-172, 1987. Reprinted in
Shavlik, J. and Dietterich, T., Readings in Machine Learning, San
Francisco: Morgan Kaufmann, 1990, pp. 267{283.
BIBLIOGRAPHY 193
Friedman, et al., 1977] Friedman, J. H., Bentley, J. L., and Finkel, R. A.,
\An Algorithm for Finding Best Matches in Logarithmic Expected
Time," ACM Trans. on Math. Software, 3(3):209-226, September
1977.
Fu, 1994] Fu, L., Neural Networks in Articial Intelligence, New York:
McGraw-Hill, 1994.
Gallant, 1986] Gallant, S. I., \Optimal Linear Discriminants," in Eighth
International Conf. on Pattern Recognition, pp. 849-852, New York:
IEEE, 1986.
Genesereth & Nilsson, 1987] Genesereth, M., and Nilsson, N., Logical
Foundations of Articial Intelligence, San Francisco: Morgan Kauf-
mann, 1987.
Gluck & Rumelhart, 1989] Gluck, M. and Rumelhart, D., Neuroscience
and Connectionist Theory, The Developments in Connectionist The-
ory, Hillsdale, NJ: Erlbaum Associates, 1989.
Hammerstrom, 1993] Hammerstrom, D., \Neural Networks at Work,"
IEEE Spectrum, pp. 26-32, June 1993.
Haussler, 1988] Haussler, D., \Quantifying Inductive Bias: AI Learning
Algorithms and Valiant's Learning Framework," Articial Intelli-
gence, 36:177-221, 1988. Reprinted in Shavlik, J. and Dietterich, T.,
Readings in Machine Learning, San Francisco: Morgan Kaufmann,
1990, pp. 96-107.
Haussler, 1990] Haussler, D., \Probably Approximately Correct Learn-
ing," Proc. Eighth Nat. Conf. on AI, pp. 1101-1108. Cambridge,
MA: MIT Press, 1990.
Hebb, 1949] Hebb, D. O., The Organization of Behaviour, New York: John
Wiley, 1949.
Hertz, Krogh, & Palmer, 1991] Hertz, J., Krogh, A, and Palmer, R., In-
troduction to the Theory of Neural Computation, Lecture Notes, vol.
1, Santa Fe Inst. Studies in the Sciences of Complexity, New York:
Addison-Wesley, 1991.
Hirsh, 1994] Hirsh, H., \Generalizing Version Spaces," Machine Learning,
17, 5-45, 1994.
194 BIBLIOGRAPHY
Holland, 1975] Holland, J., Adaptation in Natural and Articial Systems,
Ann Arbor: The University of Michigan Press, 1975. (Second edition
printed in 1992 by MIT Press, Cambridge, MA.)
Holland, 1986] Holland, J. H., \Escaping Brittleness The Possibilities
of General-Purpose Learning Algorithms Applied to Parallel Rule-
Based Systems." In Michalski, R., Carbonell, J., and Mitchell, T.
(eds.) , Machine Learning: An Articial Intelligence Approach, Vol-
ume 2, chapter 20, San Francisco: Morgan Kaufmann, 1986.
Hunt, Marin, & Stone, 1966] Hunt, E., Marin, J., and Stone, P., Experi-
ments in Induction, New York: Academic Press, 1966.
Jabbour, K., et al., 1987] Jabbour, K., et al., \ALFA: Automated Load
Forecasting Assistant," Proc. of the IEEE Pwer Engineering Society
Summer Meeting, San Francisco, CA, 1987.
John, 1995] John, G., \Robust Linear Discriminant Trees," Proc. of the
Conf. on Articial Intelligence and Statistics, Ft. Lauderdale, FL,
January, 1995.
Kaelbling, 1993] Kaelbling, L. P., Learning in Embedded Systems, Cam-
bridge, MA: MIT Press, 1993.
Kohavi, 1994] Kohavi, R., \Bottom-Up Induction of Oblivious Read-Once
Decision Graphs," Proc. of European Conference on Machine Learn-
ing (ECML-94), 1994.
Kolodner, 1993] Kolodner, J., Case-Based Reasoning, San Francisco: Mor-
gan Kaufmann, 1993.
Koza, 1992] Koza, J., Genetic Programming: On the Programming of
Computers by Means of Natural Selection, Cambridge, MA: MIT
Press, 1992.
Koza, 1994] Koza, J., Genetic Programming II: Automatic Discovery of
Reusable Programs, Cambridge, MA: MIT Press, 1994.
Laird, et al., 1986] Laird, J., Rosenbloom, P., and Newell, A., \Chunking
in Soar: The Anatomy of a General Learning Mechanism," Ma-
chine Learning, 1, pp. 11-46, 1986. Reprinted in Buchanan, B. and
Wilkins, D., (eds.), Readings in Knowledge Acquisition and Learn-
ing, pp. 518-535, Morgan Kaufmann, San Francisco, CA, 1993.
Langley, 1992] Langley, P., \Areas of Application for Machine Learning,"
Proc. of Fifth Int'l. Symp. on Knowledge Engineering, Sevilla, 1992.
BIBLIOGRAPHY 195
Langley, 1996] Langley, P., Elements of Machine Learning, San Francisco:
Morgan Kaufmann, 1996.
Lavrac & Dzeroski, 1994] Lavrac, N., and Dzeroski, S., Inductive Logic
Programming, Chichester, England: Ellis Horwood, 1994.
Lin, 1992] Lin, L., \Self-Improving Reactive Agents Based on Reinforce-
ment Learning, Planning, and Teaching," Machine Learning, 8, 293-
321, 1992.
Lin, 1993] Lin, L., \Scaling Up Reinforcement Learning for Robot Con-
trol," Proc. Tenth Intl. Conf. on Machine Learning, pp. 182-189,
Littlestone, 1988] Littlestone, N., \Learning Quickly When Irrelevant At-
tributes Abound: A New Linear-Threshold Algorithm," Machine
Learning 2: 285-318, 1988.
Maass & Turan, 1994] Maass, W., and Turan, G., \How Fast Can a
Threshold Gate Learn?," in Hanson, S., Drastal, G., and Rivest,
R., (eds.), Computational Learning Theory and Natural Learning
Systems, Volume 1: Constraints and Prospects, pp. 381-414, Cam-
Mahadevan & Connell, 1992] Mahadevan, S., and Connell, J., \Auto-
matic Programming of Behavior-Based Robots Using Reinforcement
Learning," Articial Intelligence, 55, pp. 311-365, 1992.
Marchand & Golea, 1993] Marchand, M., and Golea, M., \On Learning
Simple Neural Concepts: From Halfspace Intersections to Neural
Decision Lists," Network, 4:67-85, 1993.
McCulloch & Pitts, 1943] McCulloch, W. S., and Pitts, W. H., \A Logical
Calculus of the Ideas Immanent in Nervous Activity," Bulletin of
Mathematical Biophysics, Vol. 5, pp. 115-133, Chicago: University
of Chicago Press, 1943.
Michie, 1992] Michie, D., \Some Directions in Machine Intelligence," un-
published manuscript, The Turing Institute, Glasgow, Scotland,
1992.
Minton, 1988] Minton, S., Learning Search Control Knowledge: An
Explanation-Based Approach, Kluwer Academic Publishers, Boston,
MA, 1988.
196 BIBLIOGRAPHY
Minton, 1990] Minton, S., \Quantitative Results Concerning the Utility
of Explanation-Based Learning," Articial Intelligence, 42, pp. 363-
392, 1990. Reprinted in Shavlik, J. and Dietterich, T., Readings in
Machine Learning, San Francisco: Morgan Kaufmann, 1990, pp.
573-587.
Mitchell, et al., 1986] Mitchell, T., et al., \Explanation-Based Generaliza-
tion: A Unifying View," Machine Learning, 1:1, 1986. Reprinted in
Shavlik, J. and Dietterich, T., Readings in Machine Learning, San
Francisco: Morgan Kaufmann, 1990, pp. 435-451.
Mitchell, 1982] Mitchell, T., \Generalization as Search," Articial Intelli-
gence, 18:203-226, 1982. Reprinted in Shavlik, J. and Dietterich, T.,
Readings in Machine Learning, San Francisco: Morgan Kaufmann,
1990, pp. 96{107.
Moore & Atkeson, 1993] Moore, A., and Atkeson, C., \Prioritized Sweep-
ing: Reinforcement Learning with Less Data and Less Time," Ma-
chine Learning, 13, pp. 103-130, 1993.
Moore, et al., 1994] Moore, A. W., Hill, D. J., and Johnson, M. P.,
\An Empirical Investigation of Brute Force to Choose Features,
Smoothers, and Function Approximators," in Hanson, S., Judd, S.,
and Petsche, T., (eds.), Computational Learning Theory and Natu-
ral Learning Systems, Vol. 3, Cambridge: MIT Press, 1994.
Moore, 1990] Moore, A., E cient Memory-based Learning for Robot Con-
trol, PhD. Thesis Technical Report No. 209, Computer Laboratory,
University of Cambridge, October, 1990.
Moore, 1992] Moore, A., \Fast, Robust Adaptive Control by Learning
Only Forward Models," in Moody, J., Hanson, S., and Lippman,
R., (eds.), Advances in Neural Information Processing Systems 4,
Mueller & Page, 1988] Mueller, R. and Page, R., Symbolic Computing with
Lisp and Prolog, New York: John Wiley & Sons, 1988.
Muggleton, 1991] Muggleton, S., \Inductive Logic Programming," New
Generation Computing, 8, pp. 295-318, 1991.
Muggleton, 1992] Muggleton, S., Inductive Logic Programming, London:
Academic Press, 1992.
Muroga, 1971] Muroga, S., Threshold Logic and its Applications, New
York: Wiley, 1971.
BIBLIOGRAPHY 197
Natarjan, 1991] Natarajan, B., Machine Learning: A Theoretical Ap-
proach, San Francisco: Morgan Kaufmann, 1991.
Nilsson, 1965] Nilsson, N. J., \Theoretical and Experimental Investi-
gations in Trainable Pattern-Classifying Systems," Tech. Report
No. RADC-TR-65-257, Final Report on Contract AF30(602)-3448,
Rome Air Development Center (Now Rome Laboratories), Griss
Air Force Base, New York, September, 1965.
Nilsson, 1990] Nilsson, N. J., The Mathematical Foundations of Learning
Machines, San Francisco: Morgan Kaufmann, 1990. (This book is
a reprint of Learning Machines: Foundations of Trainable Pattern-
Classifying Systems, New York: McGraw-Hill, 1965.)
Oliver, Dowe, & Wallace, 1992] Oliver, J., Dowe, D., and Wallace, C., \In-
ferring Decision Graphs using the Minimum Message Length Princi-
ple," Proc. 1992 Australian Articial Intelligence Conference, 1992.
Pagallo & Haussler, 1990] Pagallo, G. and Haussler, D., \Boolean Feature
Discovery in Empirical Learning," Machine Learning, vol.5, no.1,
pp. 71-99, March 1990.
Pazzani & Kibler, 1992] Pazzani, M., and Kibler, D., \The Utility of
Knowledge in Inductive Learning," Machine Learning, 9, 57-94,
1992.
Peterson, 1961] Peterson, W., Error Correcting Codes, New York: John
Wiley, 1961.
Pomerleau, 1991] Pomerleau, D., \Rapidly Adapting Articial Neural
Networks for Autonomous Navigation," in Lippmann, P., et al.
(eds.), Advances in Neural Information Processing Systems, 3, pp.
429-435, San Francisco: Morgan Kaufmann, 1991.
Pomerleau, 1993] Pomerleau, D, Neural Network Perception for Mobile
Robot Guidance, Boston: Kluwer Academic Publishers, 1993.
Quinlan & Rivest, 1989] Quinlan, J. Ross, and Rivest, Ron, \Inferring De-
cision Trees Using the Minimum Description Length Principle," In-
formation and Computation, 80:227{248, March, 1989.
Quinlan, 1986] Quinlan, J. Ross, \Induction of Decision Trees," Machine
Learning, 1:81{106, 1986. Reprinted in Shavlik, J. and Dietterich,
T., Readings in Machine Learning, San Francisco: Morgan Kauf-
mann, 1990, pp. 57{69.
198 BIBLIOGRAPHY
Quinlan, 1987] Quinlan, J. R., \Generating Production Rules from De-
cision Trees," In IJCAI-87: Proceedings of the Tenth Intl. Joint
Conf. on Articial Intelligence, pp. 304-7, San Francisco: Morgan-
Kaufmann, 1987.
Quinlan, 1990] Quinlan, J. R., \Learning Logical Denitions from Rela-
tions," Machine Learning, 5, 239-266, 1990.
Quinlan, 1993] Quinlan, J. Ross, C4.5: Programs for Machine Learning,
Quinlan, 1994] Quinlan, J. R., \Comparing Connectionist and Symbolic
Learning Methods," in Hanson, S., Drastal, G., and Rivest, R.,
(eds.), Computational Learning Theory and Natural Learning Sys-
tems, Volume 1: Constraints and Prospects, pp. 445-456,, Cam-
Ridgway, 1962] Ridgway, W. C., An Adaptive Logic System with General-
izing Properties, PhD thesis, Tech. Rep. 1556-1, Stanford Electronics
Labs., Stanford, CA, April 1962.
Rissanen, 1978] Rissanen, J., \Modeling by Shortest Data Description,"
Automatica, 14:465-471, 1978.
Rivest, 1987] Rivest, R. L., \Learning Decision Lists," Machine Learning,
2, 229-246, 1987.
Rosenblatt, 1958] Rosenblatt, F., Principles of Neurodynamics, Washing-
ton: Spartan Books, 1961.
Ross, 1983] Ross, S., Introduction to Stochastic Dynamic Programming,
New York: Academic Press, 1983.
Rumelhart, Hinton, & Williams, 1986] Rumelhart, D. E., Hinton, G. E.,
and Williams, R. J., \Learning Internal Representations by Error
Propagation," In Rumelhart, D. E., and McClelland, J. L., (eds.)
Parallel Distributed Processing, Vol 1, 318{362, 1986.
Russell & Norvig 1995] Russell, S., and Norvig, P., Articial Intelligence:
A Modern Approach, Englewood Clis, NJ: Prentice Hall, 1995.
Samuel, 1959] Samuel, A., \Some Studies in Machine Learning Using the
Game of Checkers," IBM Journal of Research and Development,
3:211-229, July 1959.
BIBLIOGRAPHY 199
Schwartz, 1993] Schwartz, A., \A Reinforcement Learning Method for
Maximizing Undiscounted Rewards," Proc. Tenth Intl. Conf. on
Machine Learning, pp. 298-305, San Francisco: Morgan Kaufmann,
1993.
Sejnowski, Koch, & Churchland, 1988] Sejnowski, T., Koch, C., and
Churchland, P., \Computational Neuroscience," Science, 241:
1299-1306, 1988.
Shavlik, Mooney, & Towell, 1991] Shavlik, J., Mooney, R., and Towell, G.,
\Symbolic and Neural Learning Algorithms: An Experimental Com-
parison," Machine Learning, 6, pp. 111-143, 1991.
Shavlik & Dietterich, 1990] Shavlik, J. and Dietterich, T., Readings in
Machine Learning, San Francisco: Morgan Kaufmann, 1990.
Sutton & Barto, 1987] Sutton, R. S., and Barto, A. G., \A Temporal-
Dierence Model of Classical Conditioning," in Proceedings of the
Ninth Annual Conference of the Cognitive Science Society, Hillsdale,
NJ: Erlbaum, 1987.
Sutton, 1988] Sutton, R. S., \Learning to Predict by the Methods of Tem-
poral Dierences," Machine Learning 3: 9-44, 1988.
Sutton, 1990] Sutton, R., \Integrated Architectures for Learning, Plan-
ning, and Reacting Based on Approximating Dynamic Program-
ming," Proc. of the Seventh Intl. Conf. on Machine Learning, pp.
Taylor, Michie, & Spiegalhalter, 1994] Taylor, C., Michie, D., and Spiegal-
halter, D., Machine Learning, Neural and Statistical Classication,
Paramount Publishing International.
Tesauro, 1992] Tesauro, G., \Practical Issues in Temporal Dierence
Learning," Machine Learning, 8, nos. 3/4, pp. 257-277, 1992.
Towell & Shavlik, 1992] Towell G., and Shavlik, J., \Interpretation of Ar-
ticial Neural Networks: Mapping Knowledge-Based Neural Net-
works into Rules," in Moody, J., Hanson, S., and Lippmann, R.,
(eds.), Advances in Neural Information Processing Systems, 4, pp.
Towell, Shavlik, & Noordweier, 1990] Towell, G., Shavlik, J., and No-
ordweier, M., \Renement of Approximate Domain Theories by
Knowledge-Based Articial Neural Networks," Proc. Eighth Natl.,
Conf. on Articial Intelligence, pp. 861-866, 1990.
200 BIBLIOGRAPHY
Unger, 1989] Unger, S., The Essence of Logic Circuits, Englewood Clis,
NJ: Prentice-Hall, 1989.
Utgo, 1989] Utgo, P., \Incremental Induction of Decision Trees," Ma-
chine Learning, 4:161{186, Nov., 1989.
Valiant, 1984] Valiant, L., \A Theory of the Learnable," Communications
of the ACM, Vol. 27, pp. 1134-1142, 1984.
Vapnik & Chervonenkis, 1971] Vapnik, V., and Chervonenkis, A., \On the
Uniform Convergence of Relative Frequencies, Theory of Probability
and its Applications, Vol. 16, No. 2, pp. 264-280, 1971.
Various Editors, 1989-1994] Advances in Neural Information Processing
Systems, vols 1 through 6, San Francisco: Morgan Kaufmann, 1989
-1994.
Watkins & Dayan, 1992] Watkins, C. J. C. H., and Dayan, P., \Technical
Note: Q-Learning," Machine Learning, 8, 279-292, 1992.
Watkins, 1989] Watkins, C. J. C. H., Learning From Delayed Rewards,
PhD Thesis, University of Cambridge, England, 1989.
Weiss & Kulikowski, 1991] Weiss, S., and Kulikowski, C., Computer Sys-
tems that Learn, San Francisco: Morgan Kaufmann, 1991.
Werbos, 1974] Werbos, P., Beyond Regression: New Tools for Prediction
and Analysis in the Behavioral Sciences, Ph.D. Thesis, Harvard Uni-
versity, 1974.
Widrow & Lehr, 1990] Widrow, B., and Lehr, M. A., \30 Years of Adap-
tive Neural Networks: Perceptron, Madaline and Backpropagation,"
Proc. IEEE, vol. 78, no. 9, pp. 1415-1442, September, 1990.
Widrow & Stearns, 1985] Widrow, B., and Stearns, S., Adaptive Signal
Processing, Englewood Clis, NJ: Prentice-Hall.
Widrow, 1962] Widrow, B., \Generalization and Storage in Networks of
Adaline Neurons," in Yovits, Jacobi, and Goldstein (eds.), Self-
organizing Systems|1962, pp. 435-461, Washington, DC: Spartan
Books, 1962.
Winder, 1961] Winder, R., \Single Stage Threshold Logic," Proc. of the
AIEE Symp. on Switching Circuits and Logical Design, Conf. paper
CP-60-1261, pp. 321-332, 1961.
BIBLIOGRAPHY 201
Winder, 1962] Winder, R., Threshold Logic, PhD Dissertation, Princeton
University, Princeton, NJ, 1962.
Wnek, et al., 1990] Wnek, J., et al., \Comparing Learning Paradigms via
Diagrammatic Visualization," in Proc. Fifth Intl. Symp. on Method-
ologies for Intelligent Systems, pp. 428-437, 1990. (Also Tech. Re-
port MLI90-2, University of Illinois at Urbana-Champaign.)

Nils J. Nilsson - Introduction To Machine Learning

Uploaded by

Copyright:

Available Formats

Nils J. Nilsson - Introduction To Machine Learning

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nils J. Nilsson - Introduction To Machine Learning

Uploaded by

Copyright:

Available Formats

INTRODUCTION

Copyright c 1996 Nils J. Nilsson

8.2 PAC Learning 119

8.2.1 The Fundamental Theorem 119

8.2.2 Examples 121

8.2.3 Some Properly PAC-Learnable Classes 122

8.3 The Vapnik-Chervonenkis Dimension 124

8.3.1 Linear Dichotomies 124

8.3.2 Capacity 126

8.3.3 A More General Capacity Result 127

8.3.4 Some Facts and Speculations About the VC Dimension129

8.5 Bibliographical and Historical Remarks 130

9 Unsupervised Learning 131

Sensory signals Goals

Figure 1.1: An AI System

1.1.2 Wellsprings of Machine Learning

1.1.3 Varieties of Machine Learning

1.2 Learning Input-Output Functions

1.2.1 Types of Learning

Figure 1.2: An Input-Output Function

Figure 1.3: A Surface that Fits Four Points

1.2.2 Input Vectors

1.2.4 Training Regimes

1.2.6 Performance Evaluation

1.3 Learning Requires Bias

|Hv| = no. of functions not ruled out

Figure 1.4: Hypotheses Remaining as a Function of Labeled Patterns Pre-

|Hv| = no. of functions not ruled out

Figure 1.5: Hypotheses Remaining From a Restricted Subset

Figure 1.6: A Training Set That Completely Determines a Linearly Sepa-

1.4 Sample Applications

1.6 Bibliographical and Historical Remarks

Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved.

functions. Boolean algebra uses the connectives , +, and . For example,

2.1.2 Diagrammatic Representations

x1 even parity function

Figure 2.1: Representing Boolean Functions on Cubes

Figure 2.2: A Karnaugh Map

is called the monomials, and a conjunction of literals itself is called a term.

2.2.2 DNF Functions

function is written as the disjunction of the implicants|corresponding to

x2x3 and x1x3 are prime implicants

Figure 2.3: A Function and its Implicants

All of the terms are prime implicants, but there

Figure 2.4: Non-Uniqueness of Representation by Prime Implicants

where f1 is a term. We say that f1 subsumes xi f1 .

Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved.

Figure 2.5: A Consensus Tree

2.2.3 CNF Functions

2.2.4 Decision Lists

2.2.6 Linearly Separable Functions

otherwise. (Note that the concept of linearly separable functions can be

Figure 2.6: Classes of Boolean Functions

2.4 Bibliographical and Historical Remarks

Introduction to Machine Learning c 1996 Nils J. Nilsson. All rights reserved.

Using Version Spaces for

training set, , the version space is that subset of hypotheses, Hv , that is

of X in  if and only if (X) = (X) for all X in . We say that the

Rule out hypotheses not

training set, , the version space is that subset of hypotheses, Hv , that is

of X in if and only if (X) = (X) for all X in . We say that the

more speci c than g , b) g is sucient, c) no function (including )

that is more speci c than g is sucient, and d) g is more speci c

1. We start with a nite training set, , of vectors, Yi , and their binary

4.1.5 The Widrow-Ho Procedure

of at a time, compute the gradient of the single-pattern squared error,