Semi-Supervised Self-Training of Object Detection Models

Chuck Rosenberg Martial Hebert Henry Schneiderman

Google, Inc. Carnegie Mellon University Carnegie Mellon University
Mountain View, CA 94043 Pittsburgh, PA 15213 Pittsburgh, PA 15213
chuck@google.com hebert@ri.cmu.edu hws@ri.cmu.edu

Abstract lizing a combination of data labeled in different ways. In

what we call “weakly labeled” training data, the labeling of
The construction of appearance-based object detection
systems is time-consuming and difficult because a large each of the image regions can take the form of a probabil-
number of training examples must be collected and man- ity distribution over labels. This makes it possible to capture
ually labeled in order to capture variations in object ap- a variety of information about the training examples. For ex-
pearance. Semi-supervised training is a means for reduc- ample, it is possible to indicate that the object of interest is
ing the effort needed to prepare the training set by train- more likely to be present toward the center of the image. Or
ing the model with a small number of fully labeled examples it is possible to encode the knowledge that a specific image
and an additional set of unlabeled or weakly labeled exam- has a high likelihood of containing the object, but that the
ples. In this work we present a semi-supervised approach object’s position is unknown. We refer to this type of train-
to training object detection systems based on self-training. ing as “weakly labeled” or “semi-supervised”.
We implement our approach as a wrapper around the train-
In the recent literature, [20], [8], anecdotal evidence has
ing process of an existing object detector and present em-
pirical results. The key contributions of this empirical study been presented which suggests that semi-supervised training
is to demonstrate that a model trained in this manner can can provide a performance improvement when applied to the
achieve results comparable to a model trained in the tradi- object detection problem. In the work presented here, we
tional manner using a much larger set of fully labeled data, perform a comprehensive empirical evaluation with the goal
and that a training data selection metric that is defined in- of characterizing and understanding these issues in order to
dependently of the detector greatly outperforms a selection facilitate the broad practical application of semi-supervised
metric based on the detection confidence generated by the training to the object detection problem. Although, for prac-
detector. tical reasons, we use one detector for evaluation, the se-
lected detector is representative of other recent algorithms
1. Introduction in the literature. We believe that, in the context of computer
1.1. Object Detection vision, this is the first comprehensive scale study of semi-
Object detection systems based on statistical models of supervised training techniques which will be necessary in
object appearance have been quite successful in recent years any practical application of object detection algorithms.
[18], [19], [17], [23]. Because these systems directly model
1.2. Training Approaches
an object’s appearance in an image, a large amount of la-
beled training data is needed to provide good coverage over In order to introduce the general approaches to semi-
the space of possible appearance variations. However, col- supervised training, let us first describe a generic detection
lecting a large amount of labeled training data can be a diffi- algorithm that classifies a subwindow in an image as being a
cult and time-consuming process. In the case of the training member of the “object” class or the “clutter” class. The clas-
data for appearance-based statistical object detection, this sification is based on the values of the feature vectors asso-
typically entails labeling which regions of the image belong ciated with each subwindow in the image.
to the object of interest and which belong to the non-object We designate the image feature vectors as X , with xi be-
part of the image, and, in some cases, marking landmark ing the data at a specific location in the image, where i in-
points. For many of the object detection techniques to be dexes the image locations from i = 1 : : : n. Our goal is to
practical, it is crucial that a streamlined approach to train- compute P (Y j X ), where Y = object, or Y = clutter.
ing be used so that users are able to rapidly insert new ob- Associated with each image class is a particular model,
ject models in their systems. which is equal either to the foreground model f or back-
The goal of the approach proposed here is to simplify ground model b. We use f to indicate the parameters of the
the collection and preparation of this training data by uti- foreground model and b for the background model. If we
 then added to the training set and the process repeats. Ob-
viously the selection metric chosen is crucial here; if incor-
rect detections are included in the training set then the final
answer may be very wrong. This issue is explored through-
out the paper. When discussing the incremental addition of
training data, it is useful to define the following terms:
 The initial labeled training set is the initial set of fully la-
Figure 1. Schematic representation of the batch train- beled data: L = fL1 : : : Lml g
 The weakly labeled training set is the current set of weakly
labeled data: W = fW1 : : : Wmw g
ing approach with EM (left) and the incremental self-
training approach (right).
 The current labeled training set is the initial training set in
addition to any weakly labeled examples which have been
set P (Y = object) = P (Y = clutter), the likelihood ra- assigned labels: T = fT1 : : : Tmt g
tio over the entire window is: We use the object detection framework detailed in the
previous sections as the basis of our weakly labeled data ap-
P (Y =objectjX )
P (Y =clutter jX ) = ni=1 PP((xxii fb))PP ((MM ==fb))
j
j proach. This approach begins with an initial set of model
parameters trained using the initial labeled training set pro-
vided, M0 = ff0 ; b0 g. This serves as the starting point for
The value of this likelihood ratio can be thresholded to de-
termine the presence or absence of an object. In practice, the
our weakly labeled data approach during which we modify
detection is performed in a subwindow of the image that is
foreground model, f .
scanned across all possible locations in the input image.
The weakly labeled data approach relies on being able
For such a generic object detector, a natural approach to
to estimate where the object is in the training image using
the current model. However, since the initial model, M0 , is
weakly-labeled training is Expectation-Maximization (EM)
[2]. This is a very generic method for generating estimates
trained using a limited amount of data, this may not be pos-
of model parameters given unknown or missing data. This
sible, especially for weakly labeled training data which dif-
is implemented as an iterative process which alternates be-
fers significantly in appearance from the training images.
tween estimating the expected values of the unknown vari-
One approach is to immediately add all of the weakly la-
beled data, W , to the training set, T . However, incorrect la-
ables and the maximum likelihood of values of the model
parameters (Figure 1(left)).
bels can potentially “corrupt” the model statistics.
It would seem that, local maxima and model selection
In the approach described here, we attempt to reduce the
issues not withstanding, EM would be the ideal approach
impact of this issue by labeling weakly labeled examples
to semi-supervised learning. Indeed, work using EM in the
and adding them incrementally to the training set accord-
context of text classification [14], [12] has found that EM is
ing to our confidence in those labels, similar to the methods
a useful approach to training models using weakly labeled
described in [13], [12], [20]. Here, the order in which the
data. However, Nigam in [14] also found that there were in-
images are added is critical to allow the model to first gen-
stances in which EM did not perform well. There are many
eralize to images which are most similar to the initial train-
reasons why EM may not perform well in a particular semi-
ing set, and then incrementally extending to views which are
supervised training context. One reason is that EM solely
quite different from those in the original training set.
finds a set of model parameters which maximize the like-
A schematic representation of the training procedure,
lihood of the data. The issue is that the fully labeled data
termed “self-training” or “incremental semi-supervised
may not sufficiently constrain the solution, which means that
training”, is presented in Figure 1(right). The incremen-
there may be solutions which maximize the data likelihood
tal training procedure for using a combination of weakly
but do not optimize classification performance.
and fully labeled data is summarized as follows:
There have been a variety of approaches that attempt to Initialization:
incorporate new information into EM and to design alternate 1. Train the parameters of the initial model, M 0 , consisting of
algorithms which can utilize additional prior information 0 0
the foreground f and background b models using the fully
which we may have about a specific semi-supervised prob- labeled data subset. Initialize the initial labeled training set,
T 0 , with the provided fully labeled data.
lem [22], [15], [6]. The alternative that we chose to evaluate
in this work is often called self-training or incremental train- Beginning of Iteration j :
1. For each Wk in W j
compute the selection metric, Sk =
ing [13]. In self-training, an initial model is constructed by
using the fully labeled data. This model is used to estimate
Sel(M j
; Wk ).
2. Select the weakly labeled example, Wk^ where
labels for the weakly labeled (or unlabeled) data. A selec- ^ = argmax Sk with the highest score and update
k k
tion metric is then used to decide which of the weakly la- both the current training set and the weakly labeled train-
j +1
beled examples were labeled correctly. Those examples are ing set,T T [f g W
j
Wk^ , j +1 W j
f gWk^ .
 3. Compute a new foreground model fj +1 by using T j+1 . Search Over Wavelet Feature
End of Iteration j : While W 6= ; Location + Scale Transform Construction Classifier

The practical implementation of the semi-supervised ap-

proach is not a straightforward application of the techniques
P
F 1  o 
described in this section because of the complex nature of 1 log P
F   
1 c

real world image data. In this paper, we focus on and pro-

vide insight into specific two key issues which are funda-
mental to practical implementation of the semi-supervised
training of object detection systems: 1) What metric should Figure 2. Schematic representation of the detection
process for a single stage of the detector.
be used in deciding which examples to add to the training
set during incremental training? 2) How is the performance
of the detector affected by the size of the labeled and weakly
labeled sets? bels. Some of the earliest work in this area is that of Szum-
mer and Jaakkola [21], which was analyzed and extended in
1.3. Previous Work [25]. A promising recent direction is that of information reg-
ularization by [22] and [5]. The notion is to exactly capture
In recent years there has been a substantial amount of
the information that we hope to transfer from the uncondi-
tional underlying distribution P (x) to the class label likeli-
work that addresses the problem of incorporating unlabeled
hood P (y j x). The idea is that the unlabeled data will con-
data into the training process, [14], [9], [11]. Some of the
earliest work in the context of object detection is described
strain the final hypothesis in a particular way. Specifically,
in [1]. The authors used an Expectation-Maximization (EM)
we want hypotheses that tend not to split high density re-
approach. More recent work by Selinger in [20] uses an in-
gions in the underlying distribution.
cremental approach similar to our approach. One of the main
differences is that a contour based detection model is used.
Also the model output is used as the scoring metric to de- 2. Experimental Setup
cide which image to add next, whereas we found that other
metrics tended to work better. Recent work [8], [7] utilizes 2.1. Detector Overview
an EM based approach.
The work which is most similar to our work is that de- We chose the detector described in [18], [19] to conduct
scribed in [24]. In this work an object detection system is our experiments. It has been used successfully for face de-
trained using images which are labeled indicating the pres- tection and other rigid objects and has been demonstrated
ence or the absence of the object of interest. A search ap- to be one of the most accurate for face detection. The de-
proach is used to find likely correspondences between de- tector is able to capture certain aspects of appearance vari-
tected features and model features. The work described in ation such as intra-class variation. To handle large changes
this paper is similar, but extends the prior work in that an in scale and translation, the detector is scanned over the im-
evaluation of a mix of labeled and weakly labeled data is age at different scales and locations, and each correspond-
performed, the problem is examined in a discriminative con- ing subwindow is run through the detector.
text and the incorporation of other types of labeled data can A schematic representation of the detection process is
be accommodated. presented in Figure 2: First, the subwindow is processed
A number of authors have taken the approach of repre- for lighting correction, then a two-level wavelet transform
senting the relationships between labeled and unlabeled data is applied, from which features are computed by vector-
using a graph in which the edge weights are inversely re- quantizing groups of wavelet coefficients. Finally, the sub-
lated to the similarity between the different examples in fea- window is classified by thresholding a linear combination of
ture space [3]. They use a minimum cut algorithm to de- the log-likelihood ratios of the features. The detector uses
cide the labeling of the unlabeled data. That method aug- a cascade architecture, in which a number of detectors are
ments the graph of training examples with a pair of “class” placed in series; only image patches which are accepted by
nodes which represent the positive and negative classes. In- the first detector are passed on to the next. In this work we
finite weight edges connect labeled examples to the appro- only use a single stage of the cascade to simplify the train-
priate “class” nodes. Their analysis showed that particular ing process. Accordingly, detection performance is lower
graph structures and edge weights correspond to optimiz- than what is typically achieved for the detector. We chose
ing specific learning criteria. to limit the processing to one stage to facilitate many rep-
The next set of ideas in this area is based on using random etitions of the training process in the experiments, which
walks through the graph to capture the notion that examples would not have been possible with the full detector because
which are similar in feature space should have similar la- of the long training times.
 6. Choose a threshold for the linear function, based on the final
performance desired.
If the full detector cascade is trained, these steps are re-
peated by setting a threshold that achieves a low false neg-
ative rate at each stage. The positive examples at each it-
eration are those images in the current training set which
passed the detection test for the previous iteration. For com-
putational reasons, we limit ourselves to one stage.
Figure 3. Landmark used on a typical training im- The goal of our experiments is to train the detector with
age (left); sample training images and the training
different combinations of fully labeled and weakly labeled
examples associated with them (right).
data and to evaluate the resulting detector performance.
The semi-supervised, incremental version of the training
2.2. Data procedure that we used in the experiments reported here can
The object chosen for these experiments is a human eye be summarized as follows:
as seen in a full or near frontal face. In each fully labeled 1. Train the detector using a limited amount of fully labeled pos-
example, four landmark locations on the eye were labeled itive examples and the full set of negative examples.
2. Run the detector over the weakly labeled portion of the data
(Figure 3). The labeled regions of each training image were
set and find the locations and scales corresponding to max-
rotated to a canonical orientation, scaled and cropped to re- ima of the likelihood ratio.
sult in a 24  16 training example image. 3. Use the output of the detector to label the unlabeled training
The full set of images with positive examples consists of examples and assign a selection score to each detection.
231 images. In each of these images, there were from two 4. Select a subset of the newly labeled examples using the se-
to six training examples per image for a total of 480 train- lection metric.
ing examples. The independent test set consisted of 44 im- 5. Iterate and go back to step 1. Stop after a fixed number of it-
erations or after all of the training images have been added.
ages. In each of these images, there were from two to 10 test-
ing examples for a total of 102 test examples. In addition to Typically, once an image has been added to the training
the object training examples, we used a set of 15,000 neg- set, it is not removed, and the values of the latent variables
ative examples. In all the experiments described in this pa- are fixed.
per, we used a fixed set of negative examples. Negative ex-
amples are assumed to be plentiful [23] and can be collected 2.4. Selection metrics
cheaply. The type of selection metrics used for selecting the next
The training and test images were typically in the range example to add from the weakly labeled data set in Step
of 200-300 pixels high and 300-400 pixels wide. While the 4. above is crucial to the performance of the training. We
training examples are all normalized to 24  16 images, evaluated the difference in performance between a selection
scale invariance is achieved by scaling the image during the metric based on the classification confidence and a selec-
detection process. A total of 80 synthetic variations are ap- tion metric based on an distance measure between patches
plied to each training example including: 0:5 translation, that is defined independently from the detector. A key obser-
0.945 to 1.055 scale, 12 degrees rotation. vation is that, because it is independently defined, the sec-
ond metric will have failure modes that are “orthogonal” to
2.3. Training
the failure modes of the detector, leading to better perfor-
Training the model with fully labeled data consists of the mance, as supported by the empirical results below. The ef-
following steps: fect of using an independently-defined metric had not been
1. Given the training data landmark locations, geometrically previously investigated and it appears to contribute in a crit-
normalize the training example subimages, apply lighting ical way to training performance.
normalization to the subimages, and generate synthetic train-
The first selection metric, termed confidence selection
ing examples. The latter consists of scaling, shifting, and ro-
tating the images by small amounts. metric, is computed at every iteration by applying the detec-
2. Compute the wavelet transform of the subimages. tor trained from the current set of labeled data to the weakly
3. Quantize each group of wavelet coefficients and build a naive labeled set. The detection with the highest detection confi-
Bayes model with respect to each group to discriminate be- dence are selected and added to the training set. The second
tween positive and negative examples. selection metric, termed MSE selection metric is calculated
4. Adjust the naive Bayes model using boosting, but maintain-
for each weakly labeled example by evaluating the distance
ing a linear decision function, effectively performing gradi-
ent descent on the margin. between the corresponding image window and all of the
5. Compute an ROC curve for the detector using a cross valida- other templates in the training data (including the original
tion set. labeled examples and the weakly labeled examples added in
 we call a run. In most cases 5 runs were performed for each
experiment.
Another parameter of the experiments is the number of
images added at each iteration. Ideally, only a single im-
age would be added at each iteration. However, because of
the substantial training time of the detector, more than one
image was added at each iteration. Adding more images re-
duces the average training time per weakly labeled image,
but increases the chance that there will be an incorrect de-
Figure 4. A schematic representation of the computa- tection included in the weakly labeled data set. Typically 20
tion of the MSE score metric. The candidate image and weakly labeled images were added to the training set at each
the labeled images are first normalized with a specific set iteration.
of processing steps before the MSE based score metric One of the challenges in performing such experiments is
is computed. that the inner loop of the algorithm, training the detector on
one specific training set, takes on the order of twelve hours
prior iterations). The distance is computed after normaliza- on 3.0 GHz level machines. If the detector is trained dur-
ing 10 iterations and 5 repetitions of an experiment are per-
formed, then each experiment takes 12  10  5 = 600
tion of the detected template for scale, position, and orienta-
tion, based on the values computed by the detector, and after
3x3 high-pass filtering and normalization to zero mean and hours of compute time. As a result, the total computation
unit variance (Figure 4). Each candidate image is assigned time necessary to investigate all the variations of parame-
a score which is the minimum of these distances. The can- ters and training conditions increases rapidly (to approxi-
didate images with the smallest scores are selected for addi- mately 3 CPU-years).1
tion to the training set. If we define Wi to be the weakly la- 3.2. Evaluation Metrics
beled image under consideration, j to be the index over la-
beled images, Lj to be a specific image from the set of la- Each “run” was evaluated by using the area under the
beled images, g (X ) to be the transformation performed by ROC curve (AUC). Because different experimental condi-
the image preprocessing step, and  to be the weights for tions affect performance, the AUCs were normalized rela-
computing the Mahalanobis distance, then the overall com- tive to the full data performance of that run. So a reported
putation can be written as: performance level of 1.0 would mean that the model be-
ing evaluated has the same performance as it would if all
Score(Wi ) = minj Mahalanobis (g (Wi ); g (Lj ); ) of the labeled data was utilized. A value of less than 1.0
It is important to note that, in the computation of the MSE would mean that the model has a lower performance than
selection metric, the key information that is used is the po- that achieved with the full data set. To compute the full data
sition and scale returned by the currently detector. As a re- performance, each specific run is trained with the full data
sult, the detector must be accurate in localization but need set and its performance is recorded. The performance from
not be accurate in detection since false detection will be dis- all of the runs of a specific experiment are aggregated and
carded due to large their large MSE distances to all of the we compute a single set of performance measures: the mean,
training examples. This is crucial to ensure the performance the standard deviation, and the 95% significance interval,
of the training algorithm with small initial training sets. This computed as the mean plus and minus 1.64 times the stan-
is also part of the reason for the MSE to outperform the con- dard error of the mean. The plots show either or both the
fidence metric, which requires the detector to be accurate in standard deviation or the 95% significance interval as error
both localization and detection performance. bars.

3. Experiments and Analysis 3.3. Baseline training configuration

3.1. Experiment Scenarios It is very important to characterize sensitivity to train-
ing set size because we want to perform our experiments
We found that there was quite a bit of variance in the fi- under conditions where the addition of weakly labeled data
nal detector performance and in the behavior of the semi- will make a difference. If the performance of the detector
supervised training process. Much of this variance arose is already at its maximum, given a labeled training set of a
from the specific set of images randomly selected in the specific size, then we cannot expect weakly labeled data to
small initial training subset. To overcome this limitation,
each experiment was repeated using a different initial ran- 1 For reasons of space, we present only a summary of the experiments.
dom subset. We call a specific set of experimental condi- Detailed analysis of the influence of the number of features, the geo-
tions an experiment, and each repetition of that experiment metric variations, and other training variations are reported in [16].
 Normalized Test Set Performance versus Training Set Size
1.1 Norm Confidence Score Iter MSE Score Norm
AUC Training Set Training Set AUC
1
Full Data Normalized Area Under the ROC Curve

0.9
0.822 0 0.822
0.8

0.7 0.770 1 0.867

0.6
0.798 2 0.882
0.5

0.4
smooth saturated 0.745 3 0.922
failure
0.3
cv=025 sts=06 (+/- 95% sig, +/- std dev)
0.759 4 0.931
20 30 40 60 80 100 120 160 200 240 320 400 480
Number of Training Images on a Log Scale

Figure 5. Normalized AUC performance of the detector Figure 6. Comparison of the training images selected
plotted against training set size on a log scale with the at each iteration for the confidence and the MSE selec-
three regimes of operation labeled. The inner error bars tion metrics. The initial training set of 40 images is the
indicate the 95% significance interval, and the outer er- same for both metrics and is 1/12 of the initial training
ror bars indicate the standard deviation of the mean. set size.

help. In order to establish a baseline for the typical number The comparison between the two selection metrics is
of examples needed to train the detector, we ran the detector summarized in Figure 7. In these plots, the horizontal axis
with different training set sizes and recorded the AUC per- indicates the frequency at which the training data is sam-
formance (Figure 5). Our interpretation of this data is that is pled in order to select the initial labeled training set for each
there are three regimes in which the training process oper- run (“8” means that 1/8th of the full training data was used
ates. We call the first the “saturated” regime, which in this as initial labeled training data, while the rest was used as un-
case appears to be from approximately 160 to 480 train- labeled data). The plots show that performance is improved
ing examples. In this regime, 160 examples are sufficient by the addition of weakly labeled data over the range of data
for the detector to learn the requisite parameters; more data set sizes. However, the improvements are not significant at
does not result in better performance. Similarly, variation in the 95% level for the confidence metric. For the MSE met-
performance is relatively constant and small in this range. ric however, the improvement in performance is significant
We call the second regime the “smooth” regime, which ap- for all the data set sizes. This observation is supported by
pears in this case to be between 35 and 160 training exam- other experimental variations in which the MSE metric con-
ples. In this regime, performance decreases and variation in- sistently outperforms the confidence metric. Figure 6 shows
creases relatively smoothly as training set size decreases. In montages of the examples selected from the weakly labeled
the third regime, the “failure” regime, there is both a precip- training images selected at each iteration using the confi-
itous drop in performance and a very large increase in per- dence metric and the MSE metric for a single run. The per-
formance variation. This third regime occurs when the train- formance of the detector trained with the MSE metric im-
ing algorithm does not have sufficient data to estimate some proves with each iteration, whereas the performance of the
set of parameters. An extreme case of this would be when confidence-based one decreases. For the confidence metric,
the parameter estimation problem is ill conditioned. Based there are clearly incorrect detections included in the train-
on this set of experiments, we chose the size of the labeled ing set past the first iteration. In contrast, all of the images
training set to be in the smooth regime for the experiments that the MSE metric selects are valid except for one outlier
with weakly-labeled data. at iteration 4.

3.4. Selection Metrics 3.5. Relative size of Fully Labeled Data

The next question is whether the choice of the selection It is also important to evaluate the number of weakly la-
metric makes a substantial difference in the performance of beled exemplars that need to be added to the labeled set in
the semi-supervised training. We conducted experiments to order to reach the best detector performance. For this evalu-
compare the two main options: A confidence metric based ation, we recorded the number of examples that need to be
on the most natural approach of selecting the example with added to the initial set in order to reach the point at which
the highest detector confidence, and the MSE metric that is the performance of the detector does not change apprecia-
defined independently of the detector confidence. The over- bly for every training run. The data is summarized in Fig-
all result is that the detector-independent MSE metric out- ure 8, in which we plotted the ratio of weakly labeled data
performs the more intuitive confidence metric. to labeled data at which the training procedure converged,
 Normalized Test Set Performance versus Training Set Sampling Rate with Confidence Score Normalized Test Set Performance versus Training Set Sampling Rate with MSE Score
base performance obtained with the fully labeled data, even
1 1.05

0.95
1
when a small fraction of the training data is used in the ini-
Full Data Normalized Area Under the ROC Curve

Full Data Normalized Area Under the ROC Curve

0.9
0.95
tial training set. This observation remains valid even when
0.9
0.85

0.85
taking into account the high degree of variability in perfor-
0.8
0.8 mance across different choices of initial training sets (as il-
0.75

0.7
0.75

0.7
lustrated by the error bars in the graphs presented, and the
0.65
22 20 18
cv025 sts06 - fully labeled data only (+/- 95% sig)
cv025 sts06 - best weakly labeled performance (+/- 95% sig)
16 14 12 10 8 6
0.65
22 20 18 16
cv025 sts06 - fully labeled data only (+/- 95% sig)
cv025 sts06 - best weakly labeled performance (+/- 95% sig)
14 12 10 8 6
fact that we normalize the detector performance with respect
Training Set Sampling Rate Training Set Sampling Rate

to the base detector trained with all the labeled data). Sec-
(a) (b)
ond, as a practical matter, the experiments show that the self-
training approach to semi-supervised training can be applied
Figure 7. Normalized performance of the detector, in-
to an existing detector that was originally designed for su-
corporating weakly labeled data by using the confidence
metric (a) or the MSE metric (b), as the fully labeled train- pervised training. In fact, in our case, we used a detector that
ing set size varies. The bottom plot line is the perfor- was already highly optimized and we were able to integrate
mance with labeled data only and the top plot line is the it in the training framework. This suggests a general proce-
performance with the addition of weakly labeled data. dure for using semi-supervised training with existing detec-
Error bars indicate the 95% significance interval of the tors.
mean value. Finally, a more fundamental observation is that the MSE
selection metric consistently outperforms the confidence
metric. Experiments with simulated data and other, filter-
5
Weakly versus Fully Labeled Training Set Size with Confidence Score over Sampling Rate
based detectors (from [16], not reported here from reasons
Ratio of Weakly to Fully Labeled Training Set Size (+/- 95% sig)

4.5 of space) show that, more generally, the self-training ap-

4
proach using an independently-defined selection metric out-
3.5
performs both the same approach with confidence metrics,
3

2.5
but also batch EM approaches. These results bring to light an
2 important aspect of the the self-training process which is of-
1.5 ten overlooked. The issue is that during the training process,
1
the distribution of the labeled data at any particular iteration
0.5
22 20 18 16 14 12
Training Set Sampling Rate
10 8 6
may not match the actual underlying distribution of the data.
As a result, confidence metrics may perform poorly because
Figure 8. Ratio of weakly labeled to fully labeled data the labeled data distribution created by this metric is quite
as the fully labeled training set size increases. different from the underlying distribution, even when all of
the weakly labeled data selected by the metric is correctly
against the size of the initial training set (or, more precisely, labeled. To illustrate this observation, Figure 9 shows a sim-
the sampling rate that was used for generating the initial ple simulated example in which the labeled and unlabeled
training set). This data shows that, as expected, the ratio in- examples are drawn from two Gaussian distributions in the
creases as the size of the initial training set decreases since plane. Comparing the labels obtained after five iterations by
more weakly labeled examples are needed to compensate using the confidence metric (Figure9(c)) and the Euclidean
for smaller training sets. More importantly, the total size metric, we see that the labeled points cluster around exist-
of the training set (initial labeled training images + exam- ing data points. We believe a closer examination of this issue
ples added during training) is within the “saturated” operat- from both a theoretical and practical standpoint is an impor-
ing regime identified in Figure 5. This is important because tant interesting topic for future research toward the effective
it shows that, even for small initial training sets, the total application of the semi-supervised approaches to object de-
number of examples is on the same order as the number that tection problems.
would be needed to train the detector with a single set of la- 4. Summary and Conclusions
beled examples. In other words, using a small set of labeled
examples does not cause us to pay a penalty in terms of a The goal of this work was to explore and evaluate ap-
greater size of the total training set. proaches to semi-supervised training using weakly labeled
data for appearance-based object detection. We conducted
3.6. Discussion
extensive experiments with a state-of-the art detector that
These experiments lead us to several observations that led to several important conclusions including a quantita-
will be useful in developing future detection systems based tive evaluation of the performance gained by adding weakly
on weakly-labeled training. First, the results show that it is labeled data to an initial small set of labeled data; a demon-
possible to achieve detection performance that is close to the stration of the feasibility of modifying an existing detector
 14 14
NIPS, 1998.
12

10
12

10
[2] C. M. Bishop. Neural Networks for Pattern Recognition. Ox-
8

6
8

6
ford University Press, 1995.
4 4
[3] A. Blum and S. Chawla. Learning from labeled and unla-
Feature 2

Feature 2
2 2

−2
0

−2
0
beled data using graph mincuts. ICML, 2001.
−4

−6
−4

−6
[4] A. Blum and T. Mitchell. Combining labeled and unlabeled
−8

−10
−8

−10 Unlabeled Class 1

data with co-training. COLT, 1998.
Unlabeled Unlabeled Class 2
−12

−14
Class 1
Class 2
−12

−14
Labeled Class 1
Labeled Class 2
[5] A. Corduneanu and T. Jaakkola. On information regulariza-
−14 −12 −10 −8 −6 −4 −2 0
Feature 1
2 4 6 8 10 12 14 −14 −12 −10 −8 −6 −4 −2 0
Feature 1
2 4 6 8 10 12 14
tion. UAI, 2003.
(a) (b) [6] F. Cozman, I. Cohen, and M. Cirelo. Semi-supervised learn-
14

12
14

12
ing and model search. ICML Workshop on the Continuum
10

8
10

8
from Labeled to Unlabeled Data in Machine Learning and
6 6 Data Mining., 2003.
4 4

[7] L. Fei-Fei, R. Fergus, and P. Perona. A bayesian approach to

Feature 2

Feature 2

2 2

0 0

−2 −2 unsupervised one-shot learning of object categories. ICCV,

−4 −4

−6 −6 2003.
−8 −8

−10
Unlabeled
Unlabeled Class 1
Unlabeled Class 2
−10
Unlabeled
Unlabeled Class 1
Unlabeled Class 2
[8] R. Fergus, O. Perona, and A. Zisserman. Object class recog-
−12

−14
−14 −12 −10 −8 −6 −4 −2 0 2 4 6
Labeled Class 1
Labeled Class 2
8 10 12 14
−12

−14
−14 −12 −10 −8 −6 −4 −2 0 2 4 6
Labeled Class 1
Labeled Class 2
8 10 12 14
nition by unsupervised scale-invariant learning. CVPR, 2003.
Feature 1 Feature 1
[9] T. Joachims. Transductive inference for text classification us-
(c) (d) ing support vector machines. ICML, 1999.
[10] A. Levin, P. Viola, and Y. Freund. Unsupervised improve-
Figure 9. (a) Original unlabeled data and labeled data; ment of visual detectors using co-training. ICCV, 2003.
(b) Plot of the true labels for the unlabeled data; (c),(d) [11] A. McCallum, K. Nigam, and L. Ungar. Efficient cluster-
The points labeled by the incremental self-training algo- ing of high-dimensional data sets with application to refer-
rithm after 5 iterations using the confidence metric and ence matching. KDD, 2000.
the Euclidean metric, respectively. [12] K. Nigam. Using Unlabeled Data to Improve Text Classifica-
tion. PhD thesis, Carnegie Mellon University Computer Sci-
ence Dept., 2001. CMU-CS-01-126.
to use weakly labeled data; and insights into the choice of [13] K. Nigam and R. Ghani. Analyzing the effectiveness and ap-
selection metric used for training. plicability of co-training. CIKM, 2000.
Many important issues that are critical to practical appli- [14] K. Nigam, A. McCallum, S. Thrun, and T. Mitchell. Learn-
ing to classify text from labeled and unlabeled documents.
cations of these training ideas remain to be explored. First, AAAI, 1998.
it might be important to use a different version of the de- [15] Y. Rachlin. A general algorithmic framework for discover-
tector for initial training and for actual use on test images. ing discriminative and generative structure in data. Master’s
For example, we found that the position and scale accuracy thesis, ECE Dept. Carnegie Mellon University, 2002.
of the detector are important for semi-supervised training, [16] C. Rosenberg. Semi-Supervised Training of Models for
whereas they may be less important when the detector is Appearance-Based Statistical Object Detection Methods.
PhD thesis, Carnegie Mellon University Computer Science
used in an application. Second, one alternative explanation Dept., May 2004. CMU-CS-04-150.
for the success of the nearest neighbor approach (based on [17] B. Schiele and J. Crowley. Recognition without correspon-
the appropriate selection metric) is that it is performing a dence using multidimensional receptive field histograms.
type of co-training [4], [13], [10]. It would be interesting to IJCV, 36(1):31–52, 2000.
study the relation between the semi-supervised training ap- [18] H. Schneiderman. Feature-centric evaluation for efficient
proach evaluated here with the co-training approaches. As cascaded object detection. CVPR, 2004.
[19] H. Schneiderman. Learning a restricted bayesian network for
shown in the experiments, the choice of the initial training
object detection. CVPR, 2004.
set has a large effect on performance. Although we have per- [20] A. Selinger. Minimally supervised acquisition of 3d recogni-
formed experiments that compare different selections of the tion models from cluttered images. CVPR, 2001.
initial training set, it would be useful to develop more pre- [21] M. Szummer and T. Jaakkola. Partially labeled classification
cise guidelines for selecting it. Finally, the approach could with markov random walks. NIPS, 2001.
be extended to training examples that are labeled in differ- [22] M. Szummer and T. Jaakkola. Information regularization
ent ways. For example, some images may be provided with with partially labeled data. NIPS, 2002.
[23] P. Viola and M. J. Jones. Robust real-time object detection.
scale information and nothing else. Additional information
Technical report, Compaq Cambridge Research Lab, 2001.
may be provided such as the rough shape of the object, or a [24] M. Weber, M. Welling, and P. Perona. Unsupervised learning
prior distribution over its location in the image. of models for recognition. ECCV, 2000.
[25] X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised
References learning using gaussian fields and harmonic functions.
ICML, 2003.
[1] S. Baluja. Probabilistic modeling for face orientation dis-
crimination: Learning from labeled and unlabeled data.