IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO.

12, DECEMBER 2014 2035

Dynamic Image-to-Class Warping for Occluded

Face Recognition
Xingjie Wei, Chang-Tsun Li, Senior Member, IEEE, Zhen Lei, Member, IEEE,
Dong Yi, and Stan Z. Li, Fellow, IEEE

Abstract— Face recognition (FR) systems in real-world appli- occluded FR is twofold. Firstly, occlusions distort the discrim-
cations need to deal with a wide range of interferences, such as inative facial features and increases the distance between two
occlusions and disguises in face images. Compared with other face images of the same subject in the feature space. The
forms of interferences such as nonuniform illumination and pose
changes, face with occlusions has not attracted enough attention intra-class variations are larger than the inter-class variations,
yet. A novel approach, coined dynamic image-to-class warping which results in poorer recognition performance. Secondly,
(DICW), is proposed in this work to deal with this challenge when facial landmarks are occluded, large registration errors
in FR. The face consists of the forehead, eyes, nose, mouth, and usually occur and degrade the recognition rate [1].
chin in a natural order and this order does not change despite Note that there are two related but different problems to FR
occlusions. Thus, a face image is partitioned into patches, which
are then concatenated in the raster scan order to form an ordered with occlusions: occluded face detection and occluded face
sequence. Considering this order information, DICW computes recovery. The first task is to determine whether a face image
the image-to-class distance between a query face and those of an is occluded or not [2], which can be used for automatically
enrolled subject by finding the optimal alignment between the rejecting the occluded images in applications such as passport
query sequence and all sequences of that subject along both the image enrolment. This rejection mechanism is not always
time dimension and within-class dimension. Unlike most existing
methods, our method is able to deal with occlusions which exist in applicable in some scenarios (e.g., surveillance) where no
both gallery and probe images. Extensive experiments on public alternative image can be obtained due to the lack of user
face databases with various types of occlusions have confirmed cooperation. The second task is to restore the occluded region
the effectiveness of the proposed method. in face images [3], [4]. It can recover the occluded area but is
Index Terms— Face recognition, occlusion, image-to-class unable to directly contribute to recognition since the identity
distance, dynamic time warping, biometrics. information can be contaminated during inpainting.
An intuitive idea for handling occlusions in FR is to detect
I. I NTRODUCTION the occluded region first and then perform recognition using
only the unoccluded part. Min et al. [5] adopted a SVM
F ACE recognition (FR) is one of the most active research
topics in computer vision and patten recognition over the
past few decades. Nowadays, automatic FR system achieves
classifier to detect the occluded region in a face image then
used only the unoccluded area of a probe face (i.e., query
face) as well as the corresponding area of the gallery faces
significant progress in controlled conditions. However, the
(i.e., reference faces) for recognition. But note that the occlu-
performance in unconstrained conditions (e.g., large variations
sion types in the training images are the same as those in the
in illumination, pose, expression, etc.) is still unsatisfactory.
testing images. Jia and Martinez [6], [7] used a skin colour
In the real-world environments, faces are easily occluded by
based mask to remove the occluded area for recognition.
facial accessories (e.g., sunglasses, scarf, hat, veil), objects in
However, the types of occlusions are unpredictable in practical
front of the face (e.g., hand, food, mobile phone), extreme
scenarios. The location, size and shape of occlusions are
illumination (e.g., shadow), self-occlusion (e.g., non-frontal
unknown, hence increasing the difficulty in segmenting the
pose) or poor image quality (e.g., blurring). The difficulty of
occluded region from the face images. Currently most of the
Manuscript received March 15, 2014; revised July 11, 2014; accepted occlusion detectors are trained on faces with specific types
September 3, 2014. Date of publication September 22, 2014; date of current of occlusions (i.e., the training is data-dependent) and hence
version November 10, 2014. The work of Z. Lei was supported by the National
Natural Science Foundation of China under Project 61103156 and Project generalise poorly to various types of occlusions in the real-
61473291. The associate editor coordinating the review of this manuscript world environment.
and approving it for publication was Prof. Gérard Medioni. (Corresponding In this paper, we focus on performing recognition in the
author: Chang-Tsun Li.)
X. Wei and C.-T. Li are with the Department of Computer Science, Univer- presence of occlusions. There are two main categories of
sity of Warwick, Coventry CV4 7AL, U.K. (e-mail: x.wei@warwick.ac.uk; approaches in this direction. The first is the reconstruction
ct.li@warwick.ac.uk). based approaches which treat occluded FR as a reconstruction
Z. Lei, D. Yi, and S. Z. Li are with the Center for Biometrics and
Security Research & National Laboratory of Pattern Recognition, Institute of problem [6], [8]–[13]. The sparse representation based classi-
Automation, Chinese Academy of Sciences, Beijing 100190, China (e-mail: fication (SRC) [8] is a representative example. A clean image
zlei@cbsr.ia.ac.cn; dyi@cbsr.ia.ac.cn; szli@cbsr.ia.ac.cn). is reconstructed from an occluded probe image by a linear
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. combination of gallery images. Then the occluded image is
Digital Object Identifier 10.1109/TIFS.2014.2359632 assigned to the class with the minimal reconstruction error.
1556-6013 © 2014 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted,
but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2036 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

TABLE I
T HREE T YPICAL O CCLUSION C ASES

The reconstruction based approaches usually require a large finding the optimal alignment between the query sequence
number of samples per subject to represent a probe image. and all enrolled sequences of that subject. Our method allows
However, a sufficient number of samples are not always elastic match in both time and with-class directions.
available in practical scenarios. Most of the existing works that simply treat occluded FR as
The second category is the local matching based a signal recovery problem or just employ the framework for
approaches [14]–[17]. Facial features are extracted from local general object classification, neglect the inherent structure of
areas of a face, for example, overlapping or non-overlapping the face. Wang et al. proposed a Markov Random Field (MRF)
patches of an image, so the affected and unaffected parts of based method [23] for FR and confirmed that the contextual
the face can be analysed in isolation. In order to minimise information between facial features plays an important role
matching errors due to occluded parts, different strategies in recognition. In this paper, we propose a novel approach
such as local subspace learning [14], [15], partial distance that takes the facial order, which contains the geometry infor-
learning [16] and multi-task sparse representation learning [17] mation of the face, into account when recognising partially
are performed. Our method belongs to this category but does occluded faces. In uncontrolled environments with uncooper-
not require training. ative subjects, the occlusion preprocessing and the collection
In addition to the above approaches, which focus on of sufficient and representative training samples are generally
improving the robustness during the recognition stage, recently very difficult. Our method which performs recognition directly
many researchers [18], [19] also pay attention to the image in the presence of occlusions and does not require training, is
presentation stage and attempt to extract stable, occlusion- hence feasible for realistic FR applications.
insensitive features from face images. Since the forms of This paper is built upon our preliminary work reported
occlusions in real-world scenarios are unpredictable, it is still in [24] and [25]. The remainder of this paper is organised
difficult to find a suitable representation which is insensitive as follows. The proposed Dynamic Image-to-Class Warping
to the variations in occlusions. method, from image representation, modelling to implementa-
Most of the current methods assume that occlusions only tion, is described in Section II. Extensive experiments includ-
exist in the probe images and the gallery or training images ing discussions are presented in Section III. Further analysis
are clean. In practical scenarios, occlusions may occur in both about why the proposed method works; when and why it will
gallery and probe images [6], [7], [20]. When the number of fail and how to improve it is discussed in Section IV. Finally
gallery/training images is limited, excluding these occluded the work is concluded in Section V.
images would, on the one hand, lead to small sample size (SSS)
problem [21], and on the other hand, ignore useful information
for recognition [20]. We summarise three occlusion cases in II. DYNAMIC I MAGE - TO -C LASS WARPING
Table I, which a FR system may encounter in the real-world
A. Image Representation
applications. Most of the current methods rely on a clean
gallery or training set and only consider the first case. The An image is partitioned into J non-overlapping patches of
latter two cases would also occur in real environment but have d ×d  pixels. Those patches are then concatenated in the raster
not yet received much attention. scan order (i.e., from left to right and top to bottom) to form a
We propose a local matching based method, Dynamic single sequence. The reason for doing so is that the forehead,
Image-to-Class Warping (DICW), for occluded FR. eyes, nose, mouth and chin are located in the face in a natural
DICW is motivated by the Dynamic Time Warping (DTW) order, which does not change despite occlusions or imprecise
algorithm [22] which allows elastic match of two time registration. This spatial facial order, which is contained in
sequences. It has been successfully applied to the area of the patch sequence, can be viewed as the temporal order in
speech recognition [22]. In our work, an image is partitioned the time sequence. In this way, a face image can be viewed
into patches, which are then concatenated in the raster scan as a time sequence so the image matching problem can be
order to form a sequence. In this way, a face is represented handled by the time series analysis technique like DTW [22].
by a patch sequence which contains the order information of Let f (x, y) be the intensity of the pixel at coordinates (x, y)
facial features. DICW calculates the Image-to-Class distance and f j (x, y) be the j -patch. A difference patch  f j (x, y) is
between a query face and those of an enrolled subject by computed (Fig. 1) by subtracting f j (x, y) from its immediate
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2037

between similar sequences is relatively small. In addition,

the temporal order is considered during matching, thus
cross-matching (which reverses the order of data points) is
not allowed even it can lead to shorter distance (Fig. 3c).
Especially for FR, this is reasonable since the order of facial
features should not be turned back.
Adopting this idea for FR, we want to find the optimal
alignment between face sequences while minimising the dis-
tance caused by occluded patches. In this work, instead of
Fig. 1. Image representation of DICW. finding the alignment between two sequences, we seek the
alignment between a sequence and the sequence set of a
neighbouring patch f j +1 (x, y) as: given class (i.e., subject). A probe image consisting of M
patch features is denoted by P = ( p1 , . . . , pm , . . . , p M ).
 f j (x, y) = f j +1 (x, y) − f j (x, y) (1) Here P is an ordered list where each element pm is a
where j ∈ {1, 2, . . . , J − 1}. Note that here the length of the patch feature vector (e.g.,  f (·) in Section II-A). The gallery
difference patch sequence is J − 1. set of a given class containing K images is denoted by
A difference patch  f j (·) actually can be viewed as the G = {G 1 , . . . , G k , . . . , G K }. The k-th gallery image is
approximation of the first-order derivative of adjacent patch similarly represented as a sequence of N patch features as
f j +1 (·) and f j (·). The salient facial features which represent G k = (g1k , . . . , g nk , . . . , g Nk ) where g nk represents a patch
textured regions such as eyes, nose and mouth can be enhanced feature vector like pm . Note that the number of patches in the
since the first-order derivative operator is sensitive to edges. probe image and that in the gallery image can be different (i.e.,
We use 3,200 occluded-unoccluded image pairs of the same the values of M and N can be different) since the DTW model
class and different classes from the AR database [26], respec- is able to deal with sequences with different lengths [22].
tively (6,400 pairs in total) to calculate the image distance A warping path W indicating the matching correspondence
distributions.1 As shown in Fig. 2, the distance distributions of patches between P and G with T warping steps in time
of the same and different classes are separated more widely axis is defined as W = (w(1), . . . , w(t), . . . , w(T )) with:
when using the difference patches (Fig. 2b).
w(t) = (m, n, k) : {1, 2, . . . , T } → {1, 2, . . . , M}
×{1, 2, . . . , N} × {1, 2, . . . , K } (2)
B. Modelling
Face matching is implemented by defining a distance mea- where × indicates the Cartesian product operator and
surement between sequences and using the distance as the max{M, N}  T  M + N − 1. w(t) = (m, n, k) is a triple
basis for classification. Generally, a small distance is expected indicating that patch pm is matched to patch g nk at step t.
if two sequences are similar to each other. DICW is based on Similar to the DTW model [22], W in DICW satisfies the
the classical DTW algorithm [22] which is used to compute the following four constraints:
distance between two time sequences. Here we use an example 1) Boundary: w(1) = (1, 1, k) and w(T ) = (M, N, k  ). The
to quickly illustrate the main idea of DTW (more details of path starts at matching p1 to g 1k and ends at matching
the algorithm can be found in [22]). As shown in Fig. 3, there p M to g Nk  . Note that no restrictions are placed on
are two sequences (each digit indicates a data point): k and k  . From step 1 to T , k and k  can be any value
from 1 to K since the probe patch can be matched with
A = (a1 , a2 , a3 , a4 , a5 ) = (3, 1, 10, 3, 2)
patches from all K gallery images.
B = (b1 , b2 , b3 , b4 , b5 ) = (3, 1, 2, 10, 3). 2) Monotonicity: Given w(t) = (m, n, k), the preceding
The Euclidean distance (i.e., using point-wise matching, triple w(t − 1) = (m  , n  , k  ) satisfies that m   m and
 n   n. The warping path preserves the temporal order
Fig.
√ 3a) between them is (a1 − b1 )2 + · · · + (a5 − b5 )2 = and increase monotonically.
0 + 0 + 64 + 49 + 1 ≈ 10.68 which is a bit large for 3) Continuity: Given w(t) = (m, n, k), the preceding triple
these two similar sequences. However, if we warp these two w(t − 1) = (m  , n  , k  ) satisfies that m − m   1 and
sequences in a non-linear way by shrinking or expanding n − n   1. The indexes of the path increase by 1 in
them along the time axis during matching (i.e., allows flexible each step, which means that each step makes smooth
correspondences), the distance between A and B can be transitions along the time dimension.
largely reduced2 to 2 (Fig. 3b). DTW, which is based on this 4) Window constraint: Given w(t) = (m, n, k), it satisfies
idea, calculates the distance between two time sequences by |m − n|  l where l ∈ N+ is the window width [22].
finding the optimal alignment between them with the minimal The window constraint is designed to reduce the compu-
overall cost. This will help to reduce the distance error tational cost of DICW. But it is also meaningful for the
caused by some noise data points and ensure that the distance specific FR problem since a probe patch (e.g., eye) should
1 We use Euclidean distance as measurement. The image size is not match to a patch (e.g., mouth) too far away. The
83 × 60 pixels and the patch size is 5 × 5 pixels. window with a width l is able to constrain the warping
2 Computation details see [22]. path within an appropriate range.
2038 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

Fig. 2. Distributions of face image distance of the same and different classes. Using the difference patch (b), the distance distribution of the same class and
that of the different classes are separated more widely compared with those using the original patch (a).

distance measurement such as the Euclidean distance or the

Cosine distance. The overall matching cost of W is the sum
of the local distance of each warping step:

T
S(W ) = Cwt (3)
t =1
The optimal warping path W ∗ (i.e., the red grid path in Fig. 4b)
is the path that minimises S(W). The Image-to-Class distance
between P and G is simply the overall cost of W ∗ :

T
di st D I C W ( P, G) = min Cwt (4)
W
t =1
Fig. 3. Various ways of sequence matching. a) Point-wise matching, b) DTW
matching, and c) cross matching. After computing di st D I C W between P and each enrolled
subject in the database, a classifier such as the Nearest
Neighbour classifier can be adopted for classification based
These constraints are extended from the constraints of the on di st D I C W .
DTW algorithm. However, they are also very meaningful in
the context of FR with the image representation defined in C. Implementation Through Dynamic Programming
Section II-A. Our method represents a face image as a patch
sequence thus here the image matching problem can be solved To compute di st D I C W ( P, G) in (4), one could test every
by the time series analysis technique. possible warping path but with a high computational cost.
In order to explain the concept of warping path, we take the Fortunately, (4) can be solved efficiently by Dynamic Pro-
aforementioned sequences A and B as an example. In Fig. 4a, gramming. A three-dimensional matrix D ∈ R M×N×K is
each grid on the right hand side indicates a possible matching created to store the cumulative distance. The element Dm,n,k
correspondence. The indexes of the red grids indicate the stores the cost of the optimal warping path of matching
matching between A and B by DTW (i.e., the optimal warping the first m probe patches to the set of first n patches of
path with the minimal matching cost) as shown in the left each gallery sequence and at the same time the m-th patch
part (here T = 6). Likewise, the same procedure of DICW is pm is matched to the patch from the k-th gallery image. The
shown in Fig. 4b. Compared with DTW, an additional index calculation of the final optimal cost di st D I C W ( P, G) is based
is added in the warping step of DICW to index different on the results of a series of predecessors. D can be computed
gallery sequences. In this way, the warping is performed recursively as:
 
in two directions: 1) a probe sequence P is aligned to a D{(m−1,n−1)}×{1,2,...,K } ,
set of gallery sequences G according to the time dimension Dm,n,k = min D{(m−1,n)}×{1,2,...,K } , + Cm,n,k (5)
(maintaining the facial order) and 2) simultaneously, at each D{(m,n−1)}×{1,2,...,K }
warping step, each patch in P can be matched with any patch where the initialisation is done by extending D as
among all gallery sequences along the within-class dimension. an (M + 1) × (N + 1) × K matrix and setting
Our method allows elastic match in both of the aforementioned D0,0,· = 0, D 0,n,· = D m,0,· = ∞. Thus, di st D I C W ( P, G) can
two directions. be obtained as follows:
We define the local distance [22] Cm,n,k = d( pm , g nk ) as
the distance between two patches pm and g nk . d(·) can be any di st D I C W ( P, G) = min { D M,N,k } (6)
k∈{1,2,...,K }
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2039

Fig. 4. An illustration of warping path in (a) DTW and the (b) proposed DICW. The arrows indicate the matching correspondence. The dashed line marks
the optimal warping path. Black blocks indicate the occluded patches.

Algorithm 1 Dynamic Image-to-Class Warping Distance

DICW( P, G, l)
Input:
P: a probe sequence with M patches;
G: a set of K gallery sequences (each with N patches) of a given
class;
l: the window width;
Output:
dist D I C W : the Image-to-Class distance between P and G;
Fig. 5. The illustration of (a) the Image-to-Image and (b) the Image-to-Class
1: Set each element in D to ∞;
comparison. Matched features are indicated by the same symbol.
2: D[0, 0, 1 : K ] = 0;
3: l = max{l, |M − N|};
Different from the point-wise matching (here each patch 4: Compute the local distance matrix C;
is viewed as a data point), our method tries every possible 5: for m = 1 to M do
6: for n = max {1, m − l} ⎧to min {N, m + l} do ⎫
warping path under the temporal constraints then selects the
⎨ D[m − 1, n − 1, 1 : K ], ⎬
one with minimal overall cost. So the warping path with 7: minNeighbour=min D[m − 1, n, 1 : K ], ;
large distance error will not be selected. The Image-to-Class ⎩ D[m, n − 1, 1 : K ] ⎭
distance is the globally optimal cost for matching. Although 8: for k = 1 to K do
occlusions are not directly removed, avoiding large distance 9: D[m, n, k] =minNeighbour+C[m, n, k];
error by warping is helpful for classification from our experi- 10: end for
11: end for
mental results (see Section III). 12: end for
In addition, a patch of the probe image can be matched 13: dist D I C W = min { D[M, N, 1 : K ]};
to patches of K different gallery images of the same class. 14: return dist D I C W ;
Because the chance that all patches at the same location of the
K images are occluded is low, the chance that a probe patch
is compared to an unoccluded patch at the same location is
III. E XPERIMENTAL A NALYSIS
thus higher. When occlusions occur in probe or/and gallery
images, the Image-to-Image distance may be large. However, In this Section, we evaluate the proposed method using four
our model is able to exploit the information from different databases (FRGC [27], AR [26], TFWM [28] and LFW [29]).
gallery images and reduce the effect of occlusions (Fig. 5). We perform identification tasks according to the three cases
Algorithm 1 summarises the procedure of computing the (i.e., Uvs.O, Ovs.U and Ovs.O) described in Section I. We first
Image-to-Class distance between a probe image and a consider the scenario with occlusions occur only in probe
class. l is the window width and usually set to 10% images (i.e., Uvs.O) and test our method using different
of max {M, N} [22]. Computational complexity is analysed number of gallery images per subject. We will demonstrate
in Section III-D6. that our method works well even when a very limited number
2040 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

of images are available for each subject. Next, we consider

the situation that occlusions exist in gallery images, which is
a case most of the current works do not take account. We fix
the number of gallery images per subject and conduct experi-
ments step by step: firstly the probe images are not occluded Fig. 6. Sample images from the FRGC database with randomly located
occlusions.
(i.e., Ovs.U); and then both the gallery and probe images are
occluded (i.e., Ovs.O). Note that, for comparison purpose, the
experiments on the FRGC and the AR databases also include
the case that no occlusion is presented in both gallery and The FRGC database contains 8,014 images from 466 sub-
probe images to confirm that DICW is also effective in general jects in two sessions. These images contain variations such as
conditions. In addition, we also extend DICW to verification illumination and expression changes, time-lapse, etc. Similar
tasks with faces containing large uncontrolled variations. to the work in [7], an image set of 100 subjects (eight images
Note that in all experiments, the gallery image set is disjoint in two sessions are selected for each subject), is used in
with all probe sets. Considering that the gallery and probe experiments. To simulate the randomly located occlusions,
images are at the same scale, in the experiments, the probe we create an occluded image set by replacing a randomly
images and the gallery images are partitioned into the same located square patch (size of 10% to 50% of the original
number of patches, i.e., M = N as defined in Section II-B. image) from each image in the original image set with a
As recommended in the work [30], the Euclidean distance and black block (Fig. 6). We design experiments according to
the Cosine distance are used as local distance measurements the three occlusion scenarios: Uvs.O, Ovs.U and Ovs.O.
for the pixel intensity feature and the LBP feature [31], There are 2,400 testing samples for each scenario. All images
respectively. are cropped and re-sized to 80 × 65 pixels and the patch
We quantitatively compare DICW with some representative size is 6 × 5 pixels (the effect of patch size is discussed
methods in the literature: the supervised linear SVM [32] in Section III-D1).
using PCA [33] for feature extraction (PCA + LSVM), the 1) Uvs.O: For each subject, we select K = 1, 2, 3 and 4
reconstruction based SRC [8] as introduced in Section I, the unoccluded images respectively to form the gallery sets and
Image-to-Class distance based Naive Bayes Nearest Neigh- use the other four images with synthetic occlusions as the
bour (NBNN) [34] as ours, and the baseline, Hidden Markov probe set. Fig. 7 shows the recognition results with different
models (HMM) [35] which also considers the order informa- values of K . The correct identification rates of all methods
tion in a face. We use the difference patch representation as increase when more gallery images are available (i.e., greater
defined in Section II-A in NBNN and DICW. For comparison value of K ). When there are multiple gallery images per class
purpose, we also report the results of using the original patche and no occlusion (level = 0%) in images, HMM performs
(referred to OP-NBNN and OP-Warp, respectively). better than the supervised method SVM and the local matching
Note that NBNN is a local patch based method which also based NBNN. But its performance is significantly affected by
exploits the Image-to-Class distance. But it does not consider the increasing occlusions. In addition, when K = 1, HMM
the spatial relationship between patches like ours. To improve performs worst among all methods since there are not enough
the performance, a location weight α [34] is used in NBNN gallery images to train a HMM for each class. For NBNN
to constrain matching patches according to their locations. and DICW, using the difference patch achieves better results
We tested different values of α and found that the performance than using the original patch (i.e., OP-NBNN and OP-Warp).
of NBNN is highly dependent on the value of α and different Especially, by comparing DICW with OP-Warp, and NBNN
testing data (e.g., different occlusion level, location) requires with OP-NBNN, it can be found that difference patches
different value even within the same database. So we also improve the results of DICW more significantly than that of
reported the best result for each test with the optimal α NBNN. As introduced in Section II-A, the difference patches
value (as OP-NBNN-ub and NBNN-ub). The performance of are generated by the spatially continuous patches so they
OP-NBNN-ub and NBNN-ub can be seen as the upper bound enhance the order information within a patch sequence, which
of the performance of NBNN, which is a competitive compar- is compatible with DICW. With the optimal location weights,
ison for DICW. NBNN-ub and OP-NBNN-ub perform better than SVM. When
K = 1, 2, 3 and 4, the average rates for the six occlusion levels
of DICW are 2.3%, 4.3%, 5.5% and 4.4% better than that of
A. Face Identification With Randomly Located Occlusions NBNN-ub, respectively. When the occlusion level = 0%, the
We first evaluate the proposed method using the Face performance of SRC is better than DICW. However, the perfor-
Recognition Grand Challenge (FRGC) database [27] with mance drops sharply when the degree of occlusion increases.
randomly located occlusions. Note that in each image, the When K = 1, the Image-to-Class distance degenerates to the
locations of occlusions are randomly chosen and unknown to Image-to-Image distance. DICW, which allows time warping
the algorithm. Especially, in the Ovs.O scenario, the locations during matching, still achieves better results while the level of
of occlusions in the gallery images are different from those in occlusion increases.
the probe images. We use these images with randomly located 2) Ovs.U and Ovs.O: We fix the value of K to 4 and con-
occlusions to evaluate the effectiveness of DICW when there sider that occlusions exist in the galley set. For each occlusion
is no prior knowledge of the occluded location. level (from 0% to 50%), we conduct experiments with the
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2041

Fig. 7. Uvs.O: identification results on the FRGC database with different number of gallery images per subject: a) K = 1, b) K = 2, c) K = 3 and d) K = 4.

Fig. 8. a) Ovs.U and b) Ovs.O: identification results on the FRGC database with occlusions in gallery or/and probe sets.

following settings: 1) 400 occluded images (four images per occluded images as the gallery set and 400 occluded images
subject) from the original set as the gallery set and 400 images as the probe set (Ovs.O). Note that the images in the gallery
from the unoccluded set as the probe set (Ovs.U) and 2) 400 set are different from those in the probe sets. Fig. 8 shows the
2042 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

recognition results. The methods (e.g., HMM, SVM, SRC) TABLE II

which include occluded gallery images for training/modelling I DENTIFICATION R ESULTS ON THE AR D ATABASE
perform poorly in these two cases. NBNN does not perform W ITHOUT O CCLUSIONS (K = 7)
consistently in Ovs.U and Ovs.O. Using the original patch
(i.e., OP-NBNN) performs better than using the difference
patch (i.e., NBNN) in Ovs.U. For DICW, using the differ-
ence patch is always better than using the original patch
(i.e., OP-Warp). This confirms that the difference patch works
better with DICW, as analysed before. DICW outperforms the
best of NBNN (i.e., NBNN-ub) by a larger margin of 5.5%
(Fig. 8a) and 8.1% (Fig. 8b) on average than that (4.4% in
Fig. 7d) in the Uvs.O tested with K = 4. These results confirm
the effectiveness and robustness of DICW when the gallery
and probe images are occluded. On the whole, our method
performs consistently and outperforms other methods in all
three occlusion cases.
reconstruction based method (e.g., SRC) and local matching
based methods (e.g., NBNN and DICW) achieve relatively
B. Face Identification With Facial Disguises satisfactory results. DICW significantly outperforms NBNN
We next test the proposed method on the AR database [26] and is still slightly better than the upper bound of NBNN
which contains real occlusions. First, we consider that no (i.e., NBNN-ub).
occlusion is present in both gallery and probe sets. Next, we 2) Uvs.O: The unoccluded frontal view images with various
conduct experiments according to the three occlusion cases. expressions are used as the gallery images (eight images per
DICW does not rely on the prior knowledge of occlusions. subject). For each subject, we select K = 1, 2, 4, 6 and
We will demonstrate that it works well in both general and 8 images to form the gallery sets, respectively. Two separate
difficult situations later. image sets (200 images each) containing sunglasses (cover
The AR database contains over 4,000 colour images about 30% of the image) and scarves (cover about 50% of
of 126 subjects’ faces. For each subject, 26 images in the image) respectively are used as probe sets. Fig. 9 shows
total are taken in two sessions (two weeks apart). These the recognition results. The correct identification rates increase
images suffer from different variations in facial expres- when more gallery images are available. HMM and SVM are
sions, illumination conditions and occlusions (i.e., sunglasses generic training based methods and are unable to deal with
and scarf, as shown in Fig. 5). Similar to the works unseen occlusions in the probe images. In the scarf testing set,
in [6]–[8], [20], [36], and [37], a subset of the AR database the performance of SRC deteriorates significantly compared
(50 men and 50 women) is used [38]. All images are cropped with that on the sunglasses set due to the occluded area is
and re-sized to 83×60 pixels and the patch size is 5×5 pixels. much larger. Local matching based NBNN and DICW perform
1) Without Occlusion: We have evaluated the performance better than others on the whole. With the optimal location
of DICW when no occlusion exists in both gallery and probe weights, NBNN-ub achieves very comparable performance to
sets in Section III-A (i.e., occlusion level = 0% in the DICW. But DICW is slightly superior. Even at K = 1, DICW
experiments). In this section, we adopt the setting in [8] still achieves 90% and 83% on the sunglasses set and scarf
using images without occlusions to further test DICW. For set, respectively.
each subject, 14 images are chosen (four neutral faces with With the same experimental setting, we also compare DICW
different illumination conditions and three faces with different with the state-of-the-art algorithms (using eight gallery images
expressions in each session). Seven images from Session 1 are per subject, K = 8). The results are shown in Table III.
used as the gallery set and the other seven from Session 2 Only the pixel intensity feature is used except the MLERPM
as the probe set. Table II shows the identification rates. method. MLERPM, which is also a local matching based
HMM does not perform as good as others. This may be method as ours, uses SIFT [39] and SURF [40] features to
due to other variations such as illumination and expression handle the misalignment of images. Note that compared with
changes in the training images. Again, the difference patch other methods, DICW does not require training. It achieves
does not improve NBNN comparing with the original patch comparable or better recognition rates among these meth-
(i.e., OP-NBNN). With the optimal location weights, the dif- ods and with a relatively low computational complexity
ference patch (i.e., NBNN-ub) is 3.7% better than the original (see Section III-D6). In the scarf set, albeit the fact that nearly
patch (i.e., OP-NBNN-ub). For DICW, using the difference half of the face is occluded, only 2% images are misclassified
patch is 3.1% better than using the original patch (OP-Warp). by DICW. To the best of our knowledge, this is the best result
As analysed in Section III-A, the difference patch can enhance achieved on the scarf set under the same experimental setting.
the relative order of adjacent patches, the results in Table II 3) Ovs.U and Ovs.O: For the Ovs.U scenario, we select
also indicates that the difference patch is more compatible with four images with sunglasses and scarves to form the gallery set
these methods which considers the order information. When and eight unoccluded images as the probe set. For the Ovs.O
there is no occlusion in the gallery and probe images, both scenario, we conduct two experiments: 1) two images with
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2043

Fig. 9. Uvs.O: identification results on the AR database with (a) sunglasses occlusion and (b) scarf occlusion.

TABLE III
U VS .O: C OMPARISON OF DICW AND THE -S TATE - OF - THE -A RT M ETHODS (K = 8)

Fig. 11. Sample images from the TFWM database.

NBNN and DICW perform better. Comparing OP-NBNN with

OP-NBNN-ub, and NBNN with NBNN-ub, it can be found
that the performance of NBNN is highly dependent on the
optimal location weights. Overall, DICW consistently outper-
forms the best of NBNN (i.e., NBNN-ub) by about 4% on
Fig. 10. Ovs.U and Ovs.O: identification results on the AR database with
occlusions in gallery or/and probe sets. average.

scarves as the gallery set and two images with sunglasses as C. Face Identification With General Occlusions
the probe set; 2) vice versa. Note that with this setting, in each in Realistic Environment
test the occlusion type in the gallery set is different from that In this Section, we test our method on the The Face We
in the probe set, which is very challenging for recognition. Make (TFWM) [28] database captured under natural and arbi-
The results are shown in Fig. 10. On the gallery set which trary conditions. It has more than 2,000 images which contains
contains occluded faces, the results of HMM and SVM are frontal view faces of strangers on the streets with uncon-
much worse than others as expected. In the Ovs.O testing, trolled lighting. The sources of occlusions include glasses,
there are only two gallery images per subject. It is very difficult sunglasses, hat, hair and hand on the face. Besides occlusions,
for SRC to reconstruct an unoccluded probe image with these images also contain expression, pose and head rotation
such limited number of gallery images. Local matching based variations. In our experiments, we use images of 100 subjects
2044 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

Fig. 12. Identification rates (%) on the TFWM database.

Fig. 13. Identification rates (%) with respect to the patch size.

(ten images per subject) containing various types of occlusions

(Fig. 11). For each subject, we choose K = 1, 3, 5 and 8
unoccluded images as gallery sets, respectively, and the
remaining two images as the probe set. Occlusions occur at
random in the gallery or probe set or in both. This includes
all the three occlusion scenarios in Section I. The face area
of each image is cropped from the background and re-sized
to 80 × 60 pixels and the patch size is 5 × 5 pixels. Only the
pixel intensity feature is used in all methods.
The recognition results are shown in Fig. 12. Note that
the images used in the experiments are not well aligned due
to the uncontrolled variations. Some occlusions (e.g., hand)
have very similar texture as the face, which are difficult to
be detected by skin colour based models [42]. NBNN, which
only relies on the texture similarity without considering the Fig. 14. Identification rates (%) with respect to the overlap ratio comparing
structural constraint of a face, does not achieve comparable with using the difference patches.
performance as ours. As more gallery images are available,
the accuracies of all methods increase. When K = 8, most
methods reach a bottleneck with the rate around 65%. DICW 2) The Effect of Patch Overlap: In the previous experiments
outperforms these methods by a notable margin. we used the difference patch to enhance the textured features
in patches. It is interesting to see if the overlapping patch has
this similar effect. We conducted experiments on the AR data-
D. Discussion base to investigate this since it contains real occlusions with
1) The Effect of Patch Size: To investigate this the impact different textures. We selected four unoccluded images from
of patch size on the performance, we use 400 unoccluded Session 1 for each subject as the gallery set and two images
images (size of 80×65 pixels) of 100 subjects from the FRGC with sunglasses and scarves from Session 2 as the probe set
database as the gallery set and 400 images in each of six probe so the testing dataset contains variations of occlusions and
sets, which contain randomly located occlusions from 0% to illumination changes. We tested the use of different patch sizes
50% level, respectively. We test DICW with the patch sizes (4×4 to 16×15 pixels) with different overlap ratios (0%, 25%,
from 3 × 3 pixels to 10 × 10 pixels. Note that we employ 50%, 75%) and compared their results with that of using the
this dataset because the location and size of the occlusions is difference patch. 25% ratio means the adjacent patches have
independent to the patch size. a 25% horizontal overlap. So the larger the ratio is, the larger
The correct identification rates with respect to the patch size the number of patches will be in each image sequence. Note
are shown as Fig. 13. There is no sharp fluctuation in each that 0% overlap ratio means using the original patches (with
of the rate curve when the patch size is less than or equal intensity features).
to 6 × 5 pixels. Our method is robust to different patch sizes Fig. 14 shows the recognition results. On the whole, large
in an appropriate range despite the ratio of occlusions. The overlap ratio leads to better accuracy. Note that higher overlap
relatively smaller patches lead to better recognition rate since ratio also increases the number of patches in each image
they provide more flexibility to use spatial information than the sequence, which leads to a higher computational cost. For
larger ones. Based on the experimental results, sizes smaller small patch sizes (i.e., 4 × 4, 5 × 5 and 8 × 6 pixels), using the
than 6 × 5 pixels are recommended. difference patch yields significantly better results than using
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2045

Fig. 16. Sample images of the same subject from the AR database without
alignment (AR-VJ).

TABLE IV
I DENTIFICATION R ATE (%) ON THE AR-VJ D ATASET

Fig. 15. Identification rates (%) of using different image descriptors and the
difference patches.

the overlapping patch. This is compatible with our analysis in

Section II-A. A difference patch is the approximation of the
first-order derivative of adjacent small patches. The first-order
derivative operator is sensitive to edges, which is able to to the misalignment of face images, we use a subset of the AR
enhance the textured regions. When the patch becomes large database with 110 subjects (referred to AR-VJ) used in the
(i.e., 10×10, 10×15 and 16×15 pixels), the advantage of using work in [45]. The faces in AR-VJ are automatically detected
the difference patch is not obvious. This is reasonable since the by the Viola & Jones detector [46] and cropped directly from
texture in a large patch is less uniform. Note that the overall the images without any alignment. Different from the images
performance of using the small patch is better than that of in the original AR database which are well cropped (Fig. 5),
using the large patch. DICW is compatible with the small patch these images contain large crop and alignment errors as shown
as analysed in Section III-D1 so in the experiments we used in Fig. 16.
the best one, the difference patch instead of the overlapping Following the same experimental setting in [45],
patch. seven images of each subject from the first session are
3) The Effect of Image Descriptor: In Section III-D2, our used as the gallery set and the other seven images from the
experiments indicate that the difference patch leads to better second session as the probe set. All images are re-sized to
accuracy since it is able to enhance the textured regions in 65 × 65 pixels and the patch size is 5 × 5 pixels. As analysed
a face image. In this section we will carry out experiments in the last section, we use the LBPu2 8,2 descriptor [31] for
to compare the discriminative power of the proposed dif- feature extraction to handle the illumination variations.
ference patches and other local image descriptors such as The recognition results are shown in Table IV. DICW
2D-DCT (Discrete Cosine Transform coefficients), Gabor [43], outperforms other methods and achieves very close result to
LBP [31] and dense SIFT [44]. We use the same dataset in P2DW-FOSE [45], which is also a training-free method like
Section III-D2 and test both small patch size (i.e., 5×5 pixels) ours. But different from DICW, which performs warping on
and large patch size (i.e., 16 × 15 pixels). the patch level, P2DW-FOSE is a pseudo 2D warping method
Fig. 15 shows the recognition results. The 2D-DCT feature on the pixel level and its time complexity is quadratic in the
is not as discriminative as others so it performs worst. For large number of pixels [45].
patch size, as we analysed before, the difference patch does 5) Extension to Face Verification in the Wild: In this
not perform very well. For small patches, the performance of Section, we extend DICW for face verification tasks using the
difference patch is comparable with that of SIFT and LBP. The Labeled Faces in the Wild (LFW) database [29], which is the
Gabor features do not perform better than the difference patch most active benchmark for FR. The task of face verification
since the patch is too small to extract discriminative features. under the LFW database’s protocol is to determine if a pair of
Note that the computation of difference patch is much simpler face images belongs to the same subject or not. Note that in the
than other images descriptors. From Fig. 15 we can see, the verification of each pair, it is a Image-to-Image comparison.
local image descriptor is able to strengthen DICW when the So the experiments on the LFW database can be considered as
image contains uncontrolled variations such as illumination an evaluation for the effectiveness of DICW when only time
changes and occlusions. When dealing with the uncontrolled warping is used (no within-class warping).
data, applying these local features can further improve the The LFW database contains 13,233 face images of
performance of DICW. 5,749 subjects collected from the Internet. These images
4) Robustness to Misalignment: The face registration error are captured in uncontrolled environments and contain large
can largely degrade the recognition performance [1] as we variations in pose, illumination, expression, time-lapse and
mentioned in Section I. To evaluate the robustness of DICW various types of occlusions (Fig. 17). Following the testing
2046 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

Fig. 17. Sample images from the LFW database (six matched image pairs
for six subjects).

protocol of View 2, we use the most difficult experimental

setting: restricted unsupervised setting where no class label
information is available. In View 2, there are 3,000 matched
(i.e., positive) and 3,000 mismatched (i.e., negative) image
pairs. They are equally divided into ten randomly generated
sets and the final verification performance is evaluated by
Fig. 18. ROC curves of the-state-of-the-art methods and DICW on the LFW
the ten-fold cross-validation. Here image pairs are classified database.
into the same subject or different subjects by thresholding
TABLE V
on their distance. We use the LFW-a version and LBPu2 8,2
A REA U NDER ROC C URVE (AUC) ON THE LFW D ATABASE
as feature descriptor. All images are cropped and re-sized to
U NDER U NSUPERVISED S ETTING
150 × 80 pixels and the patch size is 3 × 3 pixels.
Chen et al.’s work [49] produces very competitive results on
the LFW database by using the high-dimensional LBP feature.
It is confirmed that features sampled at facial landmarks lead
to better recognition performance than those sampled from
regular grids. Motivated by this, we also select 25 land-
marks [50] of the inner face and follow the similar process
as in [49]: 1) normalise the unaligned images according to
2 facial landmarks (i.e., the tip of the nose and the centre of
the mouth), and 2) extract image blocks (size of 30×30 pixels)
centred around 25 facial landmarks from each image. Each
block is partitioned into 3 × 3 pixels patches which are each image is still maintained so our patch based DICW can be
then concatenated to form a sequence. The original DICW performed directly on the improved features by this process.
algorithm is performed according to each block (i.e., sequence) We compare DICW with other methods under the same
and a corresponding distance is generated respectively. The testing protocol without outside training data. In the exper-
sum of these distances is the final distance for each image iments, only LBPu2 8,2 descriptor [31] is used. We draw the
pair. We refer our method with this strategy (i.e., sampling the ROC (Receiver Operating Characteristic) curves of DICW
features around landmarks) as DICW-L and the original DICW and other state-of-the-art methods in Fig. 18. It shows the
(i.e., sampling features from regular grids) as DICW-G. performance of DICW-G is better than other methods which
LFW is an extremely challenging database containing large use only single feature such as SD-MATCHES (SIFT [39]),
uncontrolled variations, especially pose changes. As presented H-XS-40 (LBP [31]), GJD-BC-100 (Gabor [43]), LARK
in [51], the first several principal components (PCs) usually (locally adaptive regression kernel descriptor [52]) and LHS
capture these uncontrolled variations in the principal compo- (local higher-order statistics [53]). When extracting features
nent analysis (PCA) subspace [33]. Therefore, we adopt the around facial landmarks, the performance of DICW is further
component analysis process in [51] to remove the first several improved with a large margin. The area under the ROC curve
PCs for performance improvement by: (AUC) of DICW-L is 0.874 as shown in Table V, which
F  = F − X i X iT F (7) is the best among all methods. These experimental results
confirm the effectiveness of DICW even only time warping
where F is the original feature vector of an image by is performed.
concatenating all the patch features of the image sequence 6) Computational Complexity and Usability Analysis: From
(i.e., P or G k in Section II-B) and X i is the first i components Algorithm 1 in Section II-C we can see that the time com-
in the PCA subspace. We quantitatively test the value of i plexity of DICW for computing the distance between a query
using the View 1 dataset provided by the LFW database and set image and an enrolled class is O(max {M, N}l K ), where
the optimal value i = 8. F  is the improved feature vector used M, N are the numbers of patches in each probe sequence
in the experiments for the LFW database. In this way, the large and gallery sequence, respectively. l is the window width
uncontrolled variations can be reduced to some extent. At the as mentioned in Section II-B. For better readability, here
same time, different from the general dimension reduction we use M  to represent max{M, N}. The number of gallery
operation (i.e., the original PCA), the topological structure of images per class K is very small compared with the number
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2047

TABLE VI
C OMPARISON OF AVERAGE RUNTIME ( S )

of patches M  in each sequence (i.e., K

M  ). Thus
the complexity is represented as O(M  l). Note that usually
l = 10%M  , so the warping distance can be obtained very
efficiently. On the other hand, the computational cost of the
reconstruction based method SRC is very high [8]. To facilitate
intuitive comparisons, Table VI shows the runtime of DICW
and SRC3 for classifying a query image under the same setting
as the experiments of Table III using Matlab implementation
(running on a platform with quad-core 3.10GHz CPUs and
8 GB memory). DICW is about 15 times faster than SRC [8]
when classifying a query image.
Compared with the reconstruction based approaches, which
represent a query image using all enrolled images, DICW
computes the distance between the probe image and each
enrolled class independently. So in the real FR applications,
the distance matrix can be generated in parallel and the
enrolled database can be updated incrementally. This is very
practical for the real-world applications.

IV. F URTHER A NALYSIS AND I MPROVEMENT Fig. 19. (a) The probe image from class 74. (b) Classification result (class 5)
In the previous sections, we evaluate DICW using exten- by NBNN. (c) Classification result (class 74) by DICW. Distance to each class
computed by (d) NBNN and by (e) DICW.
sive experiments with face images with various uncontrolled
variations. In this section we will further analysis why the
DICW works compared with similar methods, and when and
why it will fail. We also discuss the idea for improving the
performance of DICW.
patches within a proper range. This is guaranteed by the four
NBNN [34] presented in the previous sections is a similar
constraints mentioned in Section II-B. In addition, with the
method to ours. It also calculates the Image-to-Class distance
help of Dynamic Programming, DICW actually tries every
between a probe patch set and a gallery patch set from a given
possible combination of matching correspondence of patch
class. The difference is that it does not consider the spatial
pairs so the final matching is the global optimum for the
relationship between patches like ours and each probe patch
probe patch set and the gallery patch set. Compared with
can be matched to any patches from any location in the gallery
NBNN, DICW considers both the texture similarity and the
patch set. Fig. 19 is an illustration example. The occluded
geometry similarity of patches. The work in [23] points out
probe image is from class 74 but is incorrectly classified to
that the contextual information between facial features plays
the class 5 by NBNN. Actually the images from class 74
an important role in recognition. Our work confirms their
and class 5 are not alike. But the texture of sunglasses is
observation. Although a location weight can be adopted in
very similar to that of beard in class 5. Without the location
NBNN, the weight needs to be manually set for different
constraint, the beard patches are wrongly matched to the
testing dataset as analysed before, which is not suitable
sunglasses thus the distance is affected by this occlusion.
for practical applications. In DICW, the order constraint is
On the other hand, DICW keeps the order information and
naturally encoded during distance computation.
matches patches within a proper range which leads to correct
DICW represents a face image as a patch sequence which
classification.
maintains the facial order of a face. To some extent, the
NBNN calculates the distance between two patch sets and
geometric information of a face is reduced from 2D to 1D.
the overall distance is the sum of patch-pair distances. On the
However, the direct 2D image warping is an NP-complete
other hand, in DICW, the probe and gallery patch set are
problem [55]. P2DW-FOSE mentioned in Section III-D4 is a
ordered. The spatial relationship between patches is encoded.
pseudo 2D warping method but with a remarkably large com-
When a probe patch is matched to a gallery patch, the
putational cost (i.e., quadratic in the number of pixels) [45].
following probe patches will only be matched to the gallery
DICW incurs a lower computational cost due to its patch
3 We use the l1_ls package for implementation. http://www.stanford.edu/~ sequence representation. In addition, each patch still contains
boyd/l1_ls/ the local 2D information which is helpful for classification.
2048 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

TABLE VII
I DENTIFICATION R ATES (%) OF DICW AND THE I MPROVEMENT
S CHEME ON THE AR D ATABASE

Fig. 20. (a) A probe image from class 51. (b) The wrong class (class 72)
classified by DICW. (c) The gallery image from class 51.

DICW is much faster than before since the number of subset

patches is much smaller than that of the whole sequence).
Generally, more experts will lead to higher accuracy since
this increases the diversity of decision views, which is more
robust to different variations. But this will also raise the
whole computational cost, which needs to be considered
to keep a balance between accuracy and computation. The
improvement is more obvious when the number of image per
class is limited. A preliminary study of using this scheme to
Fig. 21. Random selection and majority voting scheme for improving the improve DICW when K = 1 is discussed in [56].
performance of DICW.

A. Improving the Performance With Random V. C ONCLUSION AND F UTURE W ORK

Selection and Majority Voting Scheme
We have addressed the problem of face recognition with
Fig. 20 shows a fail example which can not be correctly occlusions in uncontrolled environments. Different from most
classified by both DICW and NBNN. The discriminative eyes of the current works, we consider the situation that occlusions
region is occluded by sunglasses, which makes recognition exist in both gallery and probe sets. We proposed a novel
difficult. In addition, a probe face with sunglasses (Fig. 20a) approach, Dynamic Image-to-Class Warping (DICW), which
is more similar to a gallery face with glasses (Fig. 20b) in the considers the contextual order of facial components, for the
feature space, which leads to misclassification. recognition of occluded faces. We first represent a face image
Looking back to the definition of DICW in Section II-B, as an ordered sequence, then treat the image matching problem
although warping is helpful for avoiding large distance error as the process of finding optimal alignment between a probe
caused by occlusions, the occluded area is not directly sequence and a set of gallery sequences. Finally, we employ
removed during matching. Here we employ a simple but very the Dynamic Programming technique to compute the Image-
effective scheme for improving the performance of DICW. to-Class distance for classification. Extensive experiments on
As shown in Fig. 21, we do not use all patches in a probe the FRGC, AR, TFWM and LFW face databases show that
sequence for warping, instead, we randomly select a subset of DICW achieves promising performance when handling various
patch set then compute the Image-to-Class distance based on types of occlusions. In the most challenging cases where
this subset. We repeat this n times and generate a class label occlusions exist in both gallery and probe sets and only
(the class with the shortest distance) each time according to a limited number of gallery images are available for each
the calculated distance. Finally, the final class label is decided subject, DICW still performs satisfactorily. DICW can be
by majority voting by n experts. With random selection, it is applied directly to face images without performing occlusion
possible to skip the occluded patches. It is also possible detection in advance and does not require a training process.
that the occluded patches are chosen but this effect will be All of these make our approach more applicable in real-world
eliminated by the majority voting strategy since we assume scenarios. Given its merits, DICW is applicable and extendible
that the occluded areas only take up a small part of a to deal with other problems caused by local deformations
face. This assumption is reasonable since if most parts of in FR (e.g., the facial expression problem), as well as other
a face are occluded, even a human being will feel difficult object recognition problems where the geometric relationship
to recognise it. Different from the occlusion detection based or contextual information of features should be considered.
methods which attempt to detect and remove occlusion area
as we mentioned before, this simple strategy does not rely on
R EFERENCES
any prior knowledge nor any data-dependent training.
Here we use the same setting to Section III-D2. [1] H. K. Ekenel and R. Stiefelhagen, “Why is facial occlusion a challeng-
We randomly select 15% patches in a sequence each ing problem?” in Proc. IAPR 3rd Int. Conf. Biometrics (ICB), 2009,
time as an expert and select n = 50 experts in total. Since pp. 299–308.
[2] M. Storer, M. Urschler, and H. Bischof, “Occlusion detection for
this scheme is based on random selection, we repeat the whole ICAO compliant facial photographs,” in Proc. IEEE Comput. Soc.
classification process ten times and calculate the average Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2010,
identification rate. The results are shown in Table VII. The pp. 122–129.
[3] D. Lin and X. Tang, “Quality-driven face occlusion detection and
performance of DICW is improved by 2% on average by using recovery,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
only 50 experts (Note that for each expert, the computation of Jun. 2007, pp. 1–7.
WEI et al.: DYNAMIC IMAGE-TO-CLASS WARPING FOR OCCLUDED FR 2049

[4] T. Hosoi, S. Nagashima, K. Kobayashi, K. Ito, and T. Aoki, [27] P. J. Phillips et al., “Overview of the face recognition grand challenge,”
“Restoring occluded regions using FW-PCA for face recognition,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.
in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 1. Jun. 2005, pp. 947–954.
Workshops (CVPRW), Jun. 2012, pp. 23–30. [28] D. Miranda. The Face We Make. [Online]. Available:
[5] R. Min, A. Hadid, and J.-L. Dugelay, “Improving the recognition of http://www.thefacewemake.org, accessed 2014.
faces occluded by facial accessories,” in Proc. IEEE Int. Conf. Autom. [29] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled
Face Gesture Recognit. (FG), Mar. 2011, pp. 442–447. faces in the wild: A database for studying face recognition in
[6] H. Jia and A. M. Martinez, “Face recognition with occlusions in the unconstrained environments,” Dept. Comput. Sci., Univ. Massachusetts,
training and testing sets,” in Proc. IEEE Int. Conf. Autom. Face Gesture Amherst, MA, USA, Tech. Rep. 07-49, Oct. 2007.
Recognit. (FG), Sep. 2008, pp. 1–6. [30] S. U. Hussain, T. Napoléon, and F. Jurie, “Face recognition using local
[7] H. Jia and A. M. Martinez, “Support Vector Machines in face recog- quantized patterns,” in Proc. Brit. Mach. Vis. Conf. (BMVC), 2012,
nition with occlusions,” in Proc. IEEE Conf. Comput. Vis. Pattern pp. 99.1–99.11.
Recognit. (CVPR), Jun. 2009, pp. 136–141. [31] T. Ahonen, A. Hadid, and M. Pietikainen, “Face description with local
[8] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face binary patterns: Application to face recognition,” IEEE Trans. Pattern
recognition via sparse representation,” IEEE Trans. Pattern Anal. Mach. Anal. Mach. Intell., vol. 28, no. 12, pp. 2037–2041, Dec. 2006.
Intell., vol. 31, no. 2, pp. 210–227, Feb. 2009. [32] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for sup-
[9] A. I. Naseem, B. R. Togneri, and C. M. Bennamoun, “Linear regression port vector machines,” ACM Trans. Intell. Syst. Technol., vol. 2,
for face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 3, pp. 27:1–27:27, Apr. 2011. [Online]. Available: http://www.
no. 11, pp. 2106–2112, Nov. 2010. csie.ntu.edu.tw/~cjlin/libsvm
[10] M. Yang and L. Zhang, “Gabor feature based sparse represen- [33] M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” in
tation for face recognition with Gabor occlusion dictionary,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 1991,
Proc. 11th Eur. Conf. Comput. Vis. (ECCV), vol. 6316. 2010, pp. 586–591.
pp. 448–461. [34] O. Boiman, E. Shechtman, and M. Irani, “In defense of nearest-neighbor
[11] D. Zhang, M. Yang, and X. Feng, “Sparse representation or based image classification,” in Proc. IEEE Conf. Comput. Vis. Pattern
collaborative representation: Which helps face recognition?” Recognit. (CVPR), Jun. 2008, pp. 1–8.
in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Nov. 2011, [35] A. V. Nefian and M. H. Hayes, III, “Hidden Markov models for face
pp. 471–478. recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.,
[12] R. He, W.-S. Zheng, and B.-G. Hu, “Maximum correntropy criterion vol. 5. May 1998, pp. 2721–2724, doi: 10.1109/ICASSP.1998.678085.
for robust face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., [36] K. Jia, T.-H. Chan, and Y. Ma, “Robust and practical face recognition
vol. 33, no. 8, pp. 1561–1576, Aug. 2011. via structured sparsity,” in Proc. 12th Eur. Conf. Comput. Vis. (ECCV),
[13] X. Wei, C.-T. Li, and Y. Hu, “Robust face recognition under vary- vol. 7575. 2012, pp. 331–344.
ing illumination and occlusion considering structured sparsity,” in [37] R. Weng, J. Lu, J. Hu, G. Yang, and Y.-P. Tan, “Robust feature set
Proc. Int. Conf. Digit. Image Comput. Techn. Appl. (DICTA), 2012, matching for partial face recognition,” in Proc. IEEE Int. Conf. Comput.
pp. 1–7. Vis. (ICCV), Dec. 2013, pp. 601–608.
[14] A. M. Martinez, “Recognizing imprecisely localized, partially occluded, [38] A. M. Martinez and A. C. Kak, “PCA versus LDA,” IEEE Trans. Pattern
and expression variant faces from a single sample per class,” IEEE Anal. Mach. Intell., vol. 23, no. 2, pp. 228–233, Feb. 2001.
Trans. Pattern Anal. Mach. Intell., vol. 24, no. 6, pp. 748–763, [39] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”
Jun. 2002. Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.
[15] X. Tan, S. Chen, Z.-H. Zhou, and F. Zhang, “Recognizing partially [40] H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded up robust
occluded, expression variant faces from single training image per person features,” in Proc. 9th Eur. Conf. Comput. Vis. (ECCV), vol. 3951. 2006,
with SOM and soft k-NN ensemble,” IEEE Trans. Neural Netw., vol. 16, pp. 404–417.
no. 4, pp. 875–886, Jul. 2005. [41] Z. Zhou, A. Wagner, H. Mobahi, J. Wright, and Y. Ma, “Face recognition
[16] X. Tan, S. Chen, Z.-H. Zhou, and J. Liu, “Face recognition under with contiguous occlusion using Markov random fields,” in Proc. IEEE
occlusions and variant expressions with partial similarity,” IEEE Trans. 12th Int. Conf. Comput. Vis. (ICCV), Sep./Oct. 2009, pp. 1050–1057.
Inf. Forensics Security, vol. 4, no. 2, pp. 217–230, Jun. 2009. [42] W.-H. Lai and C.-T. Li, “Skin colour-based face detection in
[17] S. Liao, A. K. Jain, and S. Z. Li, “Partial face recognition: Alignment- colour images,” in Proc. IEEE Int. Conf. Adv. Video Signal-Based
free approach,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 5, Surveill. (AVSS), Nov. 2006, pp. 56–61.
pp. 1193–1205, May 2013. [43] C. Liu and H. Wechsler, “Gabor feature based classification using the
[18] G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “Subspace learning from enhanced fisher linear discriminant model for face recognition,” IEEE
image gradient orientations,” IEEE Trans. Pattern Anal. Mach. Intell., Trans. Image Process., vol. 11, no. 4, pp. 467–476, Apr. 2002.
vol. 34, no. 12, pp. 2454–2466, Dec. 2012. [44] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial
[19] J. Zhu, D. Cao, S. Liu, Z. Lei, and S. Z. Li, “Discriminant analysis with pyramid matching for recognizing natural scene categories,” in Proc.
Gabor phase for robust face recognition,” in Proc. 5th IAPR Int. Conf. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 2.
Biometrics (ICB), Mar./Apr. 2012, pp. 13–18. Jun. 2006, pp. 2169–2178.
[20] C.-F. Chen, C.-P. Wei, and Y.-C. F. Wang, “Low-rank matrix recov- [45] L. Pishchulin, T. Gass, P. Dreuw, and H. Ney, “Image warping for face
ery with structural incoherence for robust face recognition,” in Proc. recognition: From local optimality towards global optimization,” Pattern
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, Recognit., vol. 45, no. 9, pp. 3131–3140, 2012.
pp. 2618–2625. [46] P. Viola and M. J. Jones, “Robust real-time face detection,” Int. J. Com-
[21] X. Tan, S. Chen, Z.-H. Zhou, and F. Zhang, “Face recognition from a put. Vis., vol. 57, no. 2, pp. 137–154, 2004.
single image per person: A survey,” Pattern Recognit., vol. 39, no. 9, [47] K. Tan and S. Chen, “Adaptively weighted sub-pattern PCA for face
pp. 1725–1745, 2006. recognition,” Neurocomputing, vol. 64, pp. 505–511, Mar. 2005.
[22] H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization [48] P. Dreuw, P. Steingrube, H. Hanselmann, and H. Ney, “SURF-face:
for spoken word recognition,” IEEE Trans. Acoust., Speech, Signal Face recognition under viewpoint consistency constraints,” in Proc. Brit.
Process., vol. 26, no. 1, pp. 43–49, Feb. 1978. Mach. Vis. Conf. (BMVC), 2009, pp. 7.1–7.11.
[23] R. Wang, Z. Lei, M. Ao, and S. Z. Li, “Bayesian face recognition [49] D. Chen, X. Cao, F. Wen, and J. Sun, “Blessing of dimensionality: High-
based on Markov random field modeling,” in Proc. IAPR 3rd Int. Conf. dimensional feature and its efficient compression for face verification,”
Biometrics (ICB), vol. 5558. 2009, pp. 42–51, doi: 10.1007/978-3-642- in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2013,
01793-3_5. pp. 3025–3032.
[24] X. Wei, C.-T. Li, and Y. Hu, “Face recognition with occlusion using [50] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active shape
dynamic image-to-class warping (DICW),” in Proc. IEEE Int. Conf. models-their training and application,” Comput. Vis. Image Understand.,
Autom. Face Gesture Recognit. (FG), Apr. 2013, pp. 1–6. vol. 61, no. 1, pp. 38–59, Jan. 1995.
[25] X. Wei, C.-T. Li, and Y. Hu, “Robust face recognition with occlusions [51] S. Z. Li, D. Yi, Z. Lei, and S. Liao, “The CASIA NIR-VIS 2.0
in both reference and query images,” in Proc. Int. Workshop Biometrics face database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
Forensics, Apr. 2013, pp. 1–4. Workshops (CVPRW), Jun. 2013, pp. 348–353.
[26] A. M. Martinez and R. Benavente, “The AR face database,” [52] H. J. Seo and P. Milanfar, “Face verification using the LARK
Autonomous Univ. Barcelona, Barcelona, Spain, Tech. Rep. 24, representation,” IEEE Trans. Inf. Forensics Security, vol. 6, no. 4,
1998. pp. 1275–1286, Dec. 2011.
2050 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 9, NO. 12, DECEMBER 2014

[53] G. Sharma, S. ul Hussain, and F. Jurie, “Local higher-order Zhen Lei received the B.S. degree in automation
statistics (LHS) for texture categorization and facial analysis,” in from the University of Science and Technology of
Proc. 12th Eur. Conf. Comput. Vis. (ECCV), vol. 7578. 2012, pp. 1–12. China, Hefei, China, in 2005, and the Ph.D. degree
[54] J. Ruiz-del Solar, R. Verschae, and M. Correa, “Recognition of faces in from the Institute of Automation, Chinese Academy
unconstrained environments: A comparative study,” EURASIP J. Adv. of Sciences, Beijing, China, in 2010, where he
Signal Process., vol. 2009, pp. 1:1–1:19, Jan. 2009. is currently an Associate Professor. His research
[55] D. Keysers and W. Unger, “Elastic image matching is NP-complete,” interests are in computer vision, pattern recognition,
Pattern Recognit. Lett., vol. 24, nos. 1–3, pp. 445–453, Jan. 2003. image processing, and in particular, face recognition.
[56] X. Wei and C.-T. Li, “Fixation and saccade based face recognition from He has authored over 80 papers in international
single image per person with various occlusions and expressions,” in journals and conferences. He serves as the Area
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Chair the of International Joint Conference on Bio-
Jun. 2013, pp. 70–75. metrics in 2014, the IAPR/IEEE International Conference on Biometric in
2015, and the IEEE International Conference on Automatic Face and Gesture
Recognition in 2015.

Dong Yi received the B.S. degree in electronic

engineering in 2003, the M.S. degree in communica-
tion and information system from Wuhan University,
Wuhan, China, in 2006, and the Ph.D. degree in
Xingjie Wei received the B.E. degree in computer pattern recognition and intelligent systems from the
science and technology from Central South Univer- Institute of Automation, Chinese Academy of Sci-
sity, Changsha, China, in 2009, and the Ph.D. degree ences, Beijing, China, in 2009. His research areas are
in computer science from University of Warwick, unconstrained face recognition, heterogeneous face
Coventry, U.K., in 2014. She is currently a Post- recognition, and deep learning for facial analysis.
Doctoral Researcher with the School of Comput- He has authored and acted as a reviewer for tens
ing Science, Newcastle University, Newcastle upon of articles in international conferences and journals.
Tyne, U.K. She has authored a number of research He has developed face biometric algorithms and systems for the Shenzhen-
papers in the field of face analysis and recognition. Hong Kong Immigration Control Project, the 2008 Beijing Olympic Games,
Her current research interests include biometrics, and the 2010 Shanghai World Expo.
multimedia forensics, and computer vision.

Stan Z. Li received the B.Eng. degree from Hunan

University, Changsha, China, the M.Eng. degree
from the National University of Defense Technology,
Changsha, and the Ph.D. degree from the University
of Surrey, Surrey, U.K. He is currently a Professor
with the National Laboratory of Pattern Recognition
Chang-Tsun Li received the B.E. degree in elec- and the Director of the Center for Biometrics and
trical engineering from the Chung-Cheng Institute Security Research with the Institute of Automa-
of Technology (CCIT), National Defense Univer- tion, Chinese Academy of Sciences, Beijing, China,
sity, Bade City, Taiwan, in 1987, the M.Sc. degree where he is also the Director of the Center for
in computer science from the Naval Postgraduate Visual Internet of Things Research. He was with
School, Monterey, CA, USA, in 1992, and the Ph.D. Microsoft Research Asia, Beijing, from 2000 to 2004, as a Researcher. Prior
degree in computer science from the University to that, he was an Associate Professor with Nanyang Technological University,
of Warwick, Coventry, U.K., in 1998. He was an Singapore. He was elevated to the IEEE fellow for his contributions to the
Associate Professor with the Department of Elec- fields of face recognition, pattern recognition, and computer vision.
trical Engineering, CCIT, from 1998 to 2002, and a His research interests include pattern recognition and machine learning,
Visiting Professor with the Department of Computer image and vision processing, face recognition, biometrics, and intelligent
Science, Naval Postgraduate School, in the second half of 2001. He is video surveillance. He has authored over 200 papers in international journals
currently a Professor with the Department of Computer Science, University of and conferences, and authored and edited eight books. He was an Associate
Warwick. He was the Editor-in-Chief of the International Journal of Digital Editor of the IEEE T RANSACTIONS ON PATTERN A NALYSIS AND M ACHINE
Crime and Forensic from 2009 to 2013. He is currently an Associate Editor I NTELLIGENCE. He is the Editor-in-Chief of the Encyclopedia of Biometrics.
of the EURASIP Journal on Image and Video Processing. He has been He served as the General Cochair of the IEEE International Conference
involved in the organization of many international conferences and workshops, on Automatic Face and Gesture Recognition in 2011, serves/served as the
and also served as a member of the International Program Committees for Program Cochair of the International Conference on Biometrics in 2007, 2009,
many international conferences. His research interests include biometrics, and 2015, the International Joint Conference on Biometrics in 2014, and the
digital forensics, multimedia security, computer vision, image processing, IEEE International Conference on Automatic Face and Gesture Recognition
pattern recognition, evolutionary computation, machine learning, data mining in 2015, and has been involved in organizing other international conferences
bioinformatics, and content-based image retrieval. and workshops in the fields of his research interest.