2015 IEEE 3rd International Conference on Cyber-Physical Systems, Networks, and Applications

XFace: A Face Recognition System for Android Mobile Phones

Jiawei Hu Liangrui Peng Li Zheng

Dept. of Computer Science and Tsinghua National Laboratory for Dept. of Computer Science and
Technology, Information Science and Technology, Technology,
Tsinghua University Dept. of Electronic Engineering, Tsinghua University
Beijing, China Tsinghua University Beijing, China
hjw13@mails.tsinghua.edu.cn Beijing, China zhengli@tsinghua.edu.cn
penglr@tsinghua.edu.cn

Abstract—With the rapid development of mobile computing, images, such as the direction of lighting and shadows, exact
biometric identification technologies including face recognition orientation of the face, expression of the face, and the current
on smart phones become increasingly attractive. There are two mood of the person [5].
main challenges of face recognition for mobile computing: one On the other hand, as a mobile operating system, Android
is the problem of the variations of face images acquired in has reached great success in smartphones and tablets
unconstrained environment; the other is the computing recently. When compared with traditional mobile phones,
limitation of mobile phone platform. This paper presents the Android smart phones have improved greatly in computing
design and implementation of an open source face recognition ability and storage capacity. New versions of Android
system named XFace for Android mobile phone platform,
system have become more and more powerful by integrating
which provides a feasible solution to meet these challenges. The
functions like image capture and image processing. In June
proposed face recognition approach includes face detection,
eye detection, preprocessing for ROI (Region of Interest), LBP 2009, Google released Android Native Development Kit
(Local Binary Pattern) feature extraction, feature (Android NDK [1]) to support native development in C/C++,
dimensionality reduction based on PCA (Principal Component besides the Android Software Development Kit (Android
Analysis) and LDA (Linear Discriminant Analysis), and SDK) that supports Java. Android NDK allows the
minimum distance classifier. The implementation of the developers to develop Android applications using C/C++ and
proposed approach is mainly based on OpenCV (Open Source transplant C/C++ libraries like OpenCV [2] to Android
Computer Vision) SDK. The system framework and user platform [10]. Though it has made great improvements in
interface are also presented. Experimental results on practical recent years, Android still remains the problem of computing
face image samples collected on Android mobile phones proved limitation when compared with the PC platform. So we have
the effectiveness of the proposed method. In future work, it is to make a tradeoff between the computation complexity and
possible to integrate deep learning based face recognition the processing efficiency when implementing this system.
method to the XFace system. This paper presents a prototype face recognition system
that provides a feasible solution to meet the challenges and
Keywords-component; Face Recognition; Face Detection; achieves a satisfactory face detecting and recognizing
Eigenfaces; Fisherfaces; Android Application performance on Android mobile phones. For the face
detection, we used the LBP-based frontal face detector, and
I. INTRODUCTION an effective way of ROI preprocessing method has been
Biometric identification technology, especially human applied to overcome the problem of the variations in face
face recognition, has been a very popular research topic in images. Eigenfaces and Fisherfaces are both employed and
the last decade. With the rapid development of smart phones, tested for face recognition.
it becomes an attractive topic to implement face recognition This document is organized as follows: in Section II, the
system on mobile phone platforms such as Android. There methodologies we applied in this face recognition system are
are mainly two challenges of face recognition for mobile discussed. Section III illustrates the design and
computing: one is the problem of the variations of face implementation of the face recognition system. Section IV
images acquired in unconstrained environment; the other is shows the results obtained from the experiments and the
the computing limitation of mobile phone platform. analysis performed. Finally, conclusions and future work are
The face images are often acquired under a certain presented in Section V.
environment in the traditional way of implementing a face
II. METHODOLOGY
recognition system, but things are different and more
complicated when implementing a face recognition system Face recognition procedure basically consists of two
on mobile phone platform. The current face recognition steps. The first step is the face detection, which rapidly finds
techniques are much less reliable when used in real-world a human face in the photo or video when the human is
conditions for the fact that current face recognition located within a proper distance by the smart phone. The
techniques are very sensitive to exact conditions in the second step is the face recognition that is recognizing the

978-1-4673-7785-0/15 $31.00 © 2015 IEEE 13

DOI 10.1109/CPSNA.2015.12
face as someone known according to the face database 1) Geometrical transformation and cropping: This
collected before. In the system we implemented, another step process would include scaling, rotating, and translating the
of the ROI preprocessing is performed between face input face image so that the eyes are aligned. The distance
detection and face recognition for the fact that the accuracy between the eyes is used to normalize the image [9]. Then
of the face recognition system by using the ROI
the ROI image of N*N pixels is extracted according to the
preprocessing method increased significantly.
position of the eyes. In our system, N is set to 70 as an
A. Face Detection empirical value.̘
Face detection is the process of locating one or more face Figure 1 shows a grayscale face image acquired from the
regions in an image. There are two common types of face mobile phone and its corresponding ROI image obtained by
detector: one is Haar-based face detector; the other is LBP- the first step.
based face detector. We have studied these two face
detection algorithms and evaluated the tradeoff between
detection accuracy and computational complexity, as the face
recognition system has to be implemented on mobile phones.
In this prototype system, we used the LBP-based frontal face
detector as the face detector. Here we reproduce a brief
introduction of LBP, while referring the reader to the
relevant reference.
Local Binary Pattern was invented by Ahonen, Hadid
and Pietikäinen in 2004 [4]. The basic idea of Local Binary (a) (b)
Patterns is to summarize the local structure in an image by
Figure 1. (a) A grayscale face image acquired from the mobile phone; (b)
comparing each pixel with its neighborhood. A formal The corresponding ROI image obtained by the first step.
description of the LBP operator can be given as:
2) Separate histogram equalization for left and right
P −1
­1, x ≥ 0
LBP( xc , yc ) = ¦ 2 p s (v p − vc ), s ( x) = ® (1) sides: This process standardizes the brightness and contrast
p =0 ¯0, x < 0 on both the left- and right-hand sides of the face
independently.
Where (xc, yc) denotes the central pixel in the local image
3) Smoothing: This process reduces the image noise
with intensity vc , v p is the intensity of the neighbor pixel. P
using a bilateral filter which is very good at smoothing most
is the total number of involved neighbor pixels. of an image while keeping edges sharp.
B. Eye detection and ROI preprocessing 4) Elliptical mask: The elliptical mask removes some
remaining hair and background from the face image.
As face recognition is extremely vulnerable to the
The face images before and after the ROI preprocessing
changes of the nearby circumstances, we have performed an
are shown in Figure 2.
experiment and evaluated the performances of the system
with and without the ROI preprocessing method (this method
will be discussed later), the detailed results presented in
Section IV show that, with ROI preprocessing method the
accuracy of the face recognition system increased
significantly.
For the reliability in real-world conditions, the ROI (a) (b) (c) (d)
preprocessing method should apply many other sophisticated Figure 2. The four steps in preprocessing a face image: (a) Geometrical
techniques including facial feature detection (for example, transformation and cropping; (b) Separate histogram equalization for left
eyes detection, nose detection, and mouth detection). Among and right sides; (c) Smoothing; (d) Elliptical mask.
these facial feature detections, eyes detection can be very
helpful for ROI preprocessing, because for frontal faces we
C. Face Recognition
can always assume a person's eyes should be horizontal and
on opposite locations of the face and should have a fairly As two of the commonly discussed face recognition
standard position and size within a face, despite changes in algorithms, Eigenfaces and Fisherfaces are both employed
facial expressions, lighting conditions, camera properties, and tested in this work.
distance to the camera, and so on [5]. 1) Eigenfaces
With both face and eyes detected, we then perform a Eigenfaces, also known as PCA, first used by Turk and
preprocessing method proposed by Shervin Emami in [5]. Pentland in 1991 [6]. PCA is a commonly used technology
The whole preprocessing procedure combines the following for dimensionality reduction in computer vision (particularly
four steps, among which, the first step is to get the Region of in face recognition). It chooses a dimensionality reducing
Interest: linear projection that maximizes the scatter of all projected
samples.

14
 To create a set of eigenfaces, we need a training set of n respectively, and the weights are then stored as coordinates
face images. Each image is treated as one column vector, in the feature space. The Euclidean distances between the
simply by concatenating the columns of pixels in the original test image coordinates and the training image coordinates are
image, resulting in a single column with r x c elements. Then calculated to find out the closest training image. If a
we could turn the training set into a big matrix as threshold is set to the algorithm model, the system will
accept a facial image to the person in the database or reject a
(2) facial image for the minimum distance is still larger than the
threshold.
where χ i is a column vector corresponding to an image.
After that we can calculate the mean u and the covariance III. SYSTEM DESIGN AND IMPLEMENTATION
matrix S as We designed and implemented an open source face
recognition system named XFace for Android mobile phone
n
1
¦ χi
platform. While implementing the system, we have to make
u= (3)
a tradeoff between recognition accuracy and computational
n i=1 complexity of the algorithm due to the limited computing
ability and storage capacity on mobile phone platform. This
n
1 XFace application is hosted on Github,
¦ ( χ i − u) ( χ i − u)
T
S= (4)
https://github.com/hujiaweibujidao/XFace.
n i=1
A. System Overview
Now consider a linear transformation Wopt mapping the There are two main procedures in this face recognition
original image space into a k-dimensional feature space (k system. The first procedure is registration process called
<= n << r x c). PCA chooses the projection Wopt that XFace Registration Module, which includes image
acquisition, ROI preprocessing and training an algorithm
maximizes the determinant of the covariance matrix of the model with these images. While in the recognition process
projected images, that is, called XFace Recognition Module, stages are image
acquisition, ROI preprocessing and predicting the test image
(5) as someone in the face database with the trained algorithm
model. The overview of the system is shown in Figure 3.
where wi ’s are k eigenvectors of covariance matrix S The implementation of system is mainly based on the
corresponding to the k largest eigenvalues. Each of them OpenCV SDK. OpenCV started by Intel in 1999 and
represents an “eigenface”. OpenCV 2.3.1 comes with a programming interface to
2) Fisherfaces Android. What’s more, OpenCV 2.4 comes with the very
Fisherfaces, also referred to as LDA, invented by new FaceRecognizer class for face recognition. The simplest
Belhumeur, Hespanha and Kriegman in 1997 [7]. While implementation of face recognition application for Android
PCA is clearly a powerful way to reduce the dimensionality system is taking full advantages and functionalities of
and represent data, it doesn’t consider any classes and so a OpenCV for Android SDK.
lot of discriminative information may be lost when throwing
components away, whereas LDA performs a class-specific
dimensionality reduction. It is designed to maximize the
between-class scatter while minimizing the within-class
scatter, by calculating the between-class scatter matrix SB ,
and within-class scatter matrix SW , and the optimal
projection is chosen as

W T SBW
Wopt = argmax (6)
W W T SW W
Suppose c is the number of classes in the training set, i.e.,
there are c persons’ faces collected in the face database, so
there are at most c-1 generalized eigenvectors, therefore at
most c-1 “Fisherfaces”.
D. Design of Classfier
The classifier is simply a distance classifier based on
Euclidean distance. The test image is projected onto
Eigenfaces and Fisherfaces in these two approaches, Figure 3. System Overview.

15
 At present, there are mainly three approaches to integrate
OpenCV with Android development:
a) Using JavaCV. JavaCV [3] provides most of
OpenCV functions with a programming interface to Java:
This way is the easiest for only using Java programming
language, but soemtimes it may be less efficient when
compared with the performance of pure C++ implemented
OpenCV SDK.
b) Using OpenCV for Android SDK. This SDK is
developed for Android platform and provides a variety of
functions that support the programming of operating
cameras on Android devices in both Java and native way,
but the FaceRecognizer class can not be instantiated in Java Figure 4. The procedure of registration.
for now (version 2.4.10).
Image training process is choosing Eigenfaces or
c) Using pure C++ implemented OpenCV SDK. The Fisherfaces algorithm to obtain a trained algorithm model
performance of pure OpenCV SDK is the most efficient for and saving the model data to the external storage of the smart
its many optimizations, but it increased the difficulty of phone. Though it seems simple, the OpenCV for Android
development by integrating Android NDK technology. SDK does not provide face recognition functions in Java for
Each approach has its own advantages and now as mentioned earlier. Here we use Android NDK to
disadvantages, and it depends on the functionalities of the implement the face recognition functions in C++ and
specific application to choose the best way. For face compile the native code to a dynamic library, then call these
recognition application, to achieve the best tradeoff between functions in Java programming code through JNI (Java
the computation complexity and the processing efficiency, Native Interface) technology.
Android NDK with pure OpenCV SDK is used rather than C. Recognition Module
pure Android SDK with JavaCV. But when it comes to
operating cameras on Android devices, OpenCV for Android After training the Eigenfaces or Fisherfaces algorithm
SDK provides the most helpful and efficient way to obtain model with training images and corresponding user names,
the image frames from camera and convert the raw image to the recognition process is trying to figure out who a person is
matrix format, so OpenCV for Android SDK is also adopted from a facial image captured from the camera on smart
in this prototype application. phone. The face recognition process is performed with image
frames continuously coming from either the front or rear
B. Registration Module camera. When a proper facial image detected, this image
The first step in registration module is letting the user frame will be passed to the ROI preprocessing procedure and
choose an existing user name from the user list or create a predicted by the trained algorithm model. If a threshold is set
new user name, and then it comes to the image acquisition to the algorithm model, the system will accept a facial image
step. to the person in the database or reject a facial image for the
In the common way of implementing registration minimum distance is still larger than the threshold. The
module, the step of image acquisition is always required to interface of the recognition procedure is shown in Figure 5.
take images on Android devices. But considering the fact
that current face recognition techniques are very sensitive to
exact conditions in the images, we proposed an automatic
image acquisition method to collect the most useful face
images for our algorithm model. The basic idea here is that if
we find a proper face, i.e. the ROI detected in the current
frame of camera, this application automatically passes this
frame of image to the ROI preprocessing stage and then
saves it to the face database stored on the smart phone, which
means the training face images are images needed by our
algorithm model, not the images users casually give us.
Although it somewhat increased the difficulty of registration
module, the accuracy of the system increased significantly.
The interface of registration procedure is shown in Figure
Figure 5. The procedure of face recognition.
4. The user can click on the preview image to switch
between the front and rear cameras.

16
 IV. EXPERIMENT RESULTS B. Results of Eigenfaces and Fisherfaces
In order to evaluate the performance of this face
A. Results of ROI preprocessing recognition system, we collected a face dataset with all the
In real-world conditions, the face images acquired from images taken by the front camera of Google Nexus 5, which
the mobile phones vary a lot from the type of the lighting, runs the Android system of version 4.4.4. This face dataset
the direction of lighting and shadows, the background and contains 15 classes (persons), and each person has at least 10
the emotion of the person, and so on. Figure 6 shows 20 face face images. All the face images were preprocessed using the
images of two persons with different emotions, different type method mentioned above and rescaled to the same size of
of lighting and different backgrounds. 70x70 for the computational limitation of mobile phones.
The experiment procedure is that every time randomly
select 5 face images from each person to form a training set
and the rest face images of each person are left as the test set.
This procedure will be conducted 10 times for each
algorithm’s parameter group and the accuracy on both the
training set and the test set will be evaluated as the final
result. The results of Eigenfaces and Fisherfaces are shown
in Figure 7 and Figure 8.

(a)

Figure 7. Results of Eigenfaces

(b)
Figure 6. Face images of two persons under different environments.

We have made a simple experiment using 10-fold cross

validation. Fisherfaces algorithm is adopted to test the
performance with and without the ROI preprocessing method.
Each time 5 face images of each person are randomly
selected as the training set and the rest become the test set.
The results are shown in TABLE I and show that, with ROI
preprocessing method, the performance of the face
recognition system increased significantly.

TABLE I. RECOGNITION ACCURACY WITH AND WITHOUT ROI

PREPROCESSING
Figure 8. Results of Fisherfaces
Times Without ROI With ROI preprocessing
preprocessing
1 90% 90% The results we have obtained show that, both the
2 90% 100% Eigenfaces and Fisherfaces have 100% accuracy on the
3 90% 100% training set. While on the test set, the best performance has
4 90% 100% been achieved by Fisherfaces approaching the accuracy of
5 100% 100%
96.0% when components are 14, followed by 93.8%
6 90% 90%
7 90% 100% achieved by Eigenfaces when components are 29.
8 90% 100% Besides that, when compared these two algorithms under
9 90% 100% the same components, i.e. the same dimensionality,
10 80% 80% Fisherfaces has mostly been a little better than Eigenfaces in
Average 90% 96%
this face dataset as we can see from Figure 9.

17
 ACKNOWLEDGMENTS
The second author is supported by National Natural
Science Foundation of China (No. 61261130590), 973
National Basic Research Program of China (No.
2014CB340506).

REFERENCES
[1] Android NDK, http://developer.android.com/tools/sdk/ndk/.
[2] OpenCV, http://opencv.org/.
[3] JavaCV, https://github.com/bytedeco/javacv.
[4] T. Ahonen, H. Abdenour, and M. Pietikäinen, “Face recognition with
local binary patterns,” Computer vision-eccv, 2004, pp. 469-481.
[5] D. L. Baggio, “Mastering OpenCV with practical computer vision
Figure 9. Comparition of Eigenfaces and Fisherfaces projects,” Packt Publishing Ltd, 2012.
[6] M. A. Turk., and A. P. Pentland, “Face recognition using eigenfaces,”
Proceedings CVPR'91, 1991, pp. 586-591.
V. CONCLUSIONS AND FUTURE WORK [7] R A. Fisher, “The use of multiple measurements in taxonomic
This paper presents the design and implementation of an problems,” Annals of eugenics, 7(2), 1936, pp.179-188.
open source face recognition system for Android platform, [8] G. Dave, X. Chao, K. Sriadibhatla, “Face Recognition in Mobile
which achieves a satisfactory performance on Android Phones,” Tech Report, Stanford University, USA, 2010.
mobile phone platform. After face detection and ROI [9] C. Doukas, I. Maglogiannis, “A Fast Mobile Face Recognition
System for Android OS Based on Eigenfaces Decomposition,”
preprocessing, the experiment on practical face images Artificial Intelligence Applications and Innovations, Volume
obtained the accuracy of 93.8% with Eigenfaces and 96.0% 339, 2010, pp 295-302.
with Fisherfaces. [10] G. Yang, J. Luo, “A Real-Time Face Recognition System for Android
In future work, more helpful functions like facial Smart Phone,” Information Technology Applications in Industry
expression recognition or gender recognition may be Computer Engineering & Materials Science, Vol 756-759, 2013, pp.
implemented into this system, and it is possible to integrate 4006-4010.
other face recognition methods like deep learning based face
recognition method to the XFace system.

18