FACE DETECTION AND SMILE DETECTION

1 2
Yu-Hao Huang(黃昱豪), Chiou-Shann Fuh (傅楸善)

1
Dept. of Computer Science and Information Engineering, National Taiwan University
E-mail:r94013@csie.ntu.edu.tw
2
Dept. of Computer Science and Information Engineering, National Taiwan University
E-mail:fuh@csie.ntu.edu.tw

ABSTRACT accurate smile detection system on a common personal

computer with a common webcam.
Due to the rapid development of computer hardware
design and software technology, the user demands of 1.2. Related Work
electric products are increasing gradually. Different
from the traditional user interface, such as keyboard The problem related to smile detection is facial
and mouse, some new human computer interactive expression recognition. There are many academic
system like the multi-touch technology of Apple iPhone researches on facial expression recognition, such as [12]
and the touch screen support of Windows 7 are and [4], but there is not much research about smile
catching more and more attention. For medical detection. Sony’s smile shutter algorithm and detection
treatment, there are some eye-gaze tracking systems rate are not available. Sensing component company
developed for cerebral palsy and multiple sclerosis Omron [11] has recently released smile measurement
patients. In this paper, we propose a real-time, accurate, software. It can automatically detect and identify faces
and robust smile detection system and compare our of one or more people and assign each smile a factor
method with the smile shutter function of Sony DSC from 0% to 100%. Omron uses 3D face mapping
T300. We have better performance than Sony on slight technology and claim its detection rate is more than
smile. 90%. But it is not available and we can not test how it
performs. Therefore we would test our program with
Sony DSC T300 and show that we have a better
performance on detecting slight smile and lower false
1. INTRODUCTION alarm rate on grimace expressions.
From section 2 to section 4 we would describe our
1.1. Motivation algorithm on face detection and facial feature tracking.
In section 5, we would run experiments on FGNET face
From Year 2000, the rapid development of hardware database [3] and show results with 88.5% detection rate
technology and software environment make friendly and 12.04% false alarm rate while Sony T300 performs
and fancy user interface more and more possible. For 72.7% detection rate and 0.5% false alarm rate. Section
example, for some severely injured patients who cannot 6 will compare with Sony smile shutter on some real
type or use mouse, there are eye gaze tracking system, case video sequence.
by which the user can control the mouse by simply
looking at the word or picture shown on the monitor. In 2. FACE DETECTION
2007, Sony has released its first consumer camera
Cyber-shot DSC T200 with smile shutter function. The 2.1. Histogram Equalization
smile shutter function can detect at most three human
faces in the scene and automatically takes a photograph Histogram Equalization is a method for contrast
if smile is detected. Many users have reported that enhancement. We could always take our pictures with
Sony’s smile shutter function is not accurate as under-exposure or over-exposure due to the
expected, and we find that the Sony’s smile shutter is uncontrolled environment lightness, which would make
only capable of detecting big smile but not able to the details of the images difficult to recognize. Figure 1
detect slight smile. On the other hand, smile shutter is a gray image from Wikipedia [17] that shows a scene
would also be triggered if the user makes a grimace with pixel values very concentrated. Figure 2 is the
with teeth appearing. Therefore we propose a more result after histogram equalization.
Figure 1: Before histogram equalization [17].
Figure 3: Integral image [15].
If we have the integral image, then we can define some
rectangle features shown in Figure 4:

Figure 2: After histogram equalization [17].

2.2. AdaBoost Face Detection

To obtain real-time face detection, we use the method

proposed by Viola and Jones [15]. There are three
components inside the paper. The first is the concept of Figure 4: Rectangle feature [15].
“Integral Image”, which is a new representation of an The most commonly used features are two-rectangle
image for people to calculate the features quickly. The feature, three-rectangle feature, and four-rectangle
second is the Adaboost algorithm introduced by Freund feature. The value of two-rectangle feature is the
and Schapire [5] in 1997, which can extract the most difference of the pixels sum over gray rectangle to the
important features from the others. The last component pixels sum over the white rectangle. These two regions
is ‘cascaded’ classifiers. We can eliminate the non-face have the same size and are horizontally or vertically
regions in the first few stages. With this method, we adjacent as shown in blocks A, B. Block C is a three-
can detect faces from 320 by 240 pixel images at 60 rectangle feature whose value is also defined as the
frames per second with Intel Pentium M. 740 1.73 GHz. difference of pixels sum over the gray region to the
We will briefly describe the three major components pixels sum over the white regions. Block D is an
here. example of the four-rectangle features. Since these
features have different areas, it must be normalized
2.2.1. Integral Image after calculating the difference. After calculating the
Given an image I, we define an integral image I’(x, y) integral image in advance, it would be easy to obtain
by one rectangle region’s pixels sum by one-plus operation
and two-minus operations. For example, to calculate
I ' ( x' , y ' )   I ( x, y )
x  x ', y  y '
the sum of pixels within rectangle D in Figure 5, we
can simply compute 4 + 1 – (2 + 3) value in the integral
The value of the integral image at location (x, y) is the image.
summation over all the left and upper pixel values of
the original image I.
Figure 5: Rectangle sum [15].

2.2.2. AdaBoost
There will be a large number of rectangle features with
different sizes. For example, for a 24 by 24 pixel image,
there are 160,000 features. Adaboost is a machine-
learning algorithm used to find the T best classifiers
with minimum error. To obtain the T classifiers, we
will repeat the following algorithm for T iterations:

Figure 7: Training algorithm for building cascade

detector [15].

3. FACIAL FEATURE DETECTION AND

TRACKING

3.1. Facial feature location

Although there are many features on human face, most

of them are not very useful for facial expression
representation. To obtain the facial features we need,
we analyze these features from BIOID face database [6].
The database consists of 1521 gray-level images with
resolution 384x286 pixels. There are 23 persons in the
database and every image consists of a frontal view face
from one of them. Besides, there are 20 manually
Figure 6: Boosting algorithm [15]. marked feature points as shown in Figure 8.

After running the boosting algorithm for a goal object,

we have T weak classifiers with different weighting.
Finally we have a stronger classifier C(x).

2.2.3. Cascade Classifier

Since we have the T best object detection classifiers, we
can tune our cascade classifier with user input: the Figure 8: Face and marked facial features [6].
detection rate and the false positive rate. The algorithm Here is the list of the feature points:
is shown below: 0 = right eye pupil
1 = left eye pupil
2 = right mouth corner
3 = left mouth corner
4 = outer end of right eye brow
5 = inner end of right eye brow
6 = inner end of left eye brow
7 = outer end of left eye brow
8 = right temple Table 1 Four facial feature locations and error mean
9 = outer corner of right eye with faces normalized to 100x100 pixels.
10 = inner corner of right eye
11 = inner corner of left eye
3.2. Optical flow
12 = outer corner of left eye
13 = left temple
Optical flow is the pattern of motion of objects [18],
14 = tip of nose
which is usually used for motion detection and object
15 = right nostril
segmentation. In our research, we use optical flow to
16 = left nostril
find the displacement vector of feature points. Figure
17 = centre point on outer edge of upper lip
12 shows the corresponding feature points in two
18 = centre point on outer edge of lower lip
images. Optical flow has three basic assumptions. The
19 = tip of chin
first assumption is brightness consistency, which means
We first use Adaboost algorithm to detect the face
that the brightness of a small region remains the same.
region in the image with scale factor 1.05 to get as
The second assumption is the spatial coherence, which
precise position as possible, and then normalize the
means the neighbors of a feature point usually have
face size and calculate the feature relative positions and
similar motions as the feature. The third assumption is
their standard deviation.
temporal persistence, which means that the motion of a
feature point should change gradually over time.

Figure 9: Original image. Figure 10: Image with

face detection and features
marked. Figure 12: Feature point correspondence in two images.
We detect 1467 faces from 1521 images with detection Let I(x, y, t) be the pixel value at location (x, y) at
rate 96.45%, and we drop some false positive samples time t. From the assumptions, the pixel value would be
and finally get 1312 useful data. Figure 11 shows one I(x + u, y + v, t + 1) with displacement (u, v) at time t
result, in which the center of the feature rectangle is the + 1. Vector (u, v) is also called the optical flow of (x, y).
mean of the feature position and the width and height Then we have I(x, y, t) = I(x + u, y + v, t + 1). To find
correspond to four times x and y feature point standard the best (u, v), we select a region around the pixel (for
deviation. Then we can find initial feature position fast. example, a window of size 10 x 10 pixels) and try to
minimize the sum of the square error as below:
E (u , v )   ( I ( x  u, y  v, t  1)  I ( x, y , t )) 2
R
We use Taylor series to expand the first order
derivatives of I(x + u, y + v, t + 1) as
I(xu, y v,t 1)  I(x, y,t) Ix (x, y,t)uI y (x, y,t)v It (x, y,t)

Replace the expansion in the original equation and we

Figure 11: Face and initial feature position with blue would have E (u , v)   (I x u  I yv  It )2 .
rectangle. R
Table 1 shows the first four feature points experiment
Equation I x u  I y v  I t  0 is also called the optical
results.
(Pixel) (Pixel) (Pixel) (Pixel) flow constraint equation. To find the extreme value, the
two equations below should be satisfied.
Landmark Index X Y X St Dev Y St Dev dE
0: right eye pupil 30.70 37.98 1.64 1.95   2( I x u  I y v  I t ) I x  0
du R
1: left eye pupil 68.86 38.25 1.91 1.91
dE
2: right mouth   2( I x u  I y v  I t ) I y  0
34.70 78.29 2.49 4.10 dv
corner R

3: left mouth corner 64.68 78.38 2.99 4.15 Finally we have the linear equation:
  and low misdetection rate and high false alarm rate
2  
 I x u   I x I y  v   I x I t with small Tsmile. We run different Tsmile in FGNET
R  R  R database and results are shown in Table 2. We use 0.55
   Dstd = 3.014 pixels as our standard Tsmile to have 11.5%
2
 I x I y u   I y  v   I y I t misdetection rate and 12.04% false alarm rate.
R  R  R Misdetection False Alarm
By solving the linear equation, we can obtain optical Threshold
Rate Rate
flow vector (u, v) for (x, y). We use the concept of 0.4*Dstd 6.66% 19.73%
Lucas and Kanade [8] to iteratively solve the (u, v). It is
similar to Newton’s method. 0.5*Dstd 9.25% 14.04%
1. Choose a (u, v) arbitrarily, and shift the (x, y) to (x 0.55*Dstd 11.50% 12.04%
+ u, y + v) and calculate the relative Ix and Iy.
0.6*Dstd 13.01% 8.71%
2. Solve the new (u’, v’) and update (u, v) to (u + u’,
v + v’). 0.7*Dstd 18.82% 4.24%
3. Repeat Step 1 until (u’, v’) converges. 0.8*Dstd 25.71% 2.30%
To have fast feature point tracking, we build the
pyramid images of the current and previous frames Table 2 Misdetection rate and false alarm rate with
with four levels. At each level we search the different thresholds.
corresponding point in a window size 10 by 10 pixels
and stop the search to get into next level with accuracy 5. REAL-TIME SMILE DETECTION
of 0.01 pixels.
It is important to note that the feature tracking will
4. SMILE DETECTION SCHEME accumulate errors as time goes by and that would lead
to misdetection or false alarm results. Since we do not
We have proposed a fast and generally low want users to take an initial neutral photograph every
misdetection and low false alarm video-based method few seconds, which would be annoying and unrealistic.
of smile detector. We have 11.5% smile misdetection Moreover, it is difficult to identify the timing to refine
rate and 12.04% false alarm rate on the FGNET feature position. If the user is performing some facial
database. Our smile detect algorithm is as follows: expression when we refine the feature location, it would
1. Detect the first human face in the first image frame lead us to a wrong point to track. Here we propose a
and locate the twenty standard facial features method to automatically refine for real-time usage.
position. Section 5.1 would describe our algorithm and Section
2. In every image frame, use optical flow to track the 5.2 would show some experiments.
position of left mouth corner and right mouth
corner with accuracy of 0.01 pixels and update the 5.1. Feature Refinement
standard facial feature position by face tracking
and detection. From our very first image, we have user’s face images
3. If x direction distance between the tracked left with neutral facial expression. We would build user’s
mouth corner and right mouth corner is larger than mouth pattern grey image at that time. The mouth
the standard distance plus a threshold Tsmile, then rectangle is surrounded by four feature points: right
we claim a smile detected. mouth corner, center point of upper lip, left mouth
4. Repeat from Step 2 to Step 3. corner, center point of lower lip. Actually we would
In the smile detector application, we strongly expand the rectangle wider and higher to one standard
consider that x direction distance between the right deviation in each direction. Figure 13 shows the user’s
mouth corner and left mouth corner plays an important face and Figure 14 shows the mouth pattern image. For
role in the human smile action. We do not consider y each following image, we use normalized cross
direction displacement. Since the user can have little up correlation (NCC) block matching method to calculate
or down head rotation and that will falsely alarm our the best matching block to the pattern image around the
detector. How to decide our Tsmile threshold? As shown new mouth region and calculate their cross correlation
in Table 1, we have mean distance 29.98 pixels value. The NCC equation is:
between left mouth corner and right mouth corner and
their standard deviation value 2.49 and 2.99 pixels. Let
 ( f ( x, y)  f )( g (u, v)  g )
( x , y )R ,( u ,v )R '
Dmean be 29.98 pixels and Dstd be 2.49 + 2.99 = 5.48
C
2 2
pixels. In each frame, let Dx be x distance between two  ( f ( x, y )  f )
( x , y )R
 ( g (u , v )  g )
( u ,v )R '
mouth corners. If Dx is greater than Dmean + Tsmile, then
it is a smile, otherwise, it is not. With large Tsmile, we The equation shows the cross correlation between
have high misdetection rate and low false alarm rate, two blocks R and R’. If the correlation value is larger
than some threshold, which we would describe more
clearly later, it means the mouth state is very close to
the neutral one rather than an open mouse, a smile
mouth or other state. Then we would relocate feature
positions. To not take too much computation time on
finding match block, we set the search region center by
initial position. To overcome the non-sub pixel block
matching, we set the search range to a three by three Cross correlation value 0.925
block and find the largest correlation value as our
results.

Cross correlation value 0.767

Figure 14: Grey image

Figure 13: User face of mouth pattern [39x24
and mouth region (blue pixels].
rectangle).

5.2. Experiment
Cross correlation value 0.502
As we have mentioned, we want to know the threshold Table 3 Cross correlation value of mouth pattern for
value to do the refinement. We have a real-time case in smile activity.
Section 5.2.1 to show the correlation value changes
with smile expression and off-line case on FGNET face Cross Correlation Value

database to decide the proper threshold in Section 5.2.2.

1

0.9
5.2.1. Real-Time Case
Table 3 shows a sequence of images and their 0.8
Correlation

correlation value corresponding to the initial mouth 0.7

pattern. These images give us some level of confidence 0.6
that using correlation to identify the neutral or smile
0.5
expression is possible. To show stronger evidence, we
run a real-time case by doing seven smile activities with 0.4
1 21 41 61 81 101 121 141 161 181 201 221 241
244 frames and record their correlation value. Table 4
shows the image index and their correlation values. If Image Index

we set 0.7 as our threshold, we would have mean

Table 4 Cross correlation value of mouth pattern with
correlation value 0.868 and standard deviation 0.0563
seven smile activities.
for neutral face and mean value 0.570 and standard
deviation 0.0676 for smile face. The difference of mean
5.2.2. Face Database
value 0.298 = 0.868-0.570 is greater than two times
In Section 5.2.1 we have shown clear evidence that
sum of standard deviation 0.2478 = 2 x
neutral expression and smile expression have a great
(0.0563+0.0676). To have more persuasive evidence,
difference on correlation value. We obtain more
we run on FGNET face database in Section 5.2.2.
convincing threshold value by running cross correlation
value’s mean and standard deviation on FGNET face
database. There are eighteen people, who have three
sets of image sequences for each. Each set has 101
images or 151 images and roughly half of them are
neutral face and others are smile face. We drop some
false performing datasets. By setting threshold value
Initial mouth pattern 0.7, we have neutral face mean of mean and standard
39x25 pixels. deviation correlation value 0.956 and 0.040. At the
Initial neutral expression.
same time, smile face values are 0.558 and 0.097. It is
not surprising that smile face has higher variance then
neutral face since different user has different smile type. Total Detection Rate: Total False Alarm Rate:
We set three standard deviation distances 0.12 = 3*0.04 90.6% 10.4%
as our threshold. If correlation value is beyond the Table 5 illustrates our detection result with Sony T300
original value minus 0.12, we can refine user’s feature of Person 1 in FGNET face database. Figure 21 and
position automatically and correctly. Figure 22 show the total detection and false alarm rate
results of the fifty video sequences in FGNET. We have
6. EXPERIMENTS a normalized detection rate 88.5% and false alarm rate
12% while Sony T300 has a normalized detection rate
We test our smile detector on the happy part of FGNET 72.7% and false alarm rate 0.5%.
facial expression database [3]. There are fifty-four Sony T300 Our program
video streams coming from eighteen persons and three Image index 63 with
video sequences for each. We drop four videos which misdetection (Ground
failed to perform smile procedure due to users out of Truth: Smile, Detector:
control. Additionally, the ground truths of image are Non Smile).
labeled manually. From Figure 15 to Figure 20 are six
sequential images which show the procedure of smiling.
In each frame, there are twenty blue facial features (the Image index 63 with
fixed initial) and twenty red facial features (the correct detection (Ground
dynamically updated) and the green label lying at the Truth: Smile, Detector:
left-bottom of the image. Besides, we put the word Happy).
“Happy” at the top of the image if we have smile Image index 64 with
detected. Figure 15 and Figure 20 are correctly detected misdetection (Ground
images, while from Figure 16 to Figure 19 are false Truth: Smile, Detector:
alarm results. But the false alarm samples are somehow Non Smile).
ambiguous to different people.

Image index 64 with

correct detection (Ground
Truth: Smile, Detector:
Happy).

Figure 15: Frame 1 with Figure 16: Frame 2 with

correct detection. false alarm (Ground truth:
Non Smile, Detector:
Happy).
Image index 65 with Image index 65 with
correct detection (Ground correct detection (Ground
Truth: Smile, Detector: Truth: Smile, Detector:
Sony). Happy).

Figure 17: Frame 3 with Figure 18: Frame 4 with

false alarm (Ground truth: false alarm (Ground truth:
Non Smile, Detector: Non Smile, Detector:
Happy). Happy).
Image index 66 with Image index 66 with
correct detection (Ground correct detection (Ground
Truth: Smile, Detector: Truth: Smile, Detector:
Sony). Happy).
Total Detection Rate: Total Detection Rate:
96.7% 100%
Figure 19: Frame 5 with Figure 20: Frame 6 with Total False Alarm Rate: Total False Alarm Rate:
false alarm (Ground truth: correct smile detection. 0% 0%
Non Smile, Detector: Table 5 Detection results of Person 1 in FGNET.
Happy).
 [3] J. L. Crowley, T. Cootes, “FGNET, Face and Gesture
Compare Detection Rate
Recognition Working Group,” http://www-
prima.inrialpes.fr/FGnet/html/home.html, 2009.
1.2
[4] B. Fasel and J. Luettin, “Automatic Facial Expression
1 Analysis: A Survey,” Pattern Recognition, Vol. 36, pp.
Detection Rate

0.8 259-275, 2003.

Sony [5] Y. Freund and R. E. Schapire, “A Decision-Theoretic
0.6 Generalization of On-Line Learning and an Application to
Ours
0.4 Boosting,” Journal of Computer and System Sciences, Vol.
0.2 55, No. 1, pp. 119-139, 1997.
[6] HumanScan, “BioID-Technology Research,”
0 http://www.bioid.com/downloads/facedb/index.php, 2009.
1 5 9 13 17 21 25 29 33 37 41 45 49 [7] R. E. Kalman, “A New Approach to Linear Filtering and
Index Prediction Problems,” Transactions of the American
Society of Mechanical Engineers — Journal of Basic
Engineering, Vol. 82, pp. 35-45, 1960.
Figure 21: Comparison of detection rate. [8] B. D. Lucas and T. Kanade, “An Iterative Image
Registration Technique with an Application to Stereo
Vision,” Proceedings. of International Joint Conference
Compare False Alarm Rate on Artificial Intelligence, Vancouver, pp.674-679, 1981.
[9] S. Millborrow and F. Nicolls, “Locating Facial Features
1
with an Extended Active Shape Model,” Proceedings of
False Alarm Rate

0.8 European Conference on Computer Vision, Marseille,

0.6 France, Vol. 5305, pp. 504-513,
Sony
http://www.milbo.users.sonic.net/stasm, 2008.
0.4 Ours
[10] OpenCV, Open Computer Vision Library,
0.2 http://opencv.willowgarage.com/wiki/, 2009.
0
[11] Omron, “OKAO Vision,”
http://www.omron.com/r_d/coretech/vision/okao.html,
1 5 9 13 17 21 25 29 33 37 41 45 49
2009.
Index [12] M. Pantic, S. Member, and L. J. M. Rothkrantz,
“Automatic Analysis of Facial Expressions: The State of
Figure 22: Comparison of false alarm rate. the Art,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol. 22, pp.1424-1445, 2000.
[13] M. J. Swain and D. H. Ballard, “Color Indexing,”
7. CONCLUSION International Journal of Computer Vision, Vol. 7. No. 1,
pp. 11-32, 1991.
We have proposed a relatively simple and accurate real- [14] J. Shi and C. Tomasi, “Good Features to Track,” IEEE
time smile detection system that can easily run on a Conference on Computer Vision and Pattern Recognition,
common personal computer and a webcam. Our pp. 593-600, 1994.
[15] P. Viola and M. J. Jones, “Robust Real-Time Face
program just needs an image resolution of 320 by 240
Detection,” International Journal of Computer Vision,
pixels and minimum face size of 80 by 80 pixels. We Vol. 57, No. 2, pp. 137-154, 2004.
have an intuition that the feature around the mouth [16] P. Wanga, F. Barrettb, E. Martin, M. Milonova, R. E.
right corner and left corner would have optical flow Gur, R. C. Gur, C. Kohler, and R. Verma, “Automated
vectors pointing up and outward. The feature which has Video-Based Facial Expression Analysis of
the most significant flow vector is right on the corner. Neuropsychiatric Disorders,” Neuroscience Methods, Vol.
Meanwhile, we can support a small head rotation and 168, pp. 224-238, 2008.
user’s moving toward and backward from camera. In [17] Wikipedia, “Histogram Equalization,”
the future, we would try to update our mouth pattern http://en.wikipedia.org/wiki/Histogram_equalization,
such that we can support larger head rotation and face 2009.
[18] Wikipedia, “Optical Flow,”
size scaling.
http://en.wikipedia.org/wiki/Optic_flow, 2009.
[19] M. H. Yang, D. J. Kriegman, and N. Ahuja, “Detecting
REFERENCES Faces in Images: A Survey,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, Vol. 24, No. 1,
[1] J. Y. Bouguet, “Pyramidal Implementation of the Lucas pp. 34-58, 2002.
Kanade Feature Tracker Description of the Algorithm,” [20] C. Zhan, W. Li, F. Safaei, and P. Ogunbona, “Emotional
http://robots.stanford.edu/cs223b04/algo_tracking.pdf, States Control for On-Line Game Avatars,” Proceedings
2009. of ACM SIGCOMM Workshop on Network and System
[2] G. R. Bradski, “Computer Vision Face Tracking for Use Support for Games, Melbourne, Australia, pp. 31-36,
in a Perceptual User Interface,” Intel Technology Journal, 2007.
Vol. 2, No. 2, pp. 1-15, 1998.