Semi-supervised Facial Expressions Annotation Using Co-Training with Fast Probabilistic Tri-Class SVMs

Semi-Supervised Learning(SSL) Co-Training with Tri-Class SVMs Experimental Results Conclusion
Semi-Supervised Facial Expressions
Annotation Using Co-Training with Fast
Probabilistic Tri-Class SVMs
Mohamed Farouk Abdel Hady, Martin Schels, Friedhelm
Schwenker, Günther Palm
Institute of Neural Information Processing
University of Ulm, Germany
{mohamed.abdel-hady|friedhelm.schwenker|guenther.palm}@uni-ulm.de
September 12, 2010
1 / 15

Semi-Supervised Learning
In many domains, the amount of training examples is large
but unlabeled.
Data labeling process is often tedious, expensive and
time consuming because it requires the effort of human
experts such as physicians, radiologists, chemist, etc.
Research directions of SSL
Semi-Supervised Clustering
Semi-Supervised Classiﬁcation
Semi-Supervised Regression
Semi-Supervised Dimensionality Reduction
2 / 15

How can unlabeled data be helpful?
+
+
+
+
-
-
-
-
Figure: The unlabeled examples help to put the
decision boundary in low density regions. Using labeled
data only, the maximum margin separating hyperplane is
plotted with the versicle dashed lines. Using both
labeled and unlabeled data (dots), the maximum margin
separating hyperplane is plotted with the oblique solid
lines.
3 / 15

Co-Training with Tri-Class SVMs
ωk-v-ωh
...
...
Measure
Confidence
Select the most confident examples
{(xu
(1)
, xu
(2)
, xu
(3)
, Hkh(Xu))}
train
apply
refill
U
Lkh
U'
add
h2h1 h3
Hkh(xu)
xu
(1)
xu
(3)xu
(2)
Hkh
Xu
ω1-v-ωKω1-v-ω2
...
......
...
ωK-1-v-ωK
Figure: Tri-Class Co-Training
4 / 15

Bi-Class SVMs
2
||w||
2
b b +1b -1
large margin
small margin large margin
small margin
ωh
ωk
y=1 y=3
fkh(x) = <w,ϕ (x)>
1
2
w 2
+ C
nk +nh
i=1
i (1)
subject to the constraints
yi ( w, φ(xi ) − b) ≥ 1 − i , i ≥ 0, for i = 1, . . . , nk + nh (2)
5 / 15

Tri-Class SVMs
2
||w||
2
2
||w||
2
b1 b1 +1b1-1 b2 b2 +1b2-1
large margin
small margin large margin
small margin
ωh
ωk
y=1 y=2 y=3
ϵi
*3
ϵi
2
fkh(x) = <w,ϕ (x)>
ϵi
1
ϵi
*2
min
w,b1,b2, , ∗
ΨP =
1
2
w 2
+ C(
n1
i=1
1
i +
n2
i=1
∗2
i +
n2
i=1
2
i +
n3
i=1
∗3
i ) (3)
subject to
w, φ(x1
i ) − b1 ≤ −1 + 1
i , 1
i ≥ 0 for i = 1, . . . , n1;
w, φ(x2
i ) − b1 ≥ 1 − ∗2
i , ∗2
i ≥ 0 for i = 1, . . . , n2;
w, φ(x2
i ) − b2 ≤ −1 + 2
i , 2
i ≥ 0 for i = 1, . . . , n2;
w, φ(x3
i ) − b2 ≥ 1 − ∗3
i , ∗3
i ≥ 0 for i = 1, . . . , n3;
b1 ≤ b2
(4)
6 / 15

Illustrative example for one-v-one Tri-Class SVMs
+-
x
-
- -
-
-
-
-
-
-
-
-
-
-
-
x
x
x
x
x
xx
x
x
x
xxx
x
+
+
+
+
+
+
+
+
+
+ +
+
++
ω1
ω2
ω3
(a) input space
+-
x
-
- -
-
-
-
-
-
-
-
-
-
-
-
x
x
x
x
x
xx
x
x
x
xxx
x
+
+
+
+
+
+
+
+
+
+ +
+
++
ω1
ω2
ω3
(b) Class ω1 against ω2
+-
x
-
- -
-
-
-
-
-
-
-
-
-
-
-
x
x
x
x
x
xx
x
x
x
xxx
x
+
+
+
+
+
+
+
+
+
+ +
+
++
ω1
ω2
ω3
(c) Class ω1 against ω3
+-
x
-
- -
-
-
-
-
-
-
-
-
-
-
-
x
x
x
x
x
xx
x
x
x
xxx
x
+
+
+
+
+
+
+
+
+
+ +
+
++
ω1
ω2
ω3
(d) Class ω2 against ω3 7 / 15

Probabilistic interpretation for the Tri-Class SVM output
We ﬁt a sigmoid function on the SVM output where Eq. (6)
represents the doubt that input example x belongs to ωk or ωh.
Pkh(y = 1|x) = 1 −
1
1 + exp(−(fkh(x) − b1))
; (5)
Pkh(y = 2|x) =
1
1 + exp(−(fkh(x) − b1))
1 −
1
1 + exp(−(fkh(x) − b2))
; (6)
Pkh(y = 3|x) =
1
1 + exp(−(fkh(x) − b1))
1
1 + exp(−(fkh(x) − b2))
(7)
8 / 15

Decision Fusion for Ensemble of Probabilistic Tri-Class SVMs
Table: One-against-One Decision Profile of example x
ω1 ω2 ω3 ω4
ω1 - P12(y = 3|x) P13(y = 3|x) P14(y = 3|x)
ω2 P12(y = 1|x) - P23(y = 3|x) P24(y = 3|x)
ω3 P13(y = 1|x) P23(y = 1|x) - P34(y = 3|x)
ω4 P14(y = 1|x) P24(y = 1|x) P34(y = 1|x) -
Thus the final probabilistic output of One-against-One
ensemble of Tri-Class SVMs is defined as follows, for each
k = 1, . . . , K:
P(y = ωk |x) =
k−1
h=1 Phk (y = 1|x) + K
h=k+1 Pkh(y = 3|x)
K
k =1
k −1
h=1 Phk (y = 1|x) + K
h=k +1 Pk h(y = 3|x)
(8)
9 / 15

Facial Expressions Recognition
1 The Cohn-Kanade dataset is a collection of image sequences with emotional
content, which is available for research purposes.
2 It contains image sequences, which were recorded in a resolution of 640×480
(sometimes 490) pixels with a temporal resolution of 33 frames per second.
3 Every sequence is played by an amateur actor who is recorded from a frontal
view. The sequences always start with a neutral facial expression and end with
the full emotion.
(a) happiness (b) surprise (c) disgust (d) sadness
10 / 15

Feature Extraction
Orientation Histogram or
Optical Flow Feature
extraction Algorithm
Videoi
Videoi
Videoi
Training Videos
GMM UBM
Initial Step :
MAP Adaptation:
GMM UBM
MAP Adaptation
Orientation Histogram or
Optical Flow Feature
extraction Algorithm
Video
Input Video
μ = [μ1, ..., μM ]T
GMM Super Vector
SMO for Tri -Class SVM
EM Algorithm
Figure: Calculation of GMM Super Vectors that is
performed for each feature type
11 / 15

Methodology
1 5 times of 8-fold cross validation
2 Each test set has 44 videos (13, 11, 10 and 10 per class, respectively) while
each training set consists of 314 videos.
3 10% of the training examples of each class are used in L (9, 8, 7 and 7,
respectively), while the remaining are in U.
4 Three feature vectors (views) for Co-Training: the orientation histogram from the
mouth region (V1) and the optical ﬂow features extracted from the full facial
region (V2) and from the mouth region (V3).
5 The supervectors are normalized to have zero mean and unit variance, in order
to avoid problems with outliers.
12 / 15

86.21
84.14
83.31
93.15
73.71
77.56
75.67
84.31
84.99
75.17
78.49
89.52
75.13
81.36
76.19
87.22
87.15
78.76
81.76
91.37
75.25
70.74
69.62
81.57
77.37
78.14
74.18
86.56
58.05
67.06
65.86
73.26
70.97
64.55
66.15
79.47
61.2
70.49
70.03
77.33
75.19
67.89
70.38
82.76
64.16
58.42
59.24
70.34
81.08
81.25
81.72
89.53
61.35
68.91
70.79
73.99
78.77
70.05
73.29
83.4
78.77
73.54
72.34
78.07
81.79
73.17
75.86
85.77
65.14
61.1
62.04
71.66
55 60 65 70 75 80 85 90 95
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
SVM(V1)
SVM(V2)
SVM(V3)
mvEns
ω1-v-ω2ω1-v-ω3ω1-v-ω4ω2-v-ω3ω2-v-ω4ω3-v-ω4
test accuracy (%)
20% and Co-Training
20% only
100%
13 / 15

Conclusion
there is an improvement from using unlabeled data when
training one-against-one ensembles. Thus a learning
framework is introduced that integrates multi-view
Co-Training in the one-against-one output-space
decomposition process where Tri-Class Tri-Class SVMs
are used as binary classifiers.
The experiments have shown that Co-Training improves
facial expression recognition system using unlabeled
videos where the visual recognizers are initially trained
with a small quantity of labeled videos.
A probabilistic interpretation of Tri-Class SVM outputs is
introduced to measure confidence.
Since Tri-Class SVMs are retrained several times during
Co-Training iterations in order to benefit from the
newly-labeled videos, a modified version of SMO algorithm
is introduced for fast learning of Tri-Class SVMs because it
14 / 15

Thanks for your attention
Questions ??
15 / 15

Semi-supervised Facial Expressions Annotation Using Co-Training with Fast Probabilistic Tri-Class SVMs

Related slideshows

More Related Content

Similar to Semi-supervised Facial Expressions Annotation Using Co-Training with Fast Probabilistic Tri-Class SVMs

Similar to Semi-supervised Facial Expressions Annotation Using Co-Training with Fast Probabilistic Tri-Class SVMs (20)

Recently uploaded

Recently uploaded (20)

Semi-supervised Facial Expressions Annotation Using Co-Training with Fast Probabilistic Tri-Class SVMs