APPLICATION OF SUPPORT VECTOR MACHINES CLASSIFIERS TO VISUAL SPEECH
RECOGNITION
Mihaela Gordan∗ , Constantine Kotropoulos∗∗ , Ioannis Pitas∗∗
Technical University of Cluj-Napoca
Cluj-Napoca, Romania
mihag@bel.utcluj.ro
∗∗
Department of Informatics
Aristotle University of Thessaloniki
Box 451, Thessaloniki 540 06, GREECE
{costas, pitas}@zeus.csd.auth.gr
∗
ABSTRACT
In this paper we proposed a visual speech recognition network
based on Support Vector Machines. Each word of the dictionary is
modeled by a set of temporal sequences of visemes. Each viseme
is described by a support vector machine, and the temporal character of speech is modeled by integrating the support vector machines as nodes into Viterbi decoding lattices. Experiments conducted on a small visual speech recognition task show a word
recognition rate on the level of the best rates previously reported,
even without training the state transition probabilities in the Viterbi
lattices and using very simple features. This proves the suitability
of support vector machines for visual speech recognition.
1. INTRODUCTION
The recognition of speech from the visual information only is referred as visual speech recognition or lipreading. Different shapes
of the mouth (i.e. different mouth openings, different position of
the teeth and tongue) realized during speech cause the production
of different phones. A mouth shape and mouth dynamics corresponding to the production of a phone or a group of phones indistinguishable in the visual domain defines a viseme [6]. One can
establish a correspondence between visemes and phonemes, even
if this correspondence is not one-to-one, but one-to-many, due to
the involvement of non-visible parts of the vocal tract in the speech
production. Still, for word dictionary of small size, we can perform good quality speech recognition using only a viseme-level
description of the words.
Many methods have been proposed for solving the visual speech
recognition problem in the literature. The different types of solutions adopted vary widely with respect to: the feature types; the
classifier used; the class definition. For example, Bregler uses
time-delayed neural networks (TDNN) for visual classification,
and the outer lip contour coordinates as visual features [4]. Luettin
uses active shape models for representing different mouth shapes,
gray level distribution profiles (GLDPs) around the outer and/or inner lip contours as feature vectors, and finally builds whole-word
hidden Markov models (HMM) for visual speech recognition [5].
This work was supported by the European Union Research Training
Network “Multi-modal Human-Computer Interaction (HPRN-CT-200000111).
Movellan employs also HMMs for building visual word models,
but uses directly the gray levels of mouth images as features after
some simple preprocessing to exploit the vertical symmetry of the
mouth [3].
Despite the variety of existing strategies for visual speech recognition, there is still ongoing research in this area, attempting to:
1) find the most suitable features and classification techniques to
discriminate as good as possible between different mouth shapes,
but to keep in the same class the mouth shapes corresponding
to the same phone produced by different individuals (i.e., to be
individual-independent) thus leading to higher visual speech recognition rates; 2) require as few processing of the mouth image as
possible, to allow the implementation in real time of the mouth
shape classifier considering that the end use of mouth shape classifier is in audio-visual speech recognition systems, which are supposed to work in real-time; 3) facilitate the easy integration of audio and video speech recognition.
In this paper, we aim to contribute to the first two aspects mentioned above by examining the suitability of support vector machines (SVMs) for visual speech recognition tasks motivated by
the fact that SVMs have been proved powerful classifiers in various pattern recognition applications such as face detection, face
recognition, etc., to mention a few. Very good results in audio
speech recognition using SVMs were recently reported in [1]. No
attempts in applying SVMs for visual speech recognition have
been reported so far, although a somehow closely related application is described in [2],where SVMs were applied for detecting
the degree of opening/smile of mouth images in videosequences.
This work uses SVMs for linear regression, not for classification
task. Thus, according to the best of the author’s knowledge, the
use of SVMs as visual speech classifiers is a novel idea.
One of the reasons for not using SVMs in audiovisual speech
recognition so far is the fact that they are inherently static classifiers, whilst speech is a dynamic process, where the temporal
information is essential for recognition. A solution to mitigate
this deficiency is presented in [1], where a combination of HMM
with SVM is proposed. In this paper we adopt a similar strategy
for modeling the visual speech dynamics with the difference that
we shall use only the Viterbi algorithm employed by an HMM
to create dynamically visual word models. Another novel aspect
in the visual speech recognition approach proposed here refers to
the strategy adopted for building the word models: while most of
the applications presented in the literature [1, 5, 3] build whole
word models as basic visual models, our basic visual models are
viseme-oriented models, and the visual word model is obtained by
the combination of these basic models into a temporal dynamic
sequence. This approach offers the advantage of an easier generalization to larger vocabulary word recognition tasks without significantly increasing the storage requirements by maintaining the
dictionary of basic visual models needed for word modeling into a
reasonable limit. Although using this viseme-oriented word modeling approach we could expect some performance hit in the word
recognition rate, the experimental results are on the level of the
best previous reported in literature even without learning the state
transition probabilities, which is very encouraging. In the case of
using very simple features (i.e. pixels), our word recognition rate is
superior to the ones reported in the literature. The viseme-oriented
approach can also facilitate the integration of audio and visual
speech recognition when phoneme-based audio speech recognition
is employed.
2. OVERVIEW OF SUPPORT VECTOR MACHINES
SVMs is a principled technique to train classifiers that stems from
statistical learning theory [7, 8]. Their root is the optimal hyperplane algorithm. They minimize a bound on the empirical error and the complexity of the classifier at the same time. Accordingly, they are capable of learning in sparse high-dimensional
spaces with relatively few training examples. Let {xi , yi }, i =
1, 2, . . . , N , denote N training examples where xi comprises an
M -dimensional pattern and yi is its class label. Without any loss
of generality we shall confine ourselves to the two-class pattern
recognition problem. That is, yi ∈ {−1, +1}. We agree that
yi = +1 is assigned to positive examples, whereas yi = −1 is
assigned to counterexamples.
The data to be classified by the SVM might be linearly separable in their original domain or not. If they are separable, then
a simple linear SVM can be used for their classification. However, the power of SVMs is demonstrated better in the nonseparable case, when the data cannot be separated by a hyperplane in
their original domain. In the latter case, we can project the data
into a higher dimensional Hilbert space and attempt to linearly
separate them in the higher dimensional space using kernel functions. Let Φ denote a nonlinear map Φ : RM → H where H is
a higher-dimensional Hilbert space. SVMs construct the optimal
separating hyperplane in H. Therefore, their decision boundary is
of the form:
f (x) = sign
N
X
!
αi yi K(x, xi ) + b
(1)
i=1
where K(z1 , z2 ) is a kernel function that defines the dot product between Φ(z1 ) and Φ(z2 ) in H, and αi are the nonnegative Lagrange multipliers associated with the quadratic optimization problem that aims to maximize the distance between the two
classes measured in H subject to the constraints
wT Φ(xi ) + b ≥ 1
for yi = +1
wT Φ(xi ) + b ≤ 1
for yi = −1.
Frequently used kernel functions are:
1) the polynomial kernel:
(2)
K(xi , xj ) = (mxTi xj + n)d ;
2) the Radial Basis Function (RBF) kernel:
K(xi , xj ) = exp{−γ|xi − xj |2 }.
In the following, we will omit the sign function from the decision boundary (1) that simply makes the optimal separating hyperplane an indicator function.
To enable the use of SVMs in visual speech recognition, when
we model the speech as a temporal sequence of symbols corresponding to the different phones produced, we shall employ the
SVMs as nodes in a Viterbi lattice. The nodes of such a Viterbi
lattice are supposed to generate the posterior probabilities of the
corresponding symbols to be emitted [10], and the standard SVMs
do not provide such probabilities as output. Several solutions are
proposed in the literature to map the SVM output to probabilities:
the cosine decomposition proposed by Vapnik [7], the probabilistic
approximation by applying the evidence framework to SVMs [11],
the sigmoidal approximation by Platt [12]. Here we adopt the solution proposed by Platt [12], since it is a simple solution which
was already used in a similar application of SVMs to audio speech
recognition [1]. This solution shows that having a trained SVM,
we can convert its output to probability by training the parameters
of a sigmoidal mapping function:
P (y = +1|f (x)) =
1
1 + exp(a1 f (x) + a2 )
(3)
where a1 and a2 are the parameters of the sigmoidal mapping to
be derived for the trained SVM under consideration with a1 < 0.
P (y = +1|f (x)) gives directly the posterior probability to be
used in the Viterbi decoder. The parameters a1 and a2 are derived from the training set {f (xi ), yi }, i = 1, 2, . . . , N , using
maximum likelihood estimation. The detailed description of the
training algorithm can be found in [12]. Platt shows on experimental real data that the sigmoidal model of posterior probabilities
by equation (3) approximates very well the real distribution even
for the case of non-Gaussian distributions, when the real-valued
output function of the SVM presents discontinuities around the
margins [12].
3. THE PROPOSED APPROACH TO VISUAL SPEECH
RECOGNITION
The problem of discriminating between different mouth shapes
during speech production can be viewed as a pattern recognition
problem. In this case, the set of patterns is a set of feature vectors {xi }, i = 1, 2, . . . , P , each of them describing some mouth
shape. The feature vector xi is a representation of the mouth image (either low-level, such as the gray levels from a rectangular
image region containing the mouth, geometric parameters such as
the mouth width, height, perimeter, or the coefficients of a linear
transformation of the mouth image). All the feature vectors from
the set have the same number of components, M .
Let us denote the pattern classes by Cj , j = 1, 2, . . . , Q where
Q is the total number of classes. Each pattern class Cj is a group
of patterns that represent mouth shapes corresponding to the same
viseme. The class label of the class Cj is denoted by lj .
A set of Q parallel SVMs is built where each SVM is trained
to classify test patterns in class Cj or its complement CjC (i.e., not
in class Cj ). The set of Q binary SVMs will ensure the multiclass classification of any test pattern to one of the Q classes. We
use the 1-vs-all multiclass SVM strategy [13]. To derive an unambiguous classification, we use SVMs with probabilistic outputs
[12], namely, the output of each SVM classifier SV Ml is the posterior probability for the test pattern xk to belong to the class Cl ,
l = 1, 2, . . . , Q, P (yl = 1 |fl (xk ) ).
This pattern recognition problem can be applied to visual speech
recognition in the following way: each unknown pattern represents
the image of the speaker’s face at a certain time instant; each class
label represents one viseme. Accordingly, we can compute the
probability for each viseme to be produced at any time instant in
the spoken sequence. Correlations can be established between the
different phones produced during speech and the visemes corresponding to them. The solution adopted is to define the viseme
classes and the viseme-to-phoneme mapping dependent on the application (i.e., the recognition of the first four digits in English, as
spoken by the different individuals in the Tulips1 database [3]).
The 12 viseme classes defined and their corresponding phonemes
are: • for phonemes W , U W and AO, the visemes: w, ao, wao; •
for phoneme AH, the viseme ah; • for phoneme N , the viseme n;
• for phoneme T , the viseme t; • for phoneme T H, the visemes:
th1 , th2 ; • for phoneme R, the visemes: w, ao; • for phoneme
IY , the visemes: iy, ah; • for phoneme F , the visemes: f1 , f2 ,
f3 .
By its nature, speech is a temporal process. Each spoken word
can be modelled in the visual domain as a sequence of visemes
corresponding to some basic sounds, called here visemic model.
Having defined the viseme-to-phoneme mapping for our application and having the phonetic description of each word from the
dictionary, we can build the symbolic visemic models of the words
in the dictionary. For our application, we will have a set of 28
visemic models. The most natural way of representing the word
models in the temporal domain, starting only from the symbolic
visemic model and from the total number of T frames in the word
pronunciation, is to assume that the duration of each viseme in
the word pronunciation can be whatever, but necessarily not zero.
Thus, for each symbolic visemic model, we can create a temporal network, containing as many states as many frames we have
in the videosequence, that is, T . The most straightforward way to
represent such a temporal network is the Viterbi algorithm [10].
The resulting Viterbi lattice is shown in Figure 1 for one symbolic
visemic model of the word “one”. The paths formed by the solid
lines show the possible model realizations. Each node signifies the
realization of the corresponding viseme at that particular time instant. Each visemic model of a word from the dictionary, denoted
w d , d = 1, 2, . . . , D, will have its own Viterbi lattice. Let us in-
Fig. 1. The temporal Viterbi lattice for the pronunciation of the
word “one” in a videosequence of 5 frames
terpret each node in the lattice of Figure 1 as the probability that
the corresponding symbol ok is emitted at the time instant k. We
denote this probability by bok k . Each solid line between the nodes
corresponding to the symbol ok at the time instant k and ok+1 at
the time instant k + 1 represents the transition probability from the
state that is responsible for the generation of ok to the state that
generates the symbol ok+1 . We denote the latter probability by
aok ok+1 , where ok and ok+1 may be different or not.
Having a videosequence of T frames for a word pronounced
and such a Viterbi model for each visemic word model wd , d =
1, 2, . . . , D, we can compute the probability for the visemic word
model wd to be produced following a path ℓ in the Viterbi lattice
as:
Y
k=1
Y
T −1
T
pd,ℓ =
bok k ·
aok ok+1 |d,ℓ .
(4)
k=1
and the probability for the visemic word model wd to be produced
as the maximum over all possible pd,ℓ s. Among the words that can
be produced following all the possible paths in all the D Viterbi
lattices, the most probable word, that is, the word corresponding to
the model d whose probability pd , d = 1, 2, . . . , D, is maximum
is finally recognized. In the visual speech recognition approach
discussed in this paper, the symbol emission probabilities bok k are
given by the corresponding SVMs, SV Mok . To a first approximation, we assume equal transition probabilities aok ok+1 between
whatever two symbol emission states.
The complexity of the SVM structure can be estimated by the
number of SVMs needed for the classification of each word, as a
function of the number of frames T in the current word pronunciation. Considering the total number of symbolic word models and
the number of possible states as a function of the frame index, we
get: 9 SVMs needed for the classification of the first frame, 6 for
the last and before-last frame, 11 for the second frame and all 12
SVMs for any other frame. This leads to a total of 12 × T − 16
SVMs.
4. EXPERIMENTAL RESULTS
To evaluate the recognition performances of the proposed SVMbased visual speech recognizer, we choose to solve the task of
recognizing the first four digits in English from the small audiovisual database Tulips1 [3], frequently used in similar visual speech
recognition experiments. First we define the viseme classes for
each word, based on their phonetic descriptions [14] trough the
manual annotation of the training set, then train one SVM for each
viseme considered. We used for our experiments SVMs with polynomial kernel of degree 3. We used two types of features: 1)
The first type comprises the gray levels of a rectangular region
of interest around the mouth, downsampled to the size 16 × 16
and scanned raw by raw. Each mouth image is represented by a
feature vector of length 256. 2) The second type represents each
mouth image frame at the time Tf by a vector of double size i.e.
2 × 256 = 512, that comprises the gray levels of the rectangular
region of interest as previously, and the temporal derivatives of the
gray levels normalized to the range [0, LM ax − 1] (where LM ax
is the maximum gray level value in mouth image). The temporal derivatives are simply the pixel by pixel gray level differences
between the frames Tf and Tf − 1 and are called delta features.
The complete visual speech recognizer was implemented in
C++. We used the publicly available SVMLight toolkit modules
for the training of the SVMs [9] and implemented in C++ the module for learning the sigmoidal mapping of the SVMs output to
probabilities and the module for generating the Viterbi decoding
lattices based on SVMs with probabilistic outputs.
Method
WRR [%]
Table 1. The overall WRR of the SVM dynamic network compared to other techniques.
SVM-based
SVM-based dy- AAM and HMM AAM and HMM Stochastic netdynamic net- namic network system (shape + in- system (shape + in- works, no delta
work without with delta fea- tensity model, inner tensity model, inner features [3]
delta features
tures
+ outer lip contour) + outer lip contour)
no delta features [5] + delta features [5]
76
90.6
87.5
90.6
60
We performed speaker-independent visual speech recognition
tests, using the leave-one-out testing strategy for the 12 subjects
in the Tulips1 database. More precisely, the testing strategy was
as follows: we trained the system 12 times separately, each time
using 11 subjects in the training set and leaving the 12th subject
out for testing. In this way, we obtained a total of 96 video test
sequences. We examine the overall percentaged word recognition
rate WRR, comparing this result with the ones reported in literature under similar conditions (i.e., using the same features, the
same database and the same testing procedure) [5, 3] in Table 1.
We can see that our results are on the same level as the best ones
reported in the literature (W RR = 90.6%). However the features used by us are simpler than those used in literature to obtain the same WRR. For the shape + intensity models [5] the gray
levels should be sampled in the exact subregion of the mouth image containing the lips, around the inner and outer lip contours,
and should exclude the skin areas. Accordingly, the method reported in [5] requires the tracking of the lip contour in each frame,
which increases the processing time of visual speech recognition.
Moreover we notice that our very good WRR was obtained without training the transition probabilities in the Viterbi decoding lattice from whole-word models, which could cause a performance
hit. The fact that the results are as good as the ones given by
whole word models is promising. An improvement of the WRR
is expected when training of the transition probabilities is implemented and the trained transition probabilities are incorporated in
the Viterbi decoding lattices.
5. CONCLUSIONS
We examined the suitability of SVMs with probabilistic outputs
in visual speech recognition by employing them into a dynamic
temporal network implemented by a number of Viterbi decoding
lattices as nodes and testing the proposed method on a small visual speech recognition task. Using very simple techniques, we
obtained good word recognition rates as compared to the state of
the art results from the literature. This demonstrates that SVMs
are promising classifiers for visual speech recognition tasks. Another advantage of the viseme-oriented modeling method proposed
is the possibility of easier generalization to larger vocabularies. In
our future research, we will try to improve the performance of the
visual speech recognizer by using other kernel functions, learning
the state transition probabilities of the Viterbi decoding lattices and
using better strategies for multiclass SVM implementation. Also
we intend to analyze the performance of our approach for larger
vocabulary tasks.
6. REFERENCES
[1] A. Ganapathiraju, J. Hamaker, and J. Picone. “Hybrid
SVM/HMM architectures for speech recognition,” in Proc.
Stochastic networks, + delta
features [3]
89.93
of Speech Transcription Workshop, College Park, Maryland,
USA, May 2000.
[2] V. P. Kumar and T. Poggio. “Learning-based approach to real
time tracking and analysis of faces,” in Proc. of AFGR, 2000.
[3] J. R. Movellan. “Visual speech recognition with stochastic
networks,” in Advances in Neural Information Processing
Systems, (G. Tesauro, D. Toruetzky, and T. Leen, Eds.), Vol
7, MIT Press, Cambridge, MA, 1995
[4] C. Bregler and S. Omohundro. “Nonlinear manifold learning
for visual speech recognition,” in Proc. IEEE ICCV, 1995,
pp. 494-499.
[5] J. Luettin and N. A. Thacker. “Speechreading using probabilistic models,” Computer Vision and Image Understanding,
65(2):163-178, February 1997
[6] C. Benoı̂t, T. Lallouache, T. Mohamadi, and C. Abry. “A set
of French visemes for visual speech synthesis,” in Talking
machines: Theories, Models, and Designs, (G. Bailly and C.
Benoı̂t, Eds.), 485-504, North Holland, Elvsevier, Amsterdam, 1992 J. R. Movellan.
[7] V.N. Vapnik. Statistical Learning Theory, J. Wiley, N.Y.,
1998
[8] N. Cristianini and J. Shawe-Taylor. An introduction to support vector machines, Cambridge University Press, Cambridge, U.K., 2000
[9] T. Joachims. “Making large-scale SVM learning practical,”
in Advances in Kernel Methods - Support Vector Learning,
(B. Scoelkopf, C. Burges, and A. Smola, Eds.), MIT-Press,
1999
[10] S. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and
P. Woodland. The HTK book, Entropic, Ltd., Cambridge,
UK, HTK version 2.2 edition, 1999
[11] J.T.-Y. Kwok. “Moderating the outputs of support vector machine classifiers,” in IEEE Transactions on Neural Networks,
10(5):1018-1031, September 1999
[12] J. Platt. “Probabilistic outputs for support vector machines
and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, (A. Smola, P. Bartlett,
B. Scholkopf, and D. Schuurmans, Eds.), MIT Press, Cambridge, MA, 2000
[13] D.J. Sebald and J.A. Bucklew. “Support vector machines and
the multiple hypothesis test problem,” in IEEE Transactions
on Signal Processing, 49(11):2865-2872, November 2001
[14] The Carnegie Mellon University Pronouncing Dictionary v.
0.6. http://www.speech.cs.cmu.edu/cgi-bin/cmudict