Facial Expression Recognition Using 3D Points Aware Deep Neural Network

Facial Expressions (FEs) perform a crucial role in the nonverbal communication. They afford indeed, perceptual indicators of the intention, the impression and the emotional state of persons. Managing and identifying one’s emotions and others’ is considered as a sign of high emotional intelligence.

The work of Ekman [1] is considered as one of the valuable studies on FE understanding and analysis. Ekman and Frisen [2] developed the facial action coding system to encode FEs as a combination of 44 Action Units (AUs). They also defined six universal categories of emotions, referred to as the basic emotions: Angry, Disgust, Fear, Happy, Sad, and Surprise.

A considerable amount of work on FE Recognition (FER) has been conducted on both research and industrial areas. Large and complex projects, such as mental state and behavioural studies, emotion understanding and classification, facial animation and rigging, stress signal and fatigue identification, are few examples of the widespread applications related to different FER involving different fields, such as psychology, affective computing, human-computer interaction and automatic surveillance.

FER has been extensively studied in the 2D domain (i.e. 2D images and image sequences) and various approaches have been developed [3]. Despite their valuable enhancement in terms of accuracy and expression recognition rates, 2D-based FER methods show some weaknesses when facing problems like illumination condition changes, head pose variation, and occlusion of facial appearances (e.g. hair, beard and glasses).

Therefore, 3D data have been presented as an alternative [4] to alleviate the aforementioned inherent problems with 2D data.

Due to their efficiency in analyzing the FER problem, conventional FER approaches have been prominent techniques in the last two decades. However, these handcrafted-based techniques suffer several limitations. One of the first challenging problems is how to decide which feature extraction algorithm is suitable for a given problem. There is no a priori way of knowing about the quality of the feature to be designed. The efficiency of a feature extraction method can only be established posteriori, through the statistical methods of the feature quality assessment. Moreover, most feature extraction methods induce very high feature space dimensions. Therefore, feature selection techniques such PCA and LDA are used to reduce the feature space dimension. Such techniques allow excluding highly correlated features and irrelevant ones so as to achieve higher classification accuracy and lower computational cost. Nevertheless, discarding features and reducing massive amounts of data samples can cause data loss and data restriction issues. Such operations hinder the exploitation and mining of data to the fullest extent, especially when dealing with large scale real-world data. These issues can become even more difficult to handle for challenging FER applications, such as real time 2D/3D FER or AU recognition, where the number of expressions goes beyond six basic universal emotions.

Furthermore, feature engineering methods are, in general, labor-intensive, complex and error-prone. Hence, trying to develop and compare these handcrafted features is problematic and reveals critical issues like dataset dependency, replicability and reproducibility of FER approaches.

Nowadays, Deep Learning (DL) has become the basis of most state-of-the-art techniques used in the machine learning field. Motivated by the high inference achievement demonstrated by Deep Neural Network (DNN) architectures, recent FER approaches have employed DNNs to classify FEs. Promising results in terms of FER accuracy have been achieved [5-8]. Most of the previous work on 3D FER has focused on transforming 3D faces into 2D representations in order to allow for a straightforward use of CNN-based architectures. However, such representations induce resolution-degradation and information-loss problems. Therefore, previous 3D FER approaches remain suboptimal and unable to fully describe subtle and complex details of shape cues.

The problem of 3D FER remains understudied when it comes to the use of unstructured aware DNN frameworks. Two main reasons account for this: The first is how to get the optimal 3D face representation that capture most of the facial expression cues, and the second is the lack of large facial databases.

In this work, we propose two major contributions to tackle both issues. First, along with exploiting 3D point sets extracted from curve-based representation as an input to the used DNN architecture, we propose to study the impact of the number of 3D points on the obtained accuracy in order to select the adequate one. Second, we suggest to increase the size of original data by generating realistic 3D FEs using a classic non-rigid registration technique. Thorough experiments are conducted and promising 3D FER results are obtained using the static BU-3DFE database.

The remainder of the paper is organized as follows: Section 2 reviews the related work on 3D FER. Section 3 introduces an overview of our proposed 3D FER approach. Section 4 presents the conducted experiments and obtained results. In section 5, we delve into discussion. Finally, section 6 concludes the paper and presents some future work.

We propose to address the 3D FER problem from a pure geometric view point. Among different representation types of 3D objects (i.e. point cloud, polygonal/triangular mesh, voxel grid and implicit surface), particularly for the 3D face model, we are interested in the point cloud representation.

A point cloud is a set of points $\left\{P_{i \mid i=1, \ldots, n}\right\}$ where each one is defined by its (x, y, z) coordinate attributes. Fundamentally, these spatial attributes hold fundamental facial geometric details and allow for capturing local shape information. Point clouds, known as unified structures, have the advantage of being compact and accurately encoding highly complex objects. Our goal is to develop a DL-based 3D FER approach that obviates irregularities and connectivity complexities of meshes, surfaces or volumes, and to use only the 3D point cloud instead, as presented in Figure 1. In this work, we consider treating the deep 3D FER problem as a classification one. More explicitly, we propose to apply a DNN model which accepts a set of 3D points as an input and outputs K scores for all K candidate classes, where K is the number of FEs to be recognized. These FEs are the six basic emotions. On the other hand, the main faced challenge in the 3D-point learning process is related to the fact that point clouds are unordered. Therefore, the model needs to be invariant to N! permutations.

Indeed, 3D points should create a meaningful subset allowing the model to detain combinatorial interactions among neighboring points. A symmetric function defined by ‘Maxpooling’ is consequently used.

To construct a set of symmetric functions f defining the DNN, the following observation is used: f is symmetric as long as g is symmetric.

$f\left(x_{1}, x_{2}, \ldots, x_{n}\right)=\delta \circ g\left(h\left(x_{1}\right), \ldots, h\left(x_{n}\right)\right)$

First, each 3D input point is transformed independently and identically by a small network h defined by five convolutional layers, as presented in Figure 5. Then, 3D point features are aggregated using the Maxpooling layer, which is defined by the symmetric function g. Afterwards, the aggregated information goes through a subnetwork δ, mainly composed of a set of fully connected layers.

1.png

Figure 1. Overview of proposed approach

In fact, to deal with the geometric information of 3D shapes, these shapes are typically modeled as Riemannian manifolds.

The manifolds that are equipped with a Riemannian metric are usually discretized as point clouds or meshes. Hence, the neighbourhoods of local points are defined with the distance metric and a 3D shape locally resembles to a Euclidian space. Such a standard practice allows intrinsically working with geometric shapes, i.e. to locally apply metrics and compute measurements near each point. Thus, the generalization of convolution operations, as main building blocks of classic CNN models to 3D shapes which are considered locally as Euclidean objects, becomes intuitive. The convolution filters of the new geometric CNN are deformation-invariant by construction. Pointnet ++ uses indeed the distance metric to define local regions by partitioning the set of points. Afterwards, local features are extracted from the neighbourhoods of points and then aggregated to generate higher-level features. These steps are repeated until obtaining the whole 3D facial points features.

3.1 3D FE synthesis

One major requirement for the success of DL models is the availability and abundance of labelled data to learn from. This is mainly affordable in the 2D domain especially with natural images. When acquiring such data, we only need a simple camera sensor embedded on some mobile device. Such a facility is unavailable for 3D data, especially for real data where there is a need for special 3D sensors and techniques for data acquisition. For instance, existing 3D FE databases are small-scale ones among others. For the BU-3DFE dataset, it does only contain 2500 FE labeled samples. The available datasets are still far from being enough to train DL models. Thus, there is a need to generate more ground truth data for training. One common practice to alleviate the lack of large-scale training datasets is to generate synthetic data. In order to generate 3D face expressive models, two strategies exist: 1) reconstruct 3D faces from 2D facial images, and 2) employ a statistical 3D face model and manipulate the model distribution parameters to generate synthetic 3D face samples.

The first strategy, commonly referred to as Monocular 3D Facial Shape Reconstruction (MFSR), takes advantage of the abundance of 2D face image data. Different techniques have been proposed to reconstruct 3D faces from 2D images, starting from the pioneer work of Blanz and Vetter [13], who put forward an approach to model 3D human faces from single and/or multiple facial images. This was the first work to introduce 3D Morphable Model (3DMM) for facial shape reconstruction using a PCA-based linear subspace that captures shape variations in human faces. With the emergence of DL techniques, more advanced MFSR approaches have been developed, such as RSNIEF [14], UH-E2FAR [15], Ganfit [16], MMFace [17], and AvatarMe [18].

The challenge of synthesizing 3D faces not only requires inferring accurate, smooth and fine-grained geometry information from 2D face images, but also renders high-resolution and detailed texture information (i.e. skin reflectance). Capturing this information within the constraints of arbitrary poses, lighting conditions and occlusion makes the task even more challenging. State-of-the-art MFSR approaches such as GANFIT and AvatarMe exploit the power of genarative-adversial-network models to produce high-resolution photorealistic 3D faces from in-the-wild images.

As for the second strategy, synthetic 3D FE models can be generated using 3D face morphable models, like 3DMM [13], BFM [19], and AFM [20]. Such generic 3D human-like faces are parameterized models that are controlled by a set of geometric and photo-metric parameters. Usually, they are explicitly formulated as a linear combination of; a set of shapes and blend-shape bases, which involves both identity and expression parameter vectors. Therefore, it is possible to create a large number of 3D faces. Gilani and Mian [21] proposed to synthesize new identities from 3D models by simultaneously interpolating between the facial identity and FE spaces.

Another solution consists in using 3D-face databases to compute a statistical 3DMM that likely encodes both facial identities and expressions. Samples of synthetic 3D FEs can be generated from the random sampling of the 3DMM. Even though the above methods are effective in synthesizing 3D faces, most of them fail to reconstruct highly detailed cues of geometric and texture information. The 3D face geometry reconstruction approach from 2D facial images [13] is restricted due to the various faced challenges by the used algorithm, such as illumination conditions and diversified FEs. In addition, exploiting the 3DMM [21, 22] fails to accurately depict the complicated structure of facial details [18].

According to the emotion universality hypothesis, all humans convey six prototypical expressions (anger, disgust, sadness, fear, surprise and happiness) with closely similar facial region movements.

2.png

Figure 2. Illustration of proposed DA for BU-3DFE while using eight randomly picked references to enhance DNN training of Fes

Based on this hypothesis, we propose to generate additional realistic 3D FEs by utilizing the available BU-3DFE dataset. We apply a Coherent Point Drift (CPD) non-rigid registration to reproduce the facial expression of an expressive face (target) to a neutral one (source). Compared to the state-of-the-art 3D FE synthesis methods, this technique is simple, yet effective. The size of the used database is consequently increased by reusing the same samples of the BU-3DFE database.

As presented in Figure 7, the synthetized FEs are realistic and different from the original ones.

Let S define the initial dataset and T specify the increased one. DA can be represented as:

$\phi: S \rightarrow T$

The augmented dataset is therefore represented as:

$\hat{S}=S \cup T$

where, $\hat{S}$ encloses the original training set S and the new generated data T using transformation functions $\phi$. Here, $\phi$ is the non-rigid transformation resulting from the CPD technique between two different point sets, X and Y.

The CPD algorithm defines the registration by a probability density estimation where the first point set $X=\left(x_{1}, x_{2}, \ldots, x_{N}\right)^{T}$ is the reference, and the second point set $Y=\left(y_{1}, y_{2}, \ldots, y_{M}\right)^{T}$  is the target. The latter is represented by Gaussian Mixture Model (GMM) centroids.

The GMM probability density is given by the following equation:

$P(x)=\sum_{m=1}^{M+1} P(m) p(x \mid m)$

with

$P(x=m)=\left\{\begin{aligned}

(1-\omega) \frac{1}{M} & \text { if } m \neq(M+1) \\

\omega \frac{1}{N} & \text { otherwise }

\end{aligned}\right.$

and $p(x \mid m)=\frac{1}{\left(\pi \sigma^{2}\right)^{\frac{D}{2}}} e^{-\frac{\left(x-y_{m}\right)\quad^{2}}{2 \sigma^{2}}}$.

where, $\sigma^{2}$ defines the GMM variance, (m=1, …, M)), D presents the dimension of point sets, and w defines the (M+1) uniform weight distribution comprised between 0 and 1. The extra (M+1) refers to the assignment outliers.

3.2 3D face model representation

Most 3D DL-based FER research studies have transformed 3D meshes into regular representations such as multi-view images and 2D maps in order to allow for straightforward use of CNN-based architectures. FE data need to be sampled in order to focus on relevant face parts. Therefore, we suggest a preprocessing step.

Our proposed representation consists in summarizing each 3D face shape by a set of level curves, from which we get a set of key points using a sampling technique. Figure 3 and Figure 4 illustrate the used point cloud representation resulting from a uniform curve sampling. We start first by selecting a set of 68 landmarks characterizing the informative 3D facial regions defined by the mouth, nose, eyebrows and eyes regions. Afterwards, we extract two extra points around the cheeks, which are obtained through computing the midpoints of the geodesic paths connecting both the mouth and the outside eye corners. Thus, 70 landmarks are considered as reference points, and each 3D face model is then represented by a set of level curves centered on these points.

Let S be a facial surface, and l a finite set of $n \in \mathbb{N}$  landmarks defining fixed anatomical points on S, denoted by $l=\left\{l_{1}, l_{2}, \ldots, l_{n}\right\}$. We employ the 3D Cartesian coordinates of these anatomical landmarks, represented by n ordered triplets $\left(l_{i_{x}} l_{i_{y}} l_{i_{z}}\right)_{i=\{1,2, \ldots, n\}}$, to extract patches $\left\{P_{1} \ldots P_{n}\right\}$ centered on each landmark i. Each patch $P_{i}$ is an indexed collection of level curves $\mathrm{C}_{\lambda_{1}<\lambda<\lambda_{0}}^{j}, \text { where } \lambda$ refers to a constant value of the distance function between the landmark point $l_{i}$, taken as a reference point, and all points of the curve $\mathrm{C}^{j}$. In addition, $\lambda_{0}$ defines to the maximum considered distance value. For the curve extraction, we select the Euclidean distance function $\left\|l_{i}-p\right\|$, which is sensitive to deformation. We exploit it as a function characterizing the length between a reference point $r_{i}$ and any point P on surface S, as shown in the following equation:

$\left\|r_{i}-P\right\|: C_{\lambda}^{i}=\left\{P \in S \mid\left\|r_{i}-P\right\|=\lambda\right\} \subset S, \lambda \in\left[0, \lambda_{0}\right]$

where, $\lambda_{0}$ defines the maximum value of $\lambda$ and $\mathrm{C}_{\lambda}^{i}$ refers to a closed curve comprising a group of points P placed with an identical distance $\lambda$ from the reference point $r_{i}$. Uniform sampling consists in finding a good approximation by choosing n equally spaced sample points $\left\{t_{1}, t_{2}, \ldots, t_{N}\right\} \text { defining vertices }\left\{v_{1}, v_{2}, \ldots, v_{n}\right\}$, where $v_{i}=\mathrm{C}_{\lambda}^{i}$, while keeping n small. This data representation enables accurately capturing facial surface local deformation.

3.png

Figure 3. Level curves extraction and 3D point set sampling

3.3 3D FER

3.3.1 DL on point clouds

Recent years have witnessed an increasing interest in developing deep net architectures capable of reasoning about data types, whose underlying structure lies in a non-Euclidean space. Significant effort has been directed towards extending conventional DL models to the non-Euclidean domain. A number of techniques have been proposed to generalize DNNs and adapt convolutional operations to process non-Euclidean representations of 3D objects (i.e. point cloud, mesh surface, graph). Thanks to the availability of 3D CAD models for training and the efficiency of these techniques has been proven on common 3D object applications involving classification, segmentation and recognition.

More in-depth information about the advances in 3D DL architectures and their applicability on various 3D data representations, while categorizing these representations into Euclidean and non-Euclidean ones, can be found in the interesting survey by Ahmed et al. [22]. We are particularly interested in the point cloud representation, considered as one important type of 3D geometric data offering a simple, compact and unified structure. Compared to other representation types (mesh, graph, etc.), the point representation allows avoiding complexities and combinatorial irregularities of the connectivity properties.

4.png

Figure 4. DNN architecture capturing local deformations for 3D FER

5.png

Figure 5. 3D face point cloud representation generated from different sampling of curve-based representation

Few prior approaches have studied DL on point clouds for 3D object classification and segmentation. PointNet of Qi et al. [23] was a pioneer work in this direction. It was the first DL model designed to directly process 3D point sets. Using point sets sampled from CAD models of ModelNet40, ShapeNet, and simulated kinect scans for training, evaluation and testing, the model could output, per hole and per point, labels for the inputted point set. Thus, it allowed object classification, part segmentation, and scene semantic parsing. Though simple, PointNet achieved high performance on the latter tasks, with a set of fully connected layers, max pooling, average pooling and an attention-based weighted sum. Although invariant to member permutation and rigid transformation, PointNet, by design, did not capture local features induced by point neighboring. Therefore, a second version was proposed by the same authors, PointNet++, which applied PointNet recursively and captured local features by encoding fine grained structures from small point neighborhoods. SpiderCNN [24] extended convolutional operations from regular point structures (i.e. image pixels) to irregular point sets. By parameterizing a family of convolutional filters, they introduced SpiderConvs as the new convolutional layers for point clouds.

In this work, we proceed to recursively partition the input facial 3D point sets into local overlapping regions to capture local features from neighboring points, and then group them to produce a higher-level representation. The local feature learner is defined by the original PointNet architecture.

3.3.2 DNN architecture

In this work, we exploit a 3D points aware DNN architecture where point sets are defined as vectors $\left\{P_{i \mid i=1, \ldots, n}\right\} \text { of }(x, y, z)$ coordinates. Such an architecture involves a shared MLP network with different output layer sizes (64, 128, 1024 respectively). In addition, it employs a max pooling layer followed by two connected layers with different output layer sizes: 512 and 256. A batch normalization as well as ReLU are applied to all layers. Then, we consider the registered 3D point sets as an input for our classifier. The feature learning process in this work is based on three abstraction levels. The first one, denoted by sampling layer, allows the selection of the centroids in local areas. In fact, we select a set $\left\{x_{i_{1}}, x_{i_{2}}, \ldots, x_{i_{m}}\right\}$ from the input points $\left\{x_{1}, x_{2}, \ldots, x_{m}\right\}$ using the iterative farthest point sampling [25].

The second one, called the grouping layer, retrieves the centroid neighboring points to generate several sub-point sets. The input point set size of this layer is $M \times(b+S)$, where $M, b \text { and } S$ respectively denote the number, coordinates and features of points. The output is a collection of point sets of size $M^{\prime} \times K \times\left(b+S^{\prime}\right), \quad \text { where } \quad M^{\prime}$ represents the subsampled points, K is the number of centroid neighbouring points, and $S^{\prime}$ is the new feature vector that encodes the local regions. The third one is a PointNet layer exploiting the same architecture presented by Qi et al. [23].

5.1 Classic DA

Unlike 2D massive face databases such as FaceNet [28] and VGGFace [29], 3D annotated FE databases are still limited. Consequently, the 3D data variability is insufficient, which may lead to the problem of overfitting. Most state-of-the-art 3D FER work has essentially exploited rotation as a main transformation to augment the size of the used dataset [6, 30]. We adopt a similar transformation as [6] and apply rotation on each 3D face model with different rotation angles $\left(-10^{\circ},-5^{\circ},+5^{\circ},+10^{\circ}\right)$ along three directions $(x, y, z)$ named raw, roll and pitch directions. The database size is consequently increased to reach 31200 samples ((2400 x3) x 4 + 2400 (original data)).

Table 2. Average confusion matrix (percentage values) before DA exploiting 204 points for each 3D face model

Table 3. Average confusion matrix (percentage values) with classic DA exploiting 204 points for each 3D face model

Table 4. Average confusion matrix (percentage values) before DA exploiting 3200 points for each face model

Table 5. Average confusion matrix (percentage values) with classic DA exploiting 3200 points for each face model

Table 2 and Table 4 present the resulting confusion matrices before DA using respectively 204 points and 3200 points. We notice an enhancement of accuracies values while applying classic DA (Table 3 and Table 5).

We reach an average recognition rate equal to 96.89%, while presenting each 3D face by 3200 points (Table 5).

Furthermore, we notice that by increasing the number of point sets, accuracy is significantly enhanced by 24.24% and stabilizes at 96.89%. Indeed, we notice a significant reduction of FE misclassification rates. The highest FE accuracy is mainly relative to Happy, Angry, Disgust and Sad expressions.

5.2 CPD-based DA

Applying CPD-based DA allows generating new realistic facial expressions while being different from the original ones of the BU dataset (Figure 7).

In Table 6 and Table 7, we respectively present confusion matrices using 204 points and 3200 points after our CPD-based DA.

The accuracy of 97.23% is reached, which is higher than the one obtained with classic DA (Table 7).

The misclassification rates decrease with a 6.51% in the case of Angry and Sad expressions and with 3.65% for the case of Fear and Disgust ones. The resulting misclassifications rates between the different expressions are mainly due to the FEs similarity pattern as presented in Figure 9.

Table 6. Average confusion matrix (percentage values) for our CPD-based DA with 204 points selection

Table 7. Average confusion matrix (percentage values) for our CPD-based DA with 3200 points selection

Table 8. Comparison between DL-based 3D FER works on the same BU-3DFE dataset

Comparing the two confusion matrices after DA, using 3200 points, presented in Table 5 and Table 7 for both classic and our CPD-based DA, we notice that for some cases the misclassification after CPD-based DA shows a small increase.

For instance, the misclassification between Sad and Surprise expressions augments from 0 to 0.14. Besides, Sad and Fear expressions are misclassified with 0.81%. Overall, these small variations do not influence the rise in most obtained accuracy after CPD-based DA.

Table 8 presents a comparison between different 3D FER studies using the same database (BU-3DFE). In fact, BU-3DFE is considered as the most used database in DL-based 3D FER work. This is mainly due to the database diversification in age, gender and cultural backgrounds, as well as the large FE variations related to emotional states or external environments.

Most DL-based 3D FER work has exploited 2D representations as texture, depth, curvature and normal maps, which have been generated from 3D face models.

These 2D representations facilitate the use of 2D input based DL architectures and increase the size of the used database by considering wide techniques of 2D image generation from 3D models. Despite their performance, using 2D representations for FER can degrade the useful 3D information that may be relevant for 3D FER. In fact, 2D representations are not optimal to detect complicated shape deformation. This is mainly caused by the projection of 3D data into 2D planes, which leads to the natural and significant loss of geometric information.

9.png

Figure 9. Similarity patterns between different expressions (1): Level 1 of respectively the (a) Anger and (b) Sad expressions, (2): Level 1 of respectively the (c) Happiness and (d) Anger expressions, (3): Level 1 of (e) Fear expression and level 2 of (f) disgust expression

As presented in Table 8, using 3D point sets improves the recognition accuracy up to 97.23%. In fact, focusing on the direct use of 3D point sets and exploiting the inherently homogeneous and compact representation of 3D shapes ease the learning process of shape features [34]. Being depicted by its coordinate attributes (x, y, z) facilitates indeed capturing the fine-grained local patterns in the region-based local context.

Furthermore, most literature work, as in Ref. [7, 9, 12, 31-34], has exploited only the two highest levels of the BU-3DFE database due to the fact that weak expressions (lower levels) are hardly recognized. On the other hand, in our approach we exploit the four different levels of BU-3DFE, which improves the recognition of weak expressions. This is due to the use of representations based on 3D point sets. Moreover, exploiting the DL architecture allows capturing critical and subtle FE details.