Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues

Abstract

Although state-of-the-art classifiers for facial expression recognition (FER) can achieve a high level of accuracy, they lack interpretability, an important feature for end-users. Experts typically associate spatial action units (AUs) from a codebook to facial regions for the visual interpretation of expressions. In this paper, the same expert steps are followed. A new learning strategy is proposed to explicitly incorporate AU cues into classifier training, allowing to train deep interpretable models. During training, this AU codebook is used, along with the input image expression label, and facial landmarks, to construct a AU heatmap that indicates the most discriminative image regions of interest w.r.t the facial expression. This valuable spatial cue is leveraged to train a deep interpretable classifier for FER. This is achieved by constraining the spatial layer features of a classifier to be correlated with AU heatmaps. Using a composite loss, the classifier is trained to correctly classify an image while yielding interpretable visual layer-wise attention correlated with AU maps, simulating the expert decision process. Our strategy only relies on image class expression for supervision, without additional manual annotations. Our new strategy is generic, and can be applied to any deep CNN- or transformer-based classifier without requiring any architectural change or significant additional training time. Our extensive evaluation on two public benchmarks RAF-DB, and AffectNet datasets shows that our proposed strategy can improve layer-wise interpretability without degrading classification performance. In addition, we explore a common type of interpretable classifiers that rely on class activation mapping (CAM) methods, and show that our approach can also improve CAM interpretability.

Keywords: Facial Expression Recognition, Interpretability, Action Units, Deep Models, Weakly-supervised Object Localization, Class Activation Maps (CAMs).

Figure 1: Comparison of class activation mapping (CAM) and attention maps produced at inference time using a FER classifier trained without (top) and with (bottom, ours) AU maps. In our experiments, we use an identical architecture to compare the impact of training with and without AU maps. The difference only resides in an additional AU-based training loss using AU maps. Training a FER classifier with our AU maps yields attention and CAM that are aligned with the expert’s knowledge used to assess basic facial expressions in images [51], as illustrated in the AU map ${\bm{A}}$. Consequently, our approach allows training a classifier that provides reliable interpretability, without compromising the classification accuracy. Details of our training strategy are presented in Fig.3. Note that in CAM-based models, the classification head can be fully convolutional or standard fully connected layers pooling posterior probabilities.

1 Introduction

Facial expression recognition (FER) has recently received much interest in the computer vision and machine learning communities [8, 82, 92]. Since facial expression is one of the most important ways for people to express emotions [16], FER finds a wide range of applications in, e.g., medical analysis and monitoring [1, 35, 84], e-health [68], driver fatigue detection [2, 46], safe driving [33], security [41, 58], lecturing [69], and many other fields [63].

FER remains challenging, particularly for real-world applications. This is due to the subtle differences between expressions, leading to low inter-class variability. Learning to distinguish samples from overlapping classes can be difficult. Additionally, people express the same facial expression differently depending on attributes such as level of expressiveness, age, gender, and ethnic background [38, 71]. To tackle these challenges, many deep learning methods have been proposed to train highly accurate classification models [8, 38, 82, 92]. This is achieved by learning robust feature representations compared to traditional hand-crafted features [15, 47, 67]. Deep FER methods usually rely on either global or local approaches to learn feature representations. Global approaches [24, 39] propose training loss functions to improve the overall representation robustness. To avoid incorporating noise into the global image representation, local approaches resort to learning from parts of the image [8, 30, 40, 74, 79, 80, 82, 92]. These methods use facial landmarks to extract robust local features around relevant facial parts [30, 79, 92]. Other methods rely on self-attention mechanisms to focus on relevant and discriminative parts of the facial image and suppress noisy regions [8, 40, 74, 80, 82].

State-of-the-art FER classifiers have achieved significant progress in terms of accuracy. However, end-users may not require only an accurate classifier that yields a classification score in multiple critical applications [17, 61, 94]. They may also need the model to provide an interpretable decision [17, 83, 94]. For instance, this can help clinicians and therapists understand and build trust in the FER model decisions, planning better future interventions [72]. Subsequently, this facilitates better integration of machine learning methods into daily clinical routines and health care practice [72]. In addition, interpretability can greatly help diagnose errors made by machine learning models and facilitate the identification of weaknesses for future improvement. Unfortunately, interpretability has been overlooked in the FER due to the main focus on classification accuracy. This has led to the design of FER models that lack interpretability. Recent progress with attention-based methods like APVIT [82] allows extracting internal attention explicitly to select only discriminative regions of interest (ROIs). Although they provide visual interpretability to highlight the regions used for classification, such ROIs are not necessarily aligned with expert knowledge typically used to assess facial expressions.

Experts commonly rely on a codebook of AUs to determine a basic facial expression [51] (Fig. 2). Each expression is associated with a subset of spatial AUs in the face. Ideally, an interpretable classifier should point to and localize the correct AU ROIs when predicting the corresponding expression. However, this task is challenging since predicting such ROIs using only image expression supervision is not trivial. Additionally, the localization cues are not publicly available to be learned due to the annotation costs and the complexity of building large-scale benchmarks.

Figure 2: Codebook of basic facial expressions and their associated AUs [51]. The spatial AU map is built using the image expression to select the right corresponding AU subset in combination with facial landmarks, which are employed to localize these same AUs in the image. In particular, the location of landmarks is used to estimate AU positions. For instance, the right ’Cheek’ location is estimated using landmark 47 (middle of the low right eye) and 11 (right side of the jaw). The code ’AUx’ is the identifier of the AU [51].

This paper introduces a generic learning strategy to build an interpretable deep classifier for the FER task. This is achieved using spatial AU cues constructed from the image class supervision without extra manual annotation cost. This explicitly integrates the expert facial expression assessment approach into a classifier’s decision-making. In particular, we build a visual interpretability tool consistent with AU cues used by experts [51]. To this end, a spatial discriminative heatmap is constructed that gathers relevant locations of AUs required to determine the expression in an input image. Then, layer-wise deep features of a classifier are constrained to be correlated with such heatmap cues during training. At the same time, the model is trained to classify the image correctly. Such a multi-task training scheme allows for building accurate but, most importantly, interpretable classifiers. During inference, the layer-wise attention provides a visual interpretability tool that indicates which ROIs are used to determine the predicted expression (Fig. 3, 1). This is achieved without additional annotation costs, as illustrated in Fig. 3. A discriminative AU heatmap can be built simply using image class expression, an AU codebook, and facial landmarks, which are commonly estimated using off-the-shelf methods. Interpretability in models usually emerges as a result of other tasks, such as classification without explicit supervision for interpretability [19]. It represents a tool to clarify the model decision, for instance, under the form of visual ROIs. However, since we explicitly provide interpretability cues to train our model, we refer to it as guided interpretable FER model.

Following the interpretable classifier direction, we explore a curated branch of classifiers in this work that allows us to perform image classification and yield visual interpretability. In particular, we employ CAM-based classifiers [12, 62]. These methods are popular for weakly-supervised object localization (WSOL) tasks in computer vision on different imaging types, including natural scene [93] and medical imaging [62, 73]. Given an input image, a classifier can be trained using only image-class labels to correctly classify an image and provide a per-class heatmap, i.e., CAM, to localize image ROIs related to the output prediction. Therefore, they play an important role in localization and visual interpretability, making them well-suited for our work. Consequently, we conduct a comparative study using different CAM methods from other families of methods, and assess their capacity for classification and interpretability on the FER task. Note that this is the first work to leverage CAM methods for the FER task.

Our main contributions are summarized as follows:

(1) To improve the visual interpretability of state-of-the-art FER classifiers, we introduce a learning strategy allowing to training of an accurate but, most importantly, interpretable deep classifier (Fig.1) that is consistent with the process used by experts to determine basic facial expressions. Our training relies on aligning spatial features with spatial AU maps built using facial landmarks and image-class labels. This guides the classifier’s attention to use regions around AUs leading to more interpretable decisions.

(2) Our method does not require extra manual annotation, significant extra computations during training, nor change the model architecture or the inference process. In addition, our method is generic – it can be used with any deep CNN or transformer-based model.

(3) Our approach is validated experimentally on two public FER benchmarks (RAF-DB, and AffectNet) in terms of classification and interpretability accuracy. We evaluated different CAM-based methods with and without our spatial AU cues. Different ablations are provided as well. Empirical results showed that both classification and interpretability improved. Interestingly, compared to a vanilla deep classifier, we show that its layer-wise attention can largely be shifted using our AU spatial cues without compromising classification accuracy. We show that classification accuracy improves, particularly over large datasets such as AffectNet. This demonstrates that spatial AUs are a reliable source for discriminative ROIs for basic facial expression recognition [51].

2 Related Work

This review covers related work on FER classifier systems, CAM methods for localization and interpretability, FER systems interpretability, and AUs in FER tasks.

2.1 Standard FER classifiers

Given the availability of large public benchmarks [39, 53], different methods have been proposed to achieve state-of-the-art classification accuracy [38]. This is most achieved by designing robust features to overcome the limitations of traditional hand-crafted features [15, 47, 67]. Robust features are also learned using deep models, either through a global or local approach.

Global methods typically focus on designing robust training losses to build enhanced discriminative features while using the entire image to build a global representation [24, 39]. For instance, Farzaneh and Qi [24] leverage deep metric learning and propose a deep attentive center loss to select a subset of relevant features for classification adaptively. Although successful, image faces are typically noisy, which may easily corrupt the features. Local methods tackle this issue by explicitly incorporating different mechanisms to remove the noise and build better features. Mainly, these methods rely on a part-based approach assuming that only some regions in the image are relevant to determine the expression [8, 30, 40, 74, 79, 80, 81, 82, 92].

A subset of these methods assumes that discriminative features are located around facial landmarks. Therefore, only these regions are cropped either at image or spatial feature level [30, 79, 92]. This assumes that relevant regions are located around the landmarks. Such early dropout of patches may discard relevant patches not included in the landmarks in addition to missing the context. This also requires highly accurate landmarks. However, leveraging landmarks has been successful in the recent work POSTER [50, 92] by cross-fusion of sparse key-landmarks features with standard global image features.

A second subset exploits attention mechanism [8, 80, 74, 81, 82] either self-learned or with provided cue. DAM-CNN [80] learns a spatial feature attention to filter out noise and keep only relevant spatial features to build a dense embedding later. The APVIT method [81, 82] leverages transformers and their attention potential to design a FER classifier. In particular, they propose a self-attention approach early in the network. Such attention allows the selection of relevant patch regions and performs hard attention at the spatial feature level. Patches are scored, and only the top-k are allowed to proceed into the next layer to build an image embedding. This explicitly allows the model to learn to filter out irrelevant patches. Other methods are provided external spatial cues to be aware where to look for discriminative regions [8, 40, 74]. RAN [74] is a region-based method. It performs image cropping either randomly or using landmarks. Their features are then attended using self-attention and relation-attention modules. Such attention is then used to combine different features and build an image embedding. This makes the final representation robust to pose and occlusion. Li et al. [40] follow a similar direction by decomposing the spatial features into the regions and using the attention module afterward. Bonnard et al. [8] use facial landmarks to build a heatmap, which is used to align spatial features in a deep model. This is expected to filter out irrelevant spatial features and focus mostly on features around landmarks. Our work is more aligned with this family of methods. However, we use grounded and more reliable cues, that is, spatial AUs. Since spatial AUs are class-dependent, they are more discriminative for the FER task [51], compared to facial landmarks, which are generic and class-agnostic.

While the abovementioned methods focus on FER over still images, other methods tackle video applications [42, 43, 44, 45]. These methods deal with similar issues presented in still images while leveraging temporal dependency and multi-modality in videos to capture better expressiveness and build robust features.

2.2 CAM methods for WSOL and interpretability

Using only image class as supervision, CAM-based methods can be trained to classify an image correctly and yield reliable localization of ROIs related to the image class. They are currently the dominant approach for the WSOL task [12, 62]. They achieved interesting results in different domains, including natural scene images [7, 62, 77], and medical imaging [5, 6, 62, 73]. They have also been extended for localization in videos [3, 4]. Early works have focused on building different spatial pooling layers [20, 21, 59, 93]. However, CAMs tend to highlight only small and most discriminative parts of an object [12, 62], limiting its localization performance. To overcome this, different strategies have been proposed, including data augmentation on input image or deep features [6, 13, 49], as well as architectural changes [29, 36, 77, 90].

Recently, learning via pseudo-labels has shown great potential, despite its reliance on noisy labels [7, 5, 52, 56, 54, 55, 75]. Most previous methods used only forward information in a CNN (bottom-up family [62]). However, other methods also aim to leverage backward information (top-down methods [62]). These include biologically inspired methods [9, 87], and gradient [10, 28, 34, 66] or confidence score aggregation methods [18, 57]. CAM-based methods have been used for interpretability as well [64, 78] since they provide a map that indicates ROIs relevant to the model’s decision. Gradient-based CAMs are most common in interpretability task [10, 28, 34, 66] in addition to [9, 87]. They mainly search for ROIs in the feature maps that better stimulate a class response. This inspires other methods [88] to leverage gradient and extend it to input image and perform perturbation analysis [14, 25, 26, 60, 86] where the aim is to find which ROI of the network’s input that better stimulates its output. Compared to CAM-based, these methods are often more expensive in computation and require optimization, making them less attractive for FER tasks. However, CAM-based methods are simple to use, and they are built into the model, requiring a single forward or a forward and backward computation.

2.3 Interpretability in FER systems

In critical FER applications such as e-health and behavioral health interventions and assessments [11], it may not be enough to predict the facial expression accurately. Users may require interpretability to help understand the model decision [17, 61, 83, 94]. Although important, interpretability in FER systems has been largely overlooked. This is mainly due to the focus on achieving state-of-the-art classification accuracy over challenging benchmarks [38]. The absence of public datasets with interpretability annotation has also contributed to shadowing such an important task. Some recent works have attempted to provide built-in discriminative attention under visual interpretability, such as the APVIT method [81, 82]. Due to the lack of annotation, these methods are still limited and unreliable since their interpretability has not yet been quantified. In this work, we propose a protocol to evaluate the interpretability of a FER system without the need for extra manual annotation. We use spatial facial AUs to simulate experts’ processes to assess expressions [51]. Such AUs are built and encoded automatically in a convenient spatial heatmap using only image expression as supervision. This can help to automatically label large benchmarks easily, allowing training and evaluation of FER models in terms of interpretability.

2.4 AUs in FER systems

The Facial Action Coding System (FACS) is a taxonomy for fine-grained facial expression analysis [22, 27]. They have long been used to analyze facial expressions [23, 32, 48, 65, 91]. Each basic facial expression is associated with a list of AUs as they determine which facial muscles are involved in expressing such emotion. The standard established task in the literature is AU detection [32, 48]. It is a supervised multi-label classification task. It aims at predicting the correct set of active AUs in an input image. To perform such a task, a tedious annotation is required to determine the active set of AUs in each image since not all AUs must be active at once. Another related task goes further to estimate the intensity of the AUs [23, 91], which is furthermore challenging. Other works aim to localize and estimate the intensity [65] jointly. The work of [32] is relatively close to our work. To accurately detect active AUs through multi-label classification, authors leverage multi-task learning to predict a localization heatmap for the same AUs jointly. The goal is to improve the detection of AUs via their spatial localization. However, AUs have not been used for the interpretability of FER classifiers. This makes our work the first to do so. To avoid the extra annotation costs, we used a generic discriminative AUs codebook defined in [51]. It allows the automatic labeling of extensive benchmarks without manual intervention (Fig. 3). The codebook associates a set of AUs to a basic facial expression (Fig. 2). This is used to build a discriminative heatmap that holds potential ROIs to be inspected by the model to determine the facial expression in the input image. This does not require all the AUs to be active. But, it is more likely that part of them are active for the expression to manifest in an image. A further step of our work is to use active AUs to build an ideal interpretable model, although this comes at an additional expensive annotation cost. Alternatively, a pretrained AU detector, such as [32], could be used.

Figure 3: Our interpretable classifier for the FER task (training and inference). Each basic facial expression can be determined via a set of AUs [51]. Therefore, to train our interpretable FER classifier, we first extract facial landmarks and build a discriminative spatial map ${\bm{A}}$ that contains the set of all AUs associated with the image class expression [51]. This map is used as localization cues to train layer-wise attention ${\bm{T}_{l}}$ to focus on the ROIs highlighted in the AU map. A classification loss, such as cross-entropy, is also used. Once trained, the classifier yields an interpretable layer-wise attention map. When a CAM method [62] is considered, the classifier can also produce a per-class interpretable map.

3 Proposed Approach

3.1 Notation

We denote by ${\mathbb{D}=\{(\bm{X},y)_{i}\}_{i=1}^{N}}$ a training set of $N$ images, where ${\bm{X}_{i}}$, of size ${h\times w}$, is the input image, and ${y_{i}\in\{1,\cdots,K\}}$ is its image class, i.e. facial expression, with $K$ is the total number of classes. We denote by ${f(\bm{X};\bm{\theta})}$ a deep classifier with ${\bm{\theta}}$ its parameters. The posterior per-class probabilities are referred to as ${f(\bm{X})\in[0,1]^{K}}$ where ${f(\bm{X})_{k}}$ is the probability of the class ${k}$. The spatial feature maps produced by the classifier’s encoder at a layer ${l}$ are a tensor denoted by ${\bm{F}_{l}\in\mathbb{R}^{n\times h^{\prime}\times w^{\prime}}}$, where $n$ is the number of feature maps, and ${h^{\prime}\times w^{\prime}}$ is their size which is typically smaller than the input image. CAM-based WSOL methods produce an additional spatial tensor that holds all the CAMs, ${\bm{M}\in\mathbb{R}^{K\times h^{\prime\prime}\times w^{\prime\prime}}}$, where ${\bm{M}[k]}$ is the CAM of the class ${k}$.

3.2 Building spatial AU maps

Given an input image and its label ${(\bm{X},y)}$, standard 68 facial landmarks are extracted from the image using off-the-shelf techniques such as SynergyNet [76]. This allows us to localize relevant parts of the face such as mouth, nose, eyes, and eyebrows, which will be helpful later to estimate the location of relevant AUs. Martinez et al. [51] suggest that each facial expression can be recognized by inspecting a combination of AUs (Fig. 2). For instance, the expression ’Happiness’ involves a set of AUs: ’cheek raise’, ’lip corner puller’, ’lips part’. Using the extracted facial landmarks around the mouth part, we can localize the ’lip part’ of the face. Such localization is translated into a 2D heatmap where strong activations are concentrated around the ’lip’ to indicate an ROI and tiny activations everywhere else to indicate the background. Such a 2D map is of great importance in training since it provides useful localization cues for relevant discriminative parts in an input image for the FER task used by experts. Typically, this requires manual annotation to delineate ROIs, increasing the annotation cost. Unfortunately, such valuable annotation is not available in public facial expression benchmarks. This work simulates such ROIs without an additional manual supervision cost. Using the codebook of facial expressions and their associated AUs provided in [51] (Fig. 2), in combination with the extracted facial landmarks and the image label expression ${y}$, a single 2D heatmap, ${\bm{A}\in\mathbb{R}^{h\times w}}$, is built to hold all the relevant (i.e., discriminative) AUs for the input image (Fig. 3). Consequently, the input image is augmented with an extra supervision cue to be ${(\bm{X},y,\bm{A})}$ where ${\bm{A}}$ is the estimated AU map. To ease subsequent alignment computations, this map is normalized to have values between 0 and 1: ${\bm{A}\in[0,1]^{h\times w}}$. Note that this process requires only the image class, i.e., image facial expression, as supervision. No extra manual annotations are needed.

3.3 Layer-wise attention

Given a layer-wise spatial feature tensor ${\bm{F}_{l}}$ at layer $l$, we aim to determine which spatial parts are relevant for classification. A common approach to achieve this is through features self-attention [13, 96]. Such spatial attention is estimated via the average feature map as follows,

$$\bm{T}_{l}=\frac{1}{n}\sum_{j=0}^{n}\bm{F}_{l}[j]\;,$$ (1)

where $n$ is the total number of feature maps. The discrepancy in activations in ${\bm{T}_{l}}$ differentiates relevant from irrelevant regions. High activations indicate potential spatial ROIs for classification, while low activations point to background and noisy regions. In a deep classifier, such attention maps are a great candidate for incorporating spatial learning cues into the model. Although CAM tensors ${\bm{M}}$ hold spatial discriminative cues, they are sensitive to being altered. Aligning them explicitly with other maps can easily lead to poor classification [7, 12]. Therefore, we rely on layer-wise attention to learn better and interpretable spatial features.

3.4 AU map alignment loss

After estimating both layer-wise attention ${\bm{T}_{l}}$ and AU map ${\bm{A}}$, we can proceed with their alignment to train the classifier to yield similar spatial cues. Since the dynamic range of the attention map is unknown beforehand, and since it changes during training, it is inadequate to train the attention to have the same values as the AU map ${\bm{A}}$. Instead, we resort to a loss that aims to yield attention maps that are correlated with ${\bm{A}}$. To this end, we use cosine similarity as,

$$\mathcal{R}(\bm{T}_{l},\bm{A})=\frac{\sum\left(\bm{T}_{l}\odot\bm{A}\right)}{\lVert\bm{T}_{l}\rVert_{2}\lVert\bm{A}\rVert_{2}}\;,$$ (2)

where ${\cdot\odot\cdot}$ is the Hadamard product, and ${\lVert\cdot\rVert_{2}}$ is the $\ell^{2}$ norm. Maximizing Eq. (2) pushes the layer $l$ to learn spatial features in a way their mean is highly correlated with the AU map cue ${\bm{A}}$. In practice, the map ${\bm{A}}$ is downsampled to the same size as ${\bm{T}_{l}}$ before applying the alignment in Eq. (2).

3.5 Total training loss

Given the triplet ${(\bm{X},y,\bm{A})}$, our goal is to train the classifier ${f}$ on input image ${\bm{X}}$ to yield the correct class ${y}$. Additionally, we aim to encourage layer ${l}$ to construct spatial features that are correlated with the AU map cues ${\bm{A}}$. To achieve this, we jointly minimize the following composite loss,

$$\min_{\bm{\theta}}\quad-\log(f(\bm{X};\bm{\theta})_{y})+\lambda(1-\mathcal{R}(\bm{T}_{l},\bm{A}))\;,$$ (3)

where ${\lambda\geq 0}$ is a weighting coefficient that balances the importance of attention alignment with the AU map compared to cross-entropy loss (left side term). The generalization of Eq. (3) to a combination of multiple layers is straightforward. It can be achieved by adding more layer-wise terms to the loss. Minimizing Eq. (3) incorporates explicitly the experts’ procedure in assessing basic facial expressions in images into the model decision process. The classifier is trained to localize the relevant AUs to build discriminative features for expression prediction. This justifies using layer-wise attention ${\bm{T}_{l}}$ as an interpretability tool during inference time (Fig. 3).

We note that the computational cost added by the alignment term is negligible since it can be easily computed on GPU. This leads to a training time similar to the vanilla case, where no alignment is used. The only required pre-processing is facial landmark extraction, which can be done offline once and stored on disk. AU map ${\bm{A}}$ can be computed in a negligible time on CPU on the fly during training using the image-class label only as supervision. Since image-class labels are unavailable at test time, AU maps can not be built using the lookup table emotion-AUs. Hence, training the model to produce them along with the true label is more realistic and practical. Furthermore, it allows the model itself to build them, leading to more interpretable predictions.

		RAF-DB					AffectNet
		CL		CAM-COS			CL		CAM-COS
Method		w/o AU	w/ AU	w/o AU	w/ AU		w/o AU	w/ AU	w/o AU	w/ AU
CNN-based
CAM [93] (cvpr,2016)		$88.20$	$\bm{88.95}$	$0.55$	$\bm{0.70}$		$60.88$	$\bm{62.37}$	$0.56$	$\bm{0.69}$
WILDCAT [20] (cvpr,2017)		$88.26$	$\bm{88.85}$	$0.52$	$\bm{0.69}$		$59.88$	$\bm{61.62}$	$0.62$	$\bm{0.80}$
GradCAM [66] (iccv,2017)		$88.39$	$\bm{88.85}$	$0.55$	$\bm{0.74}$		$60.77$	$\bm{62.08}$	$0.53$	$\bm{0.75}$
GradCAM++ [10] (wacv,2018)		$87.84$	$\bm{89.14}$	$0.60$	$\bm{0.82}$		$60.22$	$\bm{62.45}$	$0.66$	$\bm{0.83}$
ACoL [89] (cvpr,2018)		$87.94$	$\bm{88.68}$	$0.54$	$\bm{0.67}$		$58.28$	$\bm{61.48}$	$0.55$	$\bm{0.65}$
PRM [95] (cvpr,2018)		$88.13$	$\bm{88.88}$	$0.48$	$\bm{0.59}$		$57.77$	$\bm{60.97}$	$0.52$	$\bm{0.75}$
ADL [13] (cvpr,2019)		$87.45$	$\bm{88.65}$	$0.50$	$\bm{0.63}$		$57.88$	$\bm{61.25}$	$0.54$	$\bm{0.66}$
CutMix [85] (eccv,2019)		$88.39$	$\bm{88.59}$	$0.55$	$\bm{0.57}$		$58.74$	$\bm{59.88}$	$0.56$	$\bm{0.58}$
LayerCAM [34] (ieee,2021)		$87.90$	$\bm{88.88}$	$0.60$	$\bm{0.84}$		$60.77$	$\bm{62.45}$	$0.66$	$\bm{0.83}$
Transformer-based
TS-CAM [29] (iccv,2021)		$86.70$	$\bm{88.00}$	$0.58$	$\bm{0.71}$		$58.99$	$\bm{59.54}$	$0.57$	$\bm{0.58}$
APViT [82] (ieee,2022)		$91.00$	$\bm{91.03}$	$--$	$--$		$60.62$	$\bm{62.28}$	$--$	$--$

Table 1: Classification (CL) and CAM-localization (CAM-COS) performance on RAF-DB and AffectNet test sets with and without AUs across methods.

		RAF-DB			AffectNet
Methods / Case		w/o AU	w/ AU		w/o AU	w/ AU
CNN-based
CAM [93] (cvpr,2016)		$0.57$	$\bm{0.85}$		$0.64$	$\bm{0.82}$
WILDCAT [20] (cvpr,2017)		$0.47$	$\bm{0.85}$		$0.61$	$\bm{0.81}$
GradCAM [66] (iccv,2017)		$0.63$	$\bm{0.85}$		$0.65$	$\bm{0.82}$
GradCAM++ [10] (wacv,2018)		$0.52$	$\bm{0.87}$		$0.65$	$\bm{0.82}$
ACoL [89] (cvpr,2018)		$0.46$	$\bm{0.84}$		$0.60$	$\bm{0.81}$
PRM [95] (cvpr,2018)		$0.43$	$\bm{0.85}$		$0.55$	$\bm{0.82}$
ADL [13] (cvpr,2019)		$0.51$	$\bm{0.85}$		$0.65$	$\bm{0.83}$
CutMix [85] (eccv,2019)		$0.51$	$\bm{0.80}$		$0.57$	$\bm{0.82}$
LayerCAM [34] (ieee,2021)		$0.52$	$\bm{0.86}$		$0.65$	$\bm{0.82}$
Transformer-based
TS-CAM [29] (iccv,2021)		$0.55$	$\bm{0.88}$		$0.48$	$\bm{0.79}$
APViT [82] (ieee,2022)		$0.38$	$\bm{0.85}$		$0.45$	$\bm{0.84}$

Table 2: Attention-localization (ATT-COS) (at layer 5) performance over RAF-DB and AffectNet test sets with and without AUs.

Methods				CL	CAM-COS
CAM [93] w/o AU				$88.20$	$0.55$
CAM [93] w/ AU at layers:
1				$88.52$	$0.58$
2				$88.26$	$0.57$
3				$88.39$	$0.56$
4				$88.62$	$0.61$
5				$\bm{88.95}$	$\bm{0.70}$
5+4				$88.78$	$0.67$
5+4+3				$88.78$	$0.67$
5+4+3+2				$88.55$	$0.67$
5+4+3+2+1				$88.46$	$0.67$

Table 3: Ablation study over RAF-DB test set: impact of applying AU alignment over different layers over classification (CL) and CAM-localization (CAM-COS). Method: CAM [93] and ResNet50 [31] encoder. Layer 1 is the input layer.

4 Results and Discussion

4.1 Experimental Methodology

Datasets. To evaluate our method, experiments are conducted on two standard datasets for the FER task: RAF-DB [39], and AffectNet [53].

- RAF-DB Real-world Affective Faces Database is a large scale facial expression dataset [39]. Multiple annotators have manually annotated all images. It contains six basic expressions (’Happiness’, ’Sadness’, ’Surprise’, ’Anger’, ’Disgust’, ’Fear’) and ’Neutral’. The dataset contains 12,271 samples for training and 3,068 samples for testing. Both sets have the same distribution.

- AffectNet is one of the largest facial expression recognition datasets with 420k images were manually annotated. Following [37], 283,901 images are used for training and 3,500 for testing. In terms of labels, this dataset has the same six basic expressions as in the RAF-DB dataset, in addition to ’Neutral’.

Implementation Details. For all our experiments, we trained each method for 100 epochs over RAF-DB with a mini-batch size 32 and 20 epochs for AffectNet, with a mini-batch size 224. For CNN-based methods, we used ResNet50 [31] as an encoder. For transformer-based methods, we used Deit-S [70] for TS-CAM [29]. We used their proposed transformer architecture for the APVIT method [82], which also relies on Deit-S [70]. Images are aligned and resized to ${256\times 256}$, and randomly cropped patches of size ${224\times 224}$ are extracted for training. For APVIT [82], images are resized to ${112\times 112}$ and fed as input following the method’s protocol. The hyper-parameter ${\lambda>0}$ is estimated using validation by covering values that range from less than 1 to 20. The optimization of training loss is performed using stochastic gradient descent (SGD). The hyper-parameters search of different methods is done over the RAF-DB dataset. Due to its large size, relevant hyper-parameters are transferred from RAF-DB to AffectNet dataset.

Baseline Methods. To assess the benefits of using AU cues, we leverage WSOL classifiers. They allow the classification of an image using only the image class as a label. In addition, they provide interpretability maps. In particular, we use CAM-based WSOL methods, which provide interpretability by highlighting ROIs associated with a class. To this end, we selected methods that cover both families of CAM techniques [62]: 1- Bottom-up methods where information flows from the input layer to the output layer. This includes CAM [93], WILDCAT [20], ACoL [89], PRM [95], ADL [13], and TS-CAM [29]. 2- Top-down methods where information flows in both directions. This includes gradient-based methods: GradCAM [66], GradCAM++ [10], and LayerCAM [34]. All these methods are common in the WSOL field. In the FER task, we selected a recent state-of-the-art method, APVIT [82], which builds spatial attention in its architecture to yield interpretable maps. However, it does not have a CAM module. All the methods are trained with and without our AU loss in addition to classification loss to assess the impact of using AUs as spatial cues for learning. We note that all results are reported using our implementations.

To build AU map cues, we first extract standard 68 facial landmarks from images using SynergyNet [76]. Following [51], each facial expression can be recognized using only a set of generic AUs. Using the extracted facial landmarks, one can build a 2D map that contains a heatmap of a single AU. For an image, a single 2D heatmap is then built containing all the AUs associated with the image expression. This heatmap encodes the location of the necessary discriminative spatial ROIs needed to recognize the facial expression presented in the image. Training a classifier to focus mainly on these spatial ROIs in an image is expected to improve its interpretability and classification accuracy. Note that the only manual supervision required for our method is the image expression. No extra manual annotations are required.

Evaluation Metrics. We use standard classification accuracy (CL) for the classification task. We also report a classification confusion matrix. Localization of AUs is performed at CAM, and layer-wise attention maps. Although they are both normalized between [0, 1], they do not necessarily have the same value at the pixel level, compared to AU map, since attention is trained to be correlated with AU maps. Therefore, we resort to using cosine similarity (Eq. 2) as a localization (interpretability) measure to assess how a CAM or attention is well aligned with the AU map. To evaluate either a CAM or an attention map, it is first upsampled to the same size as the AU map ${\bm{A}}$. Higher similarity indicates better localization and, hence, better interpretability. We refer to the cosine similarity between a CAM and an AU map as CAM-COS and as ATT-COS for the case of attention map versus AU map. Similarly to the classification confusion matrix, we report a per-class cosine similarity matrix. During training, we select the model with the best classification accuracy (CL), and report its corresponding localization metrics.

Figure 4: Illustration of interpretability prediction over RAF-DB test samples using CAM method [93] with and without action units alignment. From left to right: Input image, true action units map ${\bm{A}}$, CAM ${\bm{M}[k]}$, attention ${\bm{T}_{5}}$.

4.2 Comparison with different methods

Table 1 shows the impact of using AU alignment over classification and CAM interpretability across different methods. Overall, using our proposed AU alignment helped improve the CAM interpretability by a large margin. Since CAMs are often built after the last layer where we applied the alignment, they are strongly affected, leading to better AU localization. We note that classification accuracy (CL) has also improved with a relatively large margin over AffectNet compared to RAF-DB. This may be related to the fact there is still a large room for improvement in AffectNet compared to RAF-DB. Regarding attention interpretability (Table 2), the effect of using AU loss is larger than in CAM. Most importantly, since the alignment is applied over layer-wise attention, it leads to attention that is better in interpretability than CAM. Note that achieving better classification accuracy does not necessarily translate to better interpretability. For instance, the vanilla APVIT method [82] has the highest classification accuracy CL of ${91\%}$ over RAF-DB. However, it only scores ${0.38}$ in terms of attention interpretability ATT-COS, the low est score across all models. Since the vanilla case is optimized only to maximize the classification score, it can easily find features that are not necessarily interpretable to achieve that. Interestingly, we show that the same model can achieve a similar classification accuracy of ${91.03\%}$ and an attention interpretability score of ${0.85}$ when using our alignment loss. This suggests that spatial features have largely shifted their focus, as illustrated in Fig. 6, without compromising classification performance. As a result, the model finds different spatial features that satisfy both losses: classification and interpretability (AU). On RAF-DB and AffectNet, we note that our best obtained classification performance of ${91.03\%}$ and ${62.28\%}$ are close to the current state-of-the-art performance of ${92.21\%}$ and ${67.49\%}$ achieved by [50], respectively. Additionally, building AU maps during training adds negligible computational time. On the RAF-DB dataset, we obtain a training time of 1min 19sec /epoch and 1min 47sec /epoch without and with AU maps. Over the AffectNet dataset, rates of 15min 9sec /epoch and 17min 35sec /epoch are achieved without and with AU maps. We provide in Fig.4 an illustration of prediction with and without our action units alignment. In addition, Figs. 6, 7, 8, 9 show per-class average attention and CAM over both dataset. These visuals show better improvement of our classifier interprebility compared to vanilla case.

Figure 5: Ablations on the RAF-DB test set: impact of ${\lambda}$ over classification and localization (interpretability) performance. Alignment to AUs is performed over layer 5 of ResNet50 [31] with CAM [93] method.

4.3 Ablation Studies

We performed two relevant ablations, assessing how classification and localization (interpretability) performance is affected by ${\lambda}$ and at which layer the alignment loss is applied.

1- Impact of ${\lambda}$ (Fig. 5). Given different values of ${\lambda}$, we report in Fig. 5 its impact on localization (red, left curves) on CAM, and attention at layer 5 using CAM method [93] and a ResNet50 backbone [31] over the RAF-DB dataset. It is observed that increasing ${\lambda}$ improves localization at both CAM and attention. However, localization performance starts to stagnate after ${\lambda\approx 8}$. Additionally, since the alignment with AUs is performed at the attention level, localization at layer-wise attention (ATT-COS) is higher than at CAM-level (CAM-COS). However, they are still better than their corresponding vanilla cases (dashed lines). In practice, although CAMs can be used for interpretability, the attention layer is more accurate and, hence, more reliable. We also report the impact of ${\lambda}$ on classification accuracy (green, right curve). Increasing ${\lambda}$ showed to improve localization since much importance is considered for the alignment loss term. Classification accuracy also kept improving until ${\lambda\approx 4}$. However, large values of ${\lambda}$ put substantial importance on localization compared to classification, leading to degradation in classification accuracy. Hence, a compromise value is the one that improves localization without decreasing classification accuracy (e.g., ${\lambda\approx 4}$). Even large ${\lambda}$ values can yield relatively better classification accuracy than the vanilla case (dashed green line).

2- Layer where alignment loss is applied (Table 3). Overall, applying our AU loss term across single or multiple layers improved both classification and CAM-localization. However, its application on the last feature layer has shown to be most beneficial. Top layers often capture semantic and most discriminative regions [86]. Since AU maps highlight discriminative ROIs, using them to build better features at top layers is a reasonable choice. In all our experiments, we use the top layer for alignment.

5 Conclusion

In this work, we have introduced an approach to make FER models more interpretable. We extended the FER model training with an additional loss that favors feature maps whose average activations are similar to facial AU activations. As in most datasets AU localizations are not available, we estimate them by leveraging off-the-shelf facial point localizers and expert knowledge that associates AUs to the corresponding emotion. By doing that, we manage to obtain a much more reliable and interpretable activation of the FER model that, in many cases, also leads to better recognition performance.