AD-VAE: Adversarial Disentangling Variational Autoencoder

Abstract

Face recognition (FR) is a less intrusive biometrics technology with various applications, such as security, surveillance, and access control systems. FR remains challenging, especially when there is only a single image per person as a gallery dataset and when dealing with variations like pose, illumination, and occlusion. Deep learning techniques have shown promising results in recent years using VAE and GAN, with approaches such as patch-VAE, VAE-GAN for 3D Indoor Scene Synthesis, and hybrid VAE-GAN models. However, in Single Sample Per Person Face Recognition (SSPP FR), the challenge of learning robust and discriminative features that preserve the subject’s identity persists. To address these issues, we propose a novel framework called AD-VAE, specifically for SSPP FR, using a combination of variational autoencoder (VAE) and Generative Adversarial Network (GAN) techniques. The proposed AD-VAE framework is designed to learn how to build representative identity-preserving prototypes from both controlled and wild datasets, effectively handling variations like pose, illumination, and occlusion. The method uses four networks: an encoder and decoder similar to VAE, a generator that receives the encoder output plus noise to generate an identity-preserving prototype, and a discriminator that operates as a multi-task network. AD-VAE outperforms all tested state-of-the-art face recognition techniques, demonstrating its robustness. The proposed framework achieves superior results on four controlled benchmark datasets—AR, E-YaleB, CAS-PEAL, and FERET—with recognition rates of 84.9%, 94.6%, 94.5%, and 96.0%, respectively, and achieves remarkable performance on the uncontrolled LFW dataset, with a recognition rate of 99.6%. The AD-VAE framework shows promising potential for future research and real-world applications.

1. Introduction

Active since 1960 [1], the field of automatic face recognition has grown dramatically in the last few years, particularly with the rise of deep learning in recent years [2], becoming one of the most popular biometric methods with a wide range of applications [3]. These applications include security, surveillance, and access control systems. The increase is driven by technological advances (primarily through parallel processing in graphics processing units), new technologies for information processing, and security issues. These advances have awakened the interest of researchers in image processing, neural networks, computer vision, and computer graphics [4]. The main advantage of face recognition is that it does not require user cooperation, making it the least intrusive biometrics method.

Despite these advances, face recognition remains challenging in scenarios where only a single image per person is available for training. This is particularly critical in real-world cases such as criminal investigations, where the only image might come from an identity document or a passport. This problem, known as Single Sample Per Person Face Recognition (SSPP FR), is characterized by the need to recognize subjects across varying poses, expressions, and occlusions despite having only one reference image. The lack of sufficient training data makes this problem both relevant and challenging.

SSPP methods have certain advantages, such as reduced dataset complexity and lower computational costs. However, achieving robust and discriminative identity features from a single image remains difficult. Various methods have been proposed to address this challenge, including generic datasets to learn variations (e.g., SSRC [5]), patch-based approaches like DMMA [6], and deep learning models such as VD-GAN [7].

Considering the advances in the use of VAE and GAN, like VAE-GAN for 3D Indoor Scene Synthesis of [8], and the hybrid VAE-GAN of [9] and Patch VAE-GAN of [10], we propose an Adversarial Disentangling Variational Autoencoder (AD-VAE). Specifically designed to address the challenges of SSPP FR by learning robust identity representations, the framework architecture consists of four networks. The first two networks act as a variational autoencoder (VAE) and the other two as Generative Adversarial Networks (GANs) to generate a prototype that conserves the representative identity features of the subject. The VAE part trains the encoder to learn the input image’s distribution of latent space by the variational encoder–decoder process. The GAN part uses a latent code c derived from the encoder’s distribution, combined with a Gaussian noise vector to generate an identity-preserving and realistic prototype from the input image x. The discriminator of the GAN part works as a multi-task network with three tasks: (1) distinguishing the identity of the input image x, (2) distinguishing if the input image contains variations, and (3) determining whether the input image x is real or fake. These components work together to achieve better accuracy rates in SSPP FR.

Our method objective is to create an identity-preserver prototype $\widehat{x}$ from an input image x such that $\widehat{x}$ compared to any prototype image with the same identity x is close enough using K-Nearest Neighbors (KNN) to be recognized as the same subject. Achieving this objective, we outperform the state-of-the-art SSPP FR methods on four controlled datasets and a wild dataset, LFW. The contributions of this work can be summarized as follows:

• We propose a novel framework, AD-VAE, that combines the ability of VAE to disentangle identity representations with the capacity of the GAN to generate identity-preserving prototypes.
• The AD-VAE achieves state-of-the-art results on four controlled datasets (AR, E-YaleB, CAS-PEAL, and FERET) and the uncontrolled dataset LFW, demonstrating its robustness in handling variations in pose, illumination, and occlusion.
• Unlike other methods, AD-VAE accomplishes this without requiring external pre-trained encoders, making it a self-contained solution for SSPP FR.

The paper is organized into the following sections: Section 2 provides an overview of the related works, as well as a short review of GAN, VAE, and VD-GAN, and shows the proposed method; Section 3 describes the experiments performed on five of the most used public datasets; and Section 4 concludes the study.

2. Materials and Methods

The Materials and Methods section describes with sufficient detail the proposed AD-VAE method which integrates the VAE disentangled representation learning and GAN synthesis fidelity. The section is divided into Section 2.1, related works, Section 2.2, the background of the method, and Section 2.3, the description of the proposed method.

2.1. Related Works

Throughout the last few decades, various SSPP FR methods have arisen, as well as the classifications forms. Observing the methods and classifications reviewed, we can propose a classification in two classes: (1) geometric-based methods (most old methods that use eye sizes, distances, etc.); (2) appearance-based methods, which are sub-divided into (1) holistic, that uses the whole face as the method input, (2) local, that use patches of the face, with or without superposing/superposition, thus using local features and relationships; and (3) hybrid, using the junction of cited sub-classes (1) and (2). Using this classification, we classify our method as the holistic appearance-based method.

From the reviewed methods for SSPP FR, most of them use a generic dataset to learn about face variations, being holistic-based or patch-based, like a S³RC [11] or VD-GAN [7]. The generic dataset is a face database with variations like pose, occlusion expressions, and illumination. The subject’s identities in the generic database must not be in the enrollment database. Some methods use image synthesis to supply the lack of images for training in the enrollment dataset, such as [12]. Other authors like [13] use patch-based methods that divide a face image into small parts and use these parts for recognition. For instance, 3D face reconstruction and illumination transfer techniques have been employed to enhance reference datasets [14]. Similarly, the Uniform Generic Representation (UGR) method combines local and global generic representations to handle variations in pose, illumination, and occlusion [15].

Most of the reviewed methods are based on comprehensive sensing [16], like Sparse Representation Classifier (SRC) or Collaborative Representation Classifier (CRC). Highlighting the SSRC [5] as the principal representative, considering that a sample image can be reconstructed as a linear combination of all training samples and classified leads to the minimal residual. All SSRC processes are guided by Equation (1). Various methods use variations of Equation (1) and an auxiliary generic dataset to complement the lack of information about variations in SSPP enrollment sets:

$$\begin{array}{c}\hfill \mathbf{y}=P\mathit{\alpha }+V\mathit{\beta }+\mathbf{z}\end{array}$$

(1)

where $\mathbf{P}$ is the sample dictionary, $\mathbf{V}$ is the variation dictionary, z is the noise error, $\alpha$ is the sparse coefficient vector that selects a few samples from the $\mathbf{P}$ dictionary, and $\beta$ is the sparse coefficient vector that selects a small subset from the variation dictionary $\mathbf{V}$.

In our review, methods based on machine learning have achieved the best results, like the CJR-RACF [17] and VD-GAN [7]. Both techniques use deep convolutional networks. The CJR-RACF uses a patch-based convolutional network. The method in [17] divides the input image into local regions and learns local and global discriminative features. The VD-GAN generates an identity-preserving prototype of the input image with a representative identity feature by convolutional networks. Hybrid approaches such as the combining of hand-crafted features with deep learning features from CNNs have also shown promise in improving recognition accuracy [18].

VD-GAN is based on the image synthesis method DR-GAN [19] that uses GAN [20] combined with a disentangling approach to generate realistic images. Other image synthesis methods, such as AVAE [21], ID-GAN [22], InfoGAN [23], and DisP+V [24], can serve as a foundation for creating SSPP FR methods by leveraging the combined strengths of VAE [25] and GAN. Specifically, VAE contributes by learning a continuous latent space that captures identity-preserving features while being robust to variations in pose, lighting, and occlusion, which is essential for face recognition. GAN complements this by generating high-quality images that maintain identity consistency, with its discriminator enabling multi-task learning to distinguish between real and synthetic images, identify the subject’s identity, and detect variations. This work builds on the AVAE framework and the identity disentangling of the VD-GAN framework, taking advantage of robust feature representation of VAE and the capacity of GAN to generate realistic, identity-preserving prototypes.

2.2. Background

2.2.1. Generative Adversarial Networks

Initially created by [20] to generate synthetic images, the GAN was used to solve various problems, like face recognition. This method consists in two networks that play a minimax two-player game. The first network is the generator G, which uses a random noise vector z (sampled from a random Gaussian distribution) to generate an image that aims to look like a real one. The second network is Discriminator D, which determines if an input image is real or fake. The game consists of generator G generating an image that looks real enough to fool discriminator D, and discriminator D tries not to be fooled by generator G. In practice, to provide stronger gradients, the G can be trained by maximizing $logD\left(G\right(z\left)\right)$. This adversarial game between Generator G and Discriminator D, where each strives to outsmart the other, is represented by the value function $V(G,D)$:

$$\begin{array}{c}\hfill \underset{D}{max}{V}_D(G,D)={\mathbb{E}}_{x\sim p_{\mathrm{data}}\left(x\right)}[logD\left(x\right)]\\ \hfill +{\mathbb{E}}_{z\sim p_{\mathrm{z}}\left(z\right)}[log(1-D(G\left(z\right)\left)\right)]\end{array}$$

(2)

$$\underset{G}{max}{V}_G(G,D)={\mathbb{E}}_{z\sim p_{\mathrm{z}}\left(z\right)}[log\left(D\right(G\left(z\right)\left)\right)]$$

(3)

where $p_{\mathrm{data}}$ is the distribution of training images, and $p_{\mathrm{z}}$ is a distribution of noise.

2.2.2. Variational Autoencoders

Proposed by [26], the variational autoencoder (VAE) assumes that an image x of training data X is the result from a deterministic function $f\left(z\right)$ in a random variable $z\sim p\left(z\right)$ in latent space Z such as $f:(z,ϵ)\to x$, $ϵ$ being a stochastic noise. The probability of observing z knowing x is estimated by a decoder $p_{{\theta }_d}:z\mapsto p_{{\theta }_d}(x\mid z)$ parameterized by ${\theta }_d$, and the probability that z is the latent source of x is estimated by encoder $q_{{\theta }_e}:x\mapsto q_{{\theta }_e}(z\mid x)$ parameterized by ${\theta }_d$. Being that the data $X=(x^{\left(1\right)},...,x^{\left(n\right)})$ with n is the number of samples in the data, the parameters of the model are obtained by maximizing the log-likelihood of the observations: $log{p}_{{\theta }_d}\left(x^{\left(i\right)}\right)=log{\int }_Z{p}_{{\theta }_d}(x^{\left(i\right)}\mid z)p\left(z\right)dz$. The $log{p}_{{\theta }_d}\left(x^{\left(i\right)}\right)$ is computed by maximizing a tractable lower bound, training the VAE with the following loss function:

$$\begin{array}{cc}{\mathcal{L}}_{VAE}({\theta }_e,{\theta }_d;x))\hfill & ={\mathbb{E}}_{q_{{\theta }_e}(z\mid x)}[-log{p}_{{\theta }_d}(x\mid z)]\\ \hfill & +KL\left(q_{{\theta }_e}(z\mid x)p\left(z\right)\right))\hfill \end{array}$$

(4)

where $p_{{\theta }_d}$ is usually chosen as a Gaussian distribution $\mathcal{N}(x;{\mu }_{{\theta }_d}\left(z\right),Id)$ and $KL$ is the Kullback–Leibler divergence [10,27]. This Equation (4) estimates the reconstruction error and forces the distribution of the latent space to match with $p\left(z\right)$. The standard Gaussian distribution, $\mathcal{N}(z;0,Id)$, is typically chosen for $p\left(z\right)$ due to its simplicity and flexibility.

Equation (4) uses the KL divergence to measure the difference between the approximate distribution of the latent space, $q_{{\theta }_e}\left(z\right|x)$, obtained by the encoder, and the prior distribution $p\left(z\right)$. Minimizing KL divergence ensures that the distribution of the latent space is as close as possible to the prior distribution, preventing the encoder from learning a complex distribution or one that is not useful for generating new data.

2.2.3. Variation Disentangling Generative Adversarial Networks

The Variation Disentangling Generative Adversarial Network (VD-GAN) proposed by [7] uses a structure based on GAN, with a generator G and discriminator D. The generator G in VD-GAN consists of two networks, $G_{enc}$ and $G_{dec}$, like an autoencoder. The $G_{enc}$ network receives an image x from training data X, and aims to learn an identity representation $f\left(x\right)=G_{enc}\left(x\right)$, while the network $G_{dec}$ aims to synthesize a prototype image $\widehat{x}$ with the same subject identity of x by using the learned identity $f\left(x\right)$ and a random noise vector $z\sim p\left(z\right)$ (from a Gaussian distribution) $\widehat{x}=G_{dec}(f\left(x\right),z)$.

The discriminator D is used like a multi-task network with three sub-discriminators $D^{id}$, $D^{var}$, and $D^{gan}$. The sub-discriminator $D^{id}$ is used like a classifier to define the input image identity. The network output is an $N_d$-dimensional vector, with $N_d$ being the total number of identities. The second task $D^{var}$ is a binary classifier to distinguish if an input image has a variation (any image different from the real prototype is considered with variation). The third task is a classic GAN task, where $D^{gan}$ distinguishes if the generated image is real or fake.

The method training the autoencoder G has the following objective function:

$$\begin{array}{c}\hfill \underset{G}{max}{V}_G=V_G^{gan}+{\mu }_1{V}_G^{id}+{\mu }_2{V}_G^{var}-{\mu }_3{V}_G^{rec}\end{array}$$

(5)

where ${\mu }_1$, ${\mu }_2$ and ${\mu }_3$ are the weighting hyper-parameters for the hybrid objective $V_G$, and $V_G^{rec}$ is the reconstruction loss of the prototype. The discriminator is trained with the following objective function:

$$\begin{array}{c}\hfill \underset{D}{max}{V}_D=V_D^{gan}+{\lambda }_1{V}_D^{id}+{\lambda }_2{V}_D^{var}\end{array}$$

(6)

where ${\lambda }_1$ and ${\lambda }_2$ are trade-off parameters.

2.3. The Proposed Method

The proposed method AD-VAE combines the effective disentangled representation learning of the VAE-based approach described by [23] and the high-fidelity synthesis of the GAN-based techniques described by [22]. Similar to [7], we use a variation disentangling and discriminative identity representation via a GAN-based network but joined with a disentangled representation of the VAE base. The architecture of AD-VAE can be divided into two parts, a VAE part and a GAN part. The first part of the AD-VAE architecture is illustrated in Figure 1 and the second part in Figure 2.

The proposed method is formed by four networks that have the training divided into two parts. All method parts are performed sequentially. The first two networks are similar to the original VAE. Using an encoder ${\mathbf{E}}_{nc}\left(x\right)$ that has as input an image $x\sim P_{data}\left(x\right)$ and outputs a mean $\mu$ and variance $\sigma$ from the latent space of x, from the distribution formed by outputs mean and variance, vector $c\sim \mathcal{N}(\mu ,\sigma )$ is sampled. This sample vector c is the input of ${\mathbf{D}}_{ec}\left(c\right)$ that outputs an image reconstruction $x_{rec}$ of the image x. The first part of the method aims to train the encoder to learn more disentangled representation from the image distribution. Like [21], the reconstruction error is incorporated into the ELBO (Evidence Lower Bound), which is estimated using the Kullback–Leibler (KL) divergence between the posterior distribution $q\left(z\right|x)$ and the prior distribution $p\left(z\right)$. The ELBO is defined as

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{2}}\sum _{j=1}^{dim\left(Z\right)}{\sigma }_{E_{nc}j}^2+{\mu }_{E_{nc}}^2{\left(x\right)}_j-1-log{\sigma }_{E_{nc}j}^2\end{array}$$

(7)

Then, the objective function of the VAE part is defined as the following:

$$\begin{array}{c}\hfill {\mathcal{L}}_{VAE}(E_{nc},D_{ec};x))={\displaystyle \frac{1}{2}}{∥{\mu }_{D_{ec}}\left(z\right))-x∥}^2\\ \hfill +{\displaystyle \frac{1}{2}}\sum _{j=1}^{dim\left(Z\right)}{\sigma }_{E_{nc}j}^2+{\mu }_{E_{nc}}^2{\left(x\right)}_j-1-log{\sigma }_{E_{nc}j}^2\end{array}$$

(8)

After the VAE training, the second part is trained with a frozen encoder ${\mathbf{E}}_{nc}$ and using a structure encode generator, like a normal autoencoder. The generator ${\mathbf{G}}_{en}$ uses as input a sample c from an encoder distribution of x, $c\sim \mathcal{N}({\mu }_{{\mathbf{E}}_{nc}},{\sigma }_{{\mathbf{E}}_{nc}})$, and a random noise z from a Gaussian distribution $z\sim \mathcal{N}(0,1)$. Naming the function that extracts a sample from the encoder distribution as $f\left({\mathbf{E}}_{nc}\left(x\right)\right)=f({\mu }_{{\mathbf{E}}_{nc}},{\sigma }_{{\mathbf{E}}_{nc}})=c$, we have that $\widehat{x}={\mathbf{G}}_{en}(f\left({\mathbf{E}}_{nc}\left(x\right)\right),z)$, with $\widehat{x}$ being the generated image with the same identity as x. For training the ${\mathbf{G}}_{en}$, we use the following five objective functions:

$$\begin{array}{c}\hfill \underset{{\mathbf{G}}_{en}}{max}{V}_{{\mathbf{G}}_{en}}=V_{{\mathbf{G}}_{en}}^{gan}+{\mu }_1{V}_{{\mathbf{G}}_{en}}^{id}+{\mu }_2{V}_{{\mathbf{G}}_{en}}^{var}+{\mu }_3{V}_{{\mathbf{G}}_{en}}^{rec}+{\mathcal{L}}_{\mathcal{C}}\end{array}$$

(9)

where ${\mu }_1$, ${\mu }_2$, and ${\mu }_3$ are the weighting hyper-parameters for $V_{{\mathbf{G}}_{en}}$. The sub-objectives are defined as follows:

$$\begin{array}{c}\hfill V_{{\mathbf{G}}_{en}}^{id}({\mathbf{G}}_{en},{\mathbf{D}}^{id},\mathbf{c},\mathbf{z})={\mathbb{E}}_{c,y_{id},z}[log{\mathbf{D}}_{y_{id}}^{id}\left(G(\mathbf{c},\mathbf{z})\right)]\end{array}$$

(10)

$$\begin{array}{c}\hfill V_{{\mathbf{G}}_{en}}^{var}({\mathbf{G}}_{en},{\mathbf{D}}^{var},\mathbf{c},\mathbf{z})={\mathbb{E}}_{c,y_{var},z}[log{\mathbf{D}}_{y_{var}}^{var}\left(G(\mathbf{c},\mathbf{z})\right)]\end{array}$$

(11)

$$\begin{array}{c}\hfill V_{{\mathbf{G}}_{en}}^{gan}({\mathbf{G}}_{en},{\mathbf{D}}^{gan},\mathbf{c},\mathbf{z})={\mathbb{E}}_{c,z}[log{\mathbf{D}}^{gan}\left({\mathbf{G}}_{en}(\mathbf{c},\mathbf{z})\right)]\end{array}$$

(12)

$$\begin{array}{c}\hfill V_{{\mathbf{G}}_{en}}^{rec}({\mathbf{G}}_{en},{\mathbf{x}}_{rp},\mathbf{z})={\mathbb{E}}_{{\mathbf{x}}_{rp},\mathbf{z}}\left[{\displaystyle \frac{1}{2}}{∥{\mathbf{x}}_{rp}-{\mathbf{G}}_{en}(f\left({\mathbf{E}}_{nc}\left({\mathbf{x}}_{rp}\right)\right),\mathbf{z})∥}^2\right]\end{array}$$

(13)

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{C}}({\mathbf{G}}_{en},\mathbf{x},\widehat{\mathbf{x}})={\displaystyle \frac{1}{2}}{∥{\displaystyle \frac{{\mu }_{{\mathbf{E}}_{nc}}\left({\mathbf{x}}_{rp}\right)-{\mu }_{{\mathbf{E}}_{nc}}\left(\widehat{\mathbf{x}}\right)}{{\sigma }_{{\mathbf{E}}_{nc}}\left(\widehat{\mathbf{x}}\right)}}∥}^2\end{array}$$

(14)

where $\mathbf{x},{\mathbf{x}}_{rp},{\mathbf{y}}_{id},{\mathbf{y}}_{var}$ come from the training data $X=\{[x^1,x_{rp}^1,y_{id}^1,,y_{var}^1],...,[x^n,x_{rp}^n,y_{id}^n,,y_{var}^n]\}$. Being that $x^i$ is a random image of subject i, $x_{rp}^i$ is a real prototype image of subject i, $y_{id}^i$ is the label identity of subject i, and $y_{var}^i$ is the label that distinguishes whether $x^i$ has variation.

The sub-objective functions of generator ${\mathbf{G}}_{en}$ have the following objectives:

• $V_{{\mathbf{G}}_{en}}^{id}$: Enable ${\mathbf{D}}^{id}$ to classify the generated prototype image $\widehat{\mathbf{x}}$ as the same identity of label $y^{id}$ of $\mathbf{x}$.
• $V_{{\mathbf{G}}_{en}}^{var}$: Enable ${\mathbf{D}}^{var}$ to detect that there are no variations in $\widehat{\mathbf{x}}$.
• $V_{{\mathbf{G}}_{en}}^{gan}$: Fool ${\mathbf{D}}^{gan}$ to classify the generated prototype $\widehat{\mathbf{x}}$ as a real prototype.
• $V_{{\mathbf{G}}_{en}}^{rec}$: Enable the generator to generate an image $\widehat{\mathbf{x}}$ closest to the real prototype image ${\mathbf{x}}_{rp}$.
• ${\mathcal{L}}_{\mathcal{C}}$: Enable the generator to generate an image $\widehat{\mathbf{x}}$ such that the prior distribution of ${\mathbf{E}}_{nc}\left(\widehat{\mathbf{x}}\right)$ is closest to the prior distribution of ${\mathbf{E}}_{nc}\left({\mathbf{x}}_{rp}\right)$.

The last network $\mathbf{D}$ is trained using an objective function as follows:

$$\begin{array}{c}\hfill \underset{D}{max}{V}_D=V_D^{gan}+{\lambda }_1{V}_D^{id}+{\lambda }_2{V}_D^{var}\end{array}$$

(15)

where ${\lambda }_1$ and ${\lambda }_2$ are trade-off parameters, and $V_D^{id}$, $V_D^{var}$, and $V_D^{gan}$ are defined as follows:

$$\begin{array}{c}\hfill V_D^{id}(D^{id},\mathbf{x})={\mathbb{E}}_{x,y_{id}}[log{D}_{y_{id}}^{id}\left(\mathbf{x}\right)]\end{array}$$

(16)

$$\begin{array}{c}\hfill V_D^{var}(D^{var},\mathbf{x})={\mathbb{E}}_{x,y_{var}}[log{D}_{y_{var}}^{var}\left(\mathbf{x}\right)]\end{array}$$

(17)

$$\begin{array}{c}\hfill V_D^{gan}(D^{id},\mathbf{x})={\mathbb{E}}_{{\mathbf{x}}_{rp}}[log{D}^{gan}\left({\mathbf{x}}_{rp}\right)]+\\ \hfill {\mathbb{E}}_{\mathbf{x},\mathbf{z}}[log(1-D^{gan}\left({\mathbf{G}}_{en}(\mathbf{f}\left({\mathbf{E}}_{nc}\left(\mathbf{x}\right)\right),\mathbf{z})\right))]\end{array}$$

(18)

The discriminator $\mathbf{D}$ has the following objectives:

• $V_D^{id}$: Predict a correct identity of input image $\mathbf{x}$ as labeled in $y_{id}$.
• $V_D^{var}$: Predict a correct occurrence of variation on input image $\mathbf{x}$ as labeled in $y_{var}$.
• $V_D^{gan}$: Predict the real prototype image ${\mathbf{x}}_{rp}$ as real and predict a generate prototype image $\widehat{\mathbf{x}}$ as fake.

The four networks are training sequentially, with ${\mathbf{E}}_{nc}$ and ${\mathbf{D}}_{ec}$ trained by Equation (8), ${\mathbf{G}}_{en}$ trained by Equation (9), and $\mathbf{D}$ by Equation (15). As a result, the encoder ${\mathbf{E}}_{nc}$ learns a latent space with more representative identity from image $\mathbf{x}$ by VAE-based method, and the generator ${\mathbf{G}}_{en}$ learns to create a prototype image $\widehat{\mathbf{x}}$ preserving more identity features of the subject from $\mathbf{x}$. The following section demonstrates the effectiveness of the proposed method.

3. Results

In this section, we present a dataset setup and the methodology for the experiments, including the implementation details, and the method effectiveness by experimental results.

3.1. Data Collection, Pre-Processing, and Feature Selection

For the tests, we use five widely recognized databases in face recognition, selected for their frequent use in reviewed papers and based on their variability in pose, illumination, occlusion, and real-world conditions. The datasets contain a mix of controlled and uncontrolled environments to ensure diversity and robustness in evaluation. The datasets are the following:

• AR [28] consists of 126 identities, having 26 images with expression, illumination, and conclusion per subject. From this dataset, we use a subset with 100 identities. Randomly, we choose 50 identities for the training set and 50 for the test set.
• Extend Yale B (E-YaleB) [29] consists of 38 identities under a wide range of lighting conditions, including variations in light intensity (ranging from low to high), different types of lighting (such as natural light, artificial light, and directional lighting) and various light angles (e.g., frontal, lateral, and top–down). Due to the low number of subjects, according to [7], we introduce the AR lighting subset into E-YaleB to extend the number of identities. We randomly choose 100 identities from the mixed dataset for the training set and the remaining 38 identities for the test set.
• FERET [30] consists of 1199 identities with variations in gender, age, and ethnicity. From this dataset, we use a subset of 200 identities containing only four pose variations. We randomly choose 150 identities for the training set and the remaining 50 for the test set.
• CAS-PEAL [31] consists of 1040 identities with variations like poses, occlusions, and ages. From this dataset, we use a subset with 300 identities from normal and accessory categories, with a neutral image and another 6 wearing different glasses and hats. We randomly choose 200 identities for the training set and the remaining 100 for the test set.
• LFW [32] consists of 5749 identities collected under an uncontrolled environment, with a wide range of expressions, poses, illuminations, and other variations. We use a subset of 158 identities with more than ten images per subject from the aligned version of LFW, the LFW-a. For evaluation, we choose 50 identities containing neutral face images for the test set and the other 108 for the training set.

According to [2], the AR and LFW datasets exhibit more complex facial variations compared to others, making them particularly challenging for SSPP FR.

We apply some pre-processing steps on the data to ensure the consistency across all datasets:

• Image resizing: All images are resized to 64x64 pixels to match the input dimensions of the network.
• Normalization: Pixel values are normalized to the range [0, 1] to improve convergence during training.
• Alignment: For the LFW dataset, we use the aligned version (LFW-a) to reduce variations caused by misalignment.
• Handling missing values: All datasets used in this study contain complete data, with no missing values, eliminating the need for further data imputation.

The networks process the pre-processed data as follows:

• Latent code generation: The encoder (${\mathbf{E}}_{nc}$) generates the mean ($\mu$) and variance ($\sigma$) of the latent space distribution. The latent code (c) is then sampled from this distribution using the reparameterization trick, which ensures that c is differentiable with respect to the network’s parameters. This differentiability allows for gradient-based optimization during training. A noise vector (z) is sampled independently from a Gaussian distribution for variation modeling.
• Feature concatenation: The generator (${\mathbf{G}}_{en}$) combines c and z into a single input vector ($c,z$) to create identity-preserving prototypes with controlled variations.
• Representation dimensionality: For all datasets, the latent dimension ($L_{dim}$) is set to 100, ensuring consistent feature representation across datasets.

The AD-VAE networks’ architectures are shown in

Table 1. Structure network of ${\mathbf{E}}_{nc}$ and $\mathbf{D}$. Every Conv2d layer is preceded by a BatchNorm2d normalization layer and a LeakyReLU activation layer.

Encoder ${\mathbf{E}}_{\mathit{n}\mathit{c}}$	Discriminator D
Layer	input/output	Filter/Stride/Padding
Conv2d-1	3/64	4x4/2/1
Conv2d-2	64/128	4x4/2/1
Conv2d-3	128/256	4x4/2/1
Conv2d-4	256/512	4x4/2/1
${\mathbf{E}}_{nc}$ Finals layers	Flatten
Fullconected-$\mu$	output = $L_{dim}$	FullConected-$\sigma$
$\mathbf{D}$ Final layers	Flatten
Fullconected	output = $N_{dim}+2$

and

Table 2. Structure network of ${\mathbf{D}}_{ec}$ and ${\mathbf{G}}_{en}$. The ConvTranspose2d layers 1, 2, and 3 is preceded by a BatchNorm2d normalization layer and a ReLU activation layer.

Generator ${\mathbf{G}}_{\mathit{e}\mathit{n}}$	Decoder ${\mathbf{D}}_{\mathit{e}\mathit{c}}$
Fullconected	$G(L_{dim}\ast 2/8192)$	$D(L_{dim}/8192$)
Rechape(512x4x4)>	BatchNorm2d>	ReLU
Layer	input/output	Filter/Stride/Padding
ConvTranspose2d-1	512/256	4x4/2/1
ConvTranspose2d-2	256/128	4x4/2/1
ConvTranspose2d-3	128/64	4x4/2/1
ConvTranspose2d-4	64/3	4x4/2/1
Tanh

. The networks ${\mathbf{E}}_{nc}$ and $\mathbf{D}$ are similar, differing only in the final layer, having ${\mathbf{E}}_{nc}$ two fully connected layers, one layer that outputs a mean $\mu$ and another to the variance $\sigma$, both with dimension $L_{dim}$. For network $\mathbf{D}$, the last layer is a fully connected layer with output dimension $N_{dim}+2$, where $N_{dim}$ is the number of identities ($D^{id}$) of the training set. The other two positions ($+2$) are used to distinguish the occurrence of variation ($D^{var}$) and if the input image is real or fake ($D^{gan}$).

The networks ${\mathbf{G}}_{en}$ and ${\mathbf{D}}_{ec}$ are also similar, with a difference in the top layer. The first layer of ${\mathbf{G}}_{en}$ uses an input of dimension $L_{dim}\ast 2$ because the input is a concatenation of latent code c and noise vector z. The ${\mathbf{D}}_{ec}$ first layer uses an input of dimension $L_{dim}$ because only a latent code c is used as input.

The network’s weights are initialized from a zero-centered Gaussian distribution with a standard deviation of 0.002. Training is performed using mini-batch stochastic gradient descent with a batch size of 16. The Adam optimizer [33] is employed with the following learning rates for each network: 0.0002 (${\mathbf{E}}_{nc}$), 0.0002 (${\mathbf{D}}_{ec}$), 0.0001 (${\mathbf{G}}_{en}$), and 0.0003 ($\mathbf{D}$). For all datasets, we set the parameters ${\mu }_1$, ${\mu }_2$, ${\mu }_3$ in Equation (9) and ${\lambda }_1$, ${\lambda }_2$ in Equation (15) as 2.0, 0.5, 0.1, 2.0, and 0.5, respectively, as used by [7].

3.2. Evaluation in Single Sample Face Recognition

This section describes the results of the evaluation of AD-VAE on the task of recognizing a face that has only a near-neutral face as an enrollment sample. Initially, the experiment is carried out on controlled datasets AR, CAS-PEAL, FERET, and E-YaleB, and is finally implemented on the wild dataset LFW. We define the methods and dataset configurations as in [7]. Figure 3 shows samples of dataset images, showing the test image $\mathbf{x}$, the prototype of the test image $\widehat{\mathbf{x}}$, and the real prototype of the test image ${\mathbf{x}}_{rp}$.

For comparison in the controlled datasets, we use nine methods of the literature: PCA [34], VAE [25], SRC [35], CRC [36], PCRC [37], DMMA [6], SVDL [38], SLRC [39], and S³RC [11]. The training set works as the generic set for methods that use a generic gallery set to learn the variations of faces.

For DMMA and PCRC, the patch is non-overlapped with a patch size of 16x16 pixels. The other parameters of DMMA are set as $K_1=30,K_2=2,K=2$, and $\sigma =10$. We fix the regularization parameter $\lambda$ of SRC, CRC, SLRC, and S³RC as 0.01. For SVDL, the parameters are set as ${\lambda }_1=0.001,{\lambda }_2=0.01$, and ${\lambda }_3=0.0001$. To measure the similarity of generated prototypes in PCA and VAE, we use the cosine distance metric and KNN with $k=1$ as a classifier. For VD-GAN and AD-VAE, we generate a prototype of query image y and a prototype of each ${\mathbf{x}}_{rp}$. Then, we use the KNN with a cosine distance to classify the generated prototypes of the same subject. Considering $P\left(x\right)$ as a generated prototype from x in Equation (19), we classify the generated prototypes by Equation (20):

$$\begin{array}{c}\hfill P\left(x\right)={\mathbf{G}}_{en}(f\left({\mathbf{E}}_{nc}\left(x\right)\right),z)\end{array}$$

(19)

$$\begin{array}{c}\hfill ID\left(\mathbf{y}\right)=\underset{k}{argmin}dist(P\left(y\right),P\left({\mathbf{x}}_{rp}\right))\end{array}$$

(20)

Table 3. Recognition accuracies (%) and standard errors (%) of different methods on E-YaleB&AR, CAS-PEAL, AR, and FERET datasets for SSPP FR.

Methods	AR	E-YaleB&AR	CAS-PEAL	FERET
PCA	$42.4\pm 2.2$	$58.5\pm 2.3$	$51.3\pm 1.0$	$40.5\pm 31$
VAE	$44.9\pm 1.1$	$59.9\pm 1.1$	$51.4\pm 0.9$	$55.0\pm 2.3$
SRC	$49.6\pm 2.4$	$64.0\pm 3.8$	$62.3\pm 1.4$	$51.5\pm 2.6$
CRC	$50.8\pm 4.8$	$63.5\pm 1.4$	$69.5\pm 2.7$	$43.0\pm 4.1$
DMMA	$51.9\pm 1.9$	$55.4\pm 1.1$	$59.2\pm 0.6$	$57.5\pm 1.2$
PCRC	$74.1\pm 3.7$	$80.7\pm 5.4$	$75.8\pm 0.6$	$24.0\pm 2.4$
SVDL	$76.0\pm 0.8$	$88.1\pm 1.8$	$78.7\pm 1.2$	$67.0\pm 1.7$
SLRC	$76.6\pm 1.8$	$88.8\pm 2.6$	$78.2\pm 3.3$	$68.0\pm 3.8$
S3RC	$77.8\pm 2.6$	$88.2\pm 1.5$	$80.3\pm 3.3$	$73.0\pm 2.1$
VD-GAN	$79.7\pm 0.8$	$90.6\pm 2.5$	$81.2\pm 2.2$	$90.5\pm 0.8$
AD-VAE	$84.9\pm 1.5$	$94.6\pm 1.8$	$94.5\pm 1.6$	$96.0\pm 1.0$

presents the recognition accuracy on four controlled datasets. Results show that the AD-VAE outperforms all other methods on four datasets. Our method achieves accuracy rates superior to VD-GAN, which overcomes the generic data learning methods. Like the VD-GAN, the proposed method outperforms the linear-based superposition model methods concerning non-linear variations, as pose variation in the FERET dataset. The AD-VAE overcomes the others due to the learning variation being disentangled over the encoder training, resulting in a latent vector more representative of each identity.

Even with similarities, the encoder training by VAE and the additional ${\mathcal{L}}_{\mathcal{C}}$ loss function could outperform VD-GAN. According to the authors of VD-GAN, the VAE is not competitive due to being an unsupervised method, whereas the AD-VAE uses the strengths of the VAE in a supervised form, thus achieving better rates than VD-GAN.

To test in the wild dataset, we evaluate the AD-VAE, comparing with VD-GAN_Lcnn (VD-GAN version using LightCNN-29 as the encoder in autoencoder Generator G) and four recent deep learning-based methods: JCR-ACF [40], Regular-face [41], Arc-face [42], and CJR-RACF [17].

Table 4. Recognition accuracies (%) of different deep learning-based methods on LFW dataset for SSPP FR.

Methods	Recognition Rate (%)
JCR-ACF	86.0%
Regular-face	83.7%
Arc-face	92.3%
CJR-RACF	95.5%
VD-GANLcnn	98.4%
AD-VAE	$99.6\pm 1.2$%

reports the rank-1 recognition rates of all the methods for SSPP FR, showing that our AD-VAE outperforms all the others, even the VD-GAN with encoder modifications.

4. Conclusions

We proposed the AD-VAE framework, which, to the best of our knowledge, is the first to leverage a variational autoencoder in the Single Sample Per Person Face Recognition (SSPP FR) problem. The key contributions of this research include the development of a novel method capable of learning identity-preserving prototypes from both controlled and wild SSPP FR datasets. The standard form of AD-VAE outperformed all SSPP FR techniques tested without any alteration of the AD-VAE original model, showing the robustness of our approach. AD-VAE effectively deals with huge variations, such as the pose of FERET, illumination, and occlusion of AR, the complex illumination condition of EYaleB, and the mixed variations of the wild dataset LFW.

The results highlight the effectiveness of AD-VAE in handling significant variations, such as pose variations in FERET, illumination and occlusion in AR, the challenging lighting conditions of E-YaleB, and the mixed variations present in the wild dataset LFW. These findings confirm the potential of AD-VAE to address critical challenges in SSPP FR and establish it as a strong candidate for practical applications in real-world scenarios. Additionally, the results demonstrate that variational autoencoders can be effectively used to disentangle identity and variation features in SSPP FR tasks, opening new possibilities for exploring disentangled representations in biometric applications, and the framework achieves high recognition rates without relying on external pre-trained networks, providing a cost-effective and efficient approach to SSPP FR.

However, there are several areas for further improvement and exploration. To enhance the learning capacity for representation of our framework, we could incorporate new network architectures. Potential improvements include intermediate latent space combined with adaptive instance normalization used in StyleGAN2 [43], or the application of diffusion models, such as the one used in the Dual Condition Face Generator (DCFace) [44], for style transfer methods. Furthermore, leveraging larger datasets, such as GAN-Control [45], could further improve performance. An additional enhancement could be the adoption of a separate VAE training step, as in the ID-GAN model [22], which allows for more refined control of face generation and increasing recognition accuracy.