HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field

Parametric modeling of 3D face has been intensively studied in the past two decades. In the form of explicit meshes, parametric face models are compact, controllable, and easy to be animated. The pioneer work of Blanz and Vetter [1999] builds 3D morphable model to represent facial shape, expression, and appearances. Recently, the parametric face models become more expressive by exploiting more powerful modeling techniques, including multi-linear or nonlinear models [Vlasic et al. 2006; Neumann et al. 2013; Brunton et al. 2014; Li et al. 2010; Tewari et al. 2018; Tran et al. 2019; Tran and Liu 2018; Li et al. 2020] and the articulated control of expression [Li et al. 2017]. To model detailed deformations of expression, recent state-of-the-art methods [Feng et al. 2021; Danecek et al. 2022] further learn additional displacement maps with the conditions of image inputs. Moreover, learning-based generative models such as GAN [Karras et al. 2017] or styleGAN [Karras et al. 2020; Karras et al. 2020] are also used in existing models [Gecer et al. 2021; Cheng et al. 2019; Nagano et al. 2018; Wang et al. 2022b; Nagano et al. 2019; Lattas et al. 2021; Luo et al. 2021] to enhance the accuracy of facial texture or geometry modeling. Despite the remarkable progress, all these parametric models can only capture the relatively coarse geometry and appearance of the facial region with the explicit mesh representations, which limits the realism of those reconstruction and animation approaches [Grassal et al. 2022; Cao 2016; Hu et al. 2017] built upon them. Instead of solely relying on explicit face models, our approach proposes a controllable hybrid explicit-implicit representation for photo-realistic rendering of 3D face.

To utilize the explicit facial model to represent the entire dynamic human head, some methods [Kim et al. 2018; Koujan et al. 2020; Doukas et al. 2021; Thies et al. 2019;, 2020] combine classical rendering and learned image synthesis to establish 2D avatar based on the monocular video. Deep Video Portraits [Kim et al. 2018] presented impressive full head reenactment and photo-realistic image results based on an image2image translation framework. Head2Head [Koujan et al. 2020; Doukas et al. 2021] further improved the temporal coherency with a sequential, video-based rendering network. Instead of using the raw texture of the fitted coarse facial model, Deferred Neural Render [Thies et al. 2019;, 2020] extended the idea by rendering the local feature embedded on the mesh surface. Though the rendering-to-video architecture shows an impressive performance in video portrait synthesis, it does not establish 3D representation for the full-head appearance.

In the past three years, it has been an emerging trend to model 3D scenes and objects in an implicit fashion with the success of implicit representations [Mildenhall et al. 2020; Yariv et al. 2020], based on which some works [Chan et al. 2022; Wang et al. 2022a; Kellnhofer et al. 2021; Mihajlovic et al. 2022] have explored to reconstruct high-fidelity view-consistent 3D appearance for static portraits or [Park et al. 2021b;, 2021a] model the dynamic scene with head movements. As for animatable personalized head character, many methods [Lombardi et al. 2019;, 2021; Wang et al. 2021; Gafni et al. 2021; Grassal et al. 2022; Zheng et al. 2022; Athar et al. 2022] attempted to build implicit representation-based personalized full-head avatar. Based on dense multiview capture systems, some researches [Lombardi et al. 2019;, 2021; Wang et al. 2021; Cao et al. 2021] are able to generate facial avatars with impressive subtle details and highly flexible controllability for immersive metric-telepresence. Though the recent work of Cao et al. [2022] supports creating authentic avatars from a phone scan, it relies on a prior model that is pretrained in a large-scale multiview-videos dataset captured in a complicated systems. High cost in data acquisition limited the broad applications. Under lightweight camera settings, based on implicit surface representation, I M Avatar [Zheng et al. 2022] improved generalization to novel expressions by incorporating skinning fields within an implicit morphing-based model, but showed blurry unsatisfying appearance performance. NeRFace [Gafni et al. 2021] showed state-of-the-art reenactment results with a parameter-controlled neural radiance field, but struggled to extrapolate to unseen expressions. Recently, RigNeRF [Athar et al. 2022] proposed to maintain a canonical neural radiance field with a backward deformation field guided by parametric model mesh, but suffers from the ambiguous correspondences determined by the Euclidean distance. Besides, for NeRF-based head avatar modeling methods [Gafni et al. 2021; Hong et al. 2022; Guo et al. 2021], there is a tendency to generate frame-wise inconsistent shape. The problem is originated from the unavoidable noise in the estimation of expressions and head poses, thus the similar input expressions may correspond to slightly different observed appearances, causing unstable canonical shape recovery. Skillfully incorporating the synthetic renderings of parametric model into neural radiance field, our approach achieves both expressive appearance and robust full-head control and further addresses the inconsistent shape by modulating feature generation with learnable embeddings.

NeRFace [Gafni et al. 2021] first extends NeRF [Mildenhall et al. 2020] to describe expression-related dynamic head appearance. Based on the classical backbone of eight fully connected layers, NeRFace additionally inputs low-dimensional expressions of the morphable model to condition the neural scene representation network for dynamically changing content. By employing the estimated head pose to transform the rays into the canonical space shared by all frames, the head canonical appearance volume \(H_C\) can be formulated as:where the implicit function maps the position in canonical space \(\mathbf {x_c}\) to density \(\sigma\) and color feature \(\mathbf {c}\) , under the control of facial expression parameters \(\delta _t\) , as well as per-frame embeddings \(\gamma _t\) to compensate for missing tracking information.

NeRFace relies on global expression blendshape parameters to represent diverse expression-related appearances. However, by simply learning the mapping from the global conditional vectors to appearances with only a short video sequence, it is easy to be overfit. Hence, though NeRFace is good at faithfully reconstructing the training sequences, without the awareness of the underlying 3D structure of human face, it struggles to generalize to unseen expressions.

To introduce the facial structure prior into NeRF [Mildenhall et al. 2020], we propose Parametric Face Model-conditioned Neural Radiance Field. Our definition of \(H_C\) is reformulated as:where we utilize the tracked deformed mesh model \(M_t\) in zero pose to condition the implicit function, as well as the head pose \(\mathbf {p_t}\) to describe the pose-related non-rigid deformation.

In this section, we will explain how to handle the rigid skeleton deformation driven by head poses. The straightforward treatment in NeRFace [Gafni et al. 2021], that the estimated head poses serve as camera poses, leads to the identical motion of both head and body, which is unrealistic. To render images agreeing with the ground-truth observation, the relative movement between the head and torso needs to be considered. As shown in Figure 7, the canonical appearance volume \(H_C\) should be warped to an observed posed appearance volume H with the rigid deformation T:Specifically, we compute the head rigid deformation T as inverse linear blend skinning that maps points from the posed space to the shared canonical space:where \(w_p\) represents the blend weight, \(R_{head}\) and \(t_{head}\) the head rotation and translation that comes from estimated head pose \(\mathbf {p_t}\) , and \(R_{torso}\) and \(t_{torso}\) means the torso movement, which is static by default. To avoid overfitting caused by learning backward skinning [Chen et al. 2021; Zheng et al. 2022], following HumanNerf [Weng et al. 2022], we solve for the weight volume in canonical space to derive the \(w_p\) as:Concretely, we set a 3D convolutional network \(W_c\) that inputs a constant random vector and generates the canonical weight volume \(w_c(x)\) with limited resolution that can be resampled via trilinear interpolation. With the optimized motion decoupling module, our method can separate out the head movement and stabilize the torso motion.

Although the aforementioned hybrid NeRF-based representation is more expressive than available methods, only relying on pixel supervision ( \(\mathit {MSE}\) / \(\mathit {l}_1\) RGB loss) can hardly yield high-frequency details in the rendered images. Hence, we incorporate the 3D representation into an image2image translation architecture and train the overall network jointly in an adversarial manner to enhance facial details and recover realistic portrait images.

Based on the established appearance volume, volume rendering is implemented using two-pass importance sampling as in Mildenhall et al. [2020]. To remain more 3D-aware information for the subsequent module to generate view-consistent images, similar to previous works [Niemeyer and Geiger 2021; Chan et al. 2022; Gu et al. 2021; Hong et al. 2022], we predict a low-resolution feature map \(128 \times 128 \times 64\) from a given camera pose via volumetric rendering, instead of directly rendering an RGB image. However, different from these methods leveraging up-sample super-resolution (SR) module, our approach chooses a UNet-style image2image translation network to transfer the raw feature maps to the final RGB images. The down-sampling encoding process in UNet helps the 2D network learn the global portrait features, which conduces to the view-consistent images generation.

Our architecture is presented in Figure 8, which includes two main modifications. First, we incorporate skip connections in the decoder, which map each intermediate feature image to an RGB image and integrate the previous output with the next output through addition. Second, we represent the output image as a wavelet (WT) following Gal et al. [2021], and the RGB image is generated through an inverse wavelet transform (IWT). This design choice helps reduce the number of parameters and speed up network computations.

The joint training of the overall network can guide NeRF module to provide sufficient and appropriate information for the image2image translation module to raise 3D awareness for regularizing time- and view- inconsistent tendencies. In the next section, we will explain the training procedure and the used loss functions in detail.

We mainly compare our method with the state-of-the-art 3D head avatar modeling methods: NeRFace [Gafni et al. 2021], IM Avatar (IMA) [Zheng et al. 2022], RigNeRF [Athar et al. 2022], and Neural Head Avatar (NHA) [Grassal et al. 2022]. For the monocular settings, we also compare with 2D facial reenactment method Head2Head++ (H2H++) [Doukas et al. 2021]. We conduct the comparison on the dataset of Zheng et al. [2022] and our own data. For IMA, NHA, and H2H++, the released data preprocessing codes are utilized to process the monocular videos, and we use our tracked data for NeRFace.² As the code of RigNeRF is not open-source, we re-implement it and leverage the tracked data of IMA’s preprocessing codes to train. To validate the expressiveness of our synthetic-rendering-based NeRF, we also provide a NeRF-baseline(SynR-NeRF) without the image2image translation module.

Qualitative results are presented in Figure 10. For IMA and NHA, their texture heavily relies on shape reconstruction, and the performance of appearance recovery is inferior. NeRFace cannot split the head motion and is more prone to generate unstable results for unseen expressions, as it lacks structure prior from the parametric model. H2H++ suffers from unrealistic image artifacts especially when dealing with challenging head poses. As RigNeRF is built on a backward deformation field guided by a coarse 3DMM mesh, for the unconstrained area such as the mouth interior, RigNeRF tends to establish ambiguous correspondences and generates blurry appearance. Compared with the above approaches, our SynR-NeRF baseline is capable of full-head control and accurate reconstruction of the expressions and head poses. Our full pipeline can, moreover, recover high-frequency details. The quantitative results presented in Table 1 further demonstrate the superiority of our method. Note that instead of focusing on the pixel-wise similarity, our full pipeline further improves the strength in detail generation and increases the perceptual similarity, which is proven by the gap of LPIPS scores between our method and the other methods. We also illustrate the comparison on monocular-based animation in Figure 12. In this experiment, we utilize part of the video from IMA dataset to drive an established head avatar. While dealing with novel expressions and poses obviously different from the training dataset, our approach shows significantly superior performance and robustness.

For the multi-view settings, as far as we know, there is no method focusing on sparse-views-based head avatar modeling available. To this end, similar to our extension to multiview scenario, we extend NeRFace-MV and NHA-MV by adopting multiview parametric face model tracking and optimizing the avatar according to multi-view image evidence.³ Compared with monocular data, multi-view observations can help model a more complete 3D head avatar, but also cause more obvious misalignment between the estimated mesh models and images due to the limited expressiveness of the parametric model, which raises more challenges for high-quality appearance recovery. Figure 11 illustrates the qualitative results of two different views, which demonstrates that our method can achieve fine-grained expression control and generate a view-consistent appearance. NeRFace tends to produce view-inconsistent artifacts, and NHA struggles to describe the topology-varying parts like glasses. The numeric results in Table 2 show that our method achieves higher accuracy in both metrics. We present the monocular-based animation results in Figure 12, which demonstrates our better performance on 3D reenactment.

Synthetic-rendering-based condition. In this part, two modified baselines were implemented for this ablation study. The first one, named “ExprPlanes-NeRF,” replaces our synthetic-rendering-based condition with the implicit vector-based condition used in NeRFace, with all other things being equal. The second baseline, named “ExprMLP-NeRF,” further replaces the orthogonal-planes-based neural representation with a deep MLP backbone used in NeRFace. To evaluate the effectiveness of our synthetic-rendering-based volumetric representation, we optimized a head avatar on a monocular video dataset using these two variants, and the results are presented in Figure 13 and Table 3. Comparing “ExprPlanes-NeRF” and “ExprMLP-NeRF,” we found that only applying the orthogonal-planes representation does not significantly improve performance. However, by using synthetic renderings for explicit condition, our method contributes to more accurate expression control.

Image-to-image Translation Module. Results in Section 5.1 have proven that the image translation module effectively enhances the fine-level details. We implement other two baselines for the ablation study to separately validate the choice of the 2D neural rendering network and the strategy of joint training: (1) We replace the image translation network with the up-sample SR module used by Chan et al. [2022]; Niemeyer and Geiger [2021] and train the whole pipeline end-to-end. However, when attempting to train the network with adversarial loss functions, we empirically find it hard to maintain stable training. We argue that, without the encoder part, the up-sample SR module alone is not suitable for the person-specific dataset with insufficient diversity. Instead, \(\mathit {l}_1\) loss and perceptual loss are adopted in this experiment. (2) We independently train the image translation module with GAN loss

to super-resolution the rendered images from a frozen pretrained SynR-NeRF. Experiments are conducted on a multi-view sequence, using five views for training and leaving one view for evaluation. As shown in Figure 14 and Table 4, up-sample SR-based baseline fails to generate fine details. For separate training baseline, it operates primarily in image-space and introduces undesirable inconsistent artifacts when dealing with the complex distribution of multi-view images. Through end-to-end training the whole framework, our pipeline contributes to guaranteeing realistic detail generation performance.

Zero-posed Orthogonal Mesh Rendering. In our method, we render the 3D facial model orthogonally to create a canonical feature volume in zero pose for feature conditioning. In this part, we introduce a baseline method called “Posed Rendering,” which involves rendering the pose-dependent meshes to condition the orthogonal feature planes. The results, as shown in Figure 15 and Table 5, indicate that “Posed Rendering” performs worse on the testing set. We attribute this to the coupling of expressions and poses, which creates false correlations between the facial appearance and the face location in the renderings. Plane feature generators have to remember the potential diverse locations in the input image space, leading to reduced performance. In contrast, our method orthogonally renders zero-posed meshes, enabling the generators to concentrate on extracting expression-related information from the renderings and achieve fine-grained control.

Conditional Learnable Embeddings. One strength of our method is that we solve the expression-shape coupling issue presented in previous NeRF-based head avatar methods. We owe it to our strategy of modulating plane feature generators, as mentioned in Section 4.1.2. For evaluation, similar to Figure 6, we implement a baseline that the NeRF is conditioned in an auto-decoding fashion by inputting the learnable embeddings into the MLP decoder. To fully explore the expression-pose coupling issues, we fix the head pose and only transfer the expression to animate the avatar. The results are presented in the video. NeRFace and our modified baseline both show obvious jitter, while our method illustrates stable animation results and better appearance quality.

Pose Condition. As mentioned in Section 4.1.3, by introducing pose vectors into condition, our method is able to describe pose-related non-rigid deformation in the canonical space. Figure 16 illustrates two cases. In the first case, without considering pose, artifacts are visible in the side-view observation when the head turns around. Our method, which includes pose condition, eliminates these artifacts and enhances view-consistency. In the second case, we demonstrate the ability of our method to describe simple pose-related movements of hair.

EG3D [Chan et al. 2022] is a state-of-the-art powerful generative model for HD 3D portrait, hence, we compare it with our method to demonstrate the ability on novel view synthesis. After fitting the pretrained EG3D model with a single reference frame [Roich et al. 2021], we render the reconstructed 3D head in different views. As Figure 19 shows, our method models a more vivid head avatar and presents more convincing novel view synthesis, which benefits from the joint learning of the temporal observation of the person-specific video data. Besides, EG3D performs worse on poses that are rare in FFHQ dataset. As for geometry, as we only utilize monocular view observation and do not apply any depth or sigma regularization, the geometry of our head avatar is noisier than EG3D’s result.

To validate the effectiveness of our adopted synthetic-rendering-based condition, we implement a mesh conditioned baseline (UV-NeRF) that encodes the feature map defined based on the UV parameterization⁴ instead of our orthogonal plane features. For each sample point, the local feature, obtained from the nearest surface vertices, serves as the input to the MLP decoder. This baseline network is trained under the same setting as our network. The experiment is conducted based on a monocular dataset, and the results are presented in Figure 20. Not surprisingly, though UV-NeRF can accurately reconstruct the expression and reproduce reliable facial appearance, it generates unrealistic artifacts at the edge region of the model mesh. Notice the ambiguous appearance in the mouth and the sharp seam around the neck. Our synthetic-rendering-based condition fully utilizes the powerful convolutional network to learn the reasonable correspondence between the facial model and the entire head appearance, synthesizing consistent and stable images.

To further validate our synthetic-rendering-based condition in detail, we implement two baselines that generate the feature volume directly from latent codes. The first one, called “VectorPlane,” uses the expression parameters as input for the feature generation backbone, while the second one, “VectorPlane(ExprMod),” feeds the expression parameters to a mapping network to modulate the convolutional kernels of the networks. Both baselines modulate the feature generation backbone with pose vectors and per-frame latent codes. As shown in Figure 21 and Table 6, learning the mapping from the global vectors to appearances tends to overfit training sequences, and the ability of expression control degraded for out-of-distribution expressions.

We present our analysis the differences between the two conditional methods. Two similar expressions may be too close in the parameter space for a global-vector-conditioned feature generator to distinguish between them. Our synthetic-rendering-based conditional method preserves the spatial alignment between the mesh renderings and the feature planes, hence the local spatial changes in the rendering image space can be reflected in the feature volume. Additionally, as we align individual local features at the pixel level of renderings to the global context of the entire appearance, it is more likely to infer plausible results for unseen expressions.

Besides, we also implement a baseline, named “woTexture,” that only utilizes normal and mask renderings to condition feature volume generation. Removing texture rendering is feasible for person-specific avatar modeling and does not significantly affect the robustness of expression control. However, despite the similar numerical results, the visualization results of our method exhibit more detailed appearances around the eyes. We hypothesize that the texture rendering contains more high-frequency information around the eye region, as shown in the last row of Figure 21, which may facilitate the network in effectively learning dynamic appearance around eyes.

By omitting the Nerf module, we implement a 2D re-enactment method that only utilizes our image2image translation module. In our practice, the image2image translation network takes the renderings of the fitted 3DMM in the observed view as input and generates the corresponding 2D images. This 2D re-enactment method has limitations. First, it cannot establish an avatar based on a multi-view dataset, because it cannot differentiate between camera poses and head poses. Second, when applied to monocular videos, the 2D re-enactment method is sensitive to the location of the facial model in the image space, as shown in Figure 22. The 2D re-enactment method is prone to generating artifacts or distorted faces, particularly at the edges of the image.

While fitting per-frame 3DMM model, we use the detected landmarks in multi-view images at the same instant as supervision and additionally estimate the scale parameter of 3DMM. As for NeRF optimization, we simply leverage multi-view images of the same frame to supervise the appearance with all loss terms and the training strategy as same as monocular-based setting.

Table 7 shows the detailed time-consuming during inference. Rendering a color image with a resolution of 512 \(\times\) 512 takes 0.15 second on one NVIDIA 3090 GPU, and the most time-intensive part is the volume rendering of our NeRF module.