summaries: This paper proposes a pipeline to induce geometrical deformations due to expressions on a person’s 3D face. The existing works either uses single image based 3D reconstruction that manifest the expression present in image or learn a neural representation of the deformation (latent code) which can be used to transfer the deformation due to expression from one face to other. This paper uses direct expression parameters and action units instead of a latent code and predict the deformation on 3D face for an expression instead of manifesting it from image (unlike reconstruction methods). To this end, the authors use an existing robust single image based 3D face deformation method (Chen et al.) to initialise the texture map and geometry as a displacement map of a 3D face (using Basel Face Model) from an image.  Because there is no ground truth of the new expression, they train it using adversarial loss. They produce the deformation using the Action Units(AU) as input and tries to produce those AU from the output displacement map. For rendering, the method uses the Neural Texture which is trained with the texture map produced by the Chen et al. The rendering has two components. One is the coarse level rendering which utilises the shape and expression parameters of the Basel Face model and the detail rendering consisting of the information of the displacement map due to the AU obtained through this method. Both are combined to get the final appearance.  They compared the result with DECA (Siggraph 21) which is a FLAME 3D model based method and shows that the identity, shape and expression rendering is better in the proposed method.  They propose FaceDet3D to generate facial expressions with details. To do this, they propose a Detail Hallucination Network to generate the target detail map from the source detail map along with Expression Adversarial Loss [1] and Superresolution Losses. Then they propose a Rendering Network. In addition to Photometric Loss and Expression Adversarial Loss [1],  they add Augmented Wrinkle Loss and Detailed Shading Loss. The result is an image with target expression containing details (mainly wrinkles), comparison with DECA (a previous work) shows they improve FID and FaceID by large margin.  [1] Conditional Image Synthesis With Auxiliary Classifier GANs  The paper proposes to improve the quality of the 3D reconstructed faces by taking into account the facial expression. Using a set of generative losses some of the details are recovered or hallucinated. The visual examples provided confirm the improvements, however better numerical evaluations and comparisons are required. 1. This paper proposes a new method FaceDet3D that can generate geometric facial details from a single image given any desired target expression. The method contains two components: first pass the input image to the Detail Hallucination Network to infer the geometric facial details of the subject as there expression changes; then a rendering network is used to generate the detailed rendering result using the 3D face geometry information.  2. The two component models are trained separately using a large scale in-the-wild images and a small video dataset. During the training of the rendering network, the novel Augmented Wrinkle Loss and Detailed Shading Loss were used.  3. The authors further evaluated the method and compared it to DECA to show that the proposed method could generate plausible facial details for any desired expressions and could render the result photo-realistically. 4. Ablation studies were also done to prove the effectiveness of the components in the proposed method. 
abstractText: Facial Expressions induce a variety of high-level details on the 3D face geometry. For example, a smile causes the wrinkling of cheeks or the formation of dimples, while being angry often causes wrinkling of the forehead. Morphable Models (3DMMs) of the human face fail to capture such fine details in their PCA-based representations and consequently cannot generate such details when used to edit expressions. In this work, we introduce FaceDet3D, a method that generates from a single image geometric facial details that are consistent with any desired target expression. The facial details are represented as a vertex displacement map and used then by a Neural Renderer to photo-realistically render novel images of any single image in any desired expression and view. Proxy Shading Input Reconstruction Happy Predicted Facial Detail Reconstructed Facial Geometry Detailed Shading Render of Reconstructed Facial Geometry Render of Predicted Facial Detail Figure 1: Facial detail hallucination and rendering. Given a single input image, a target expression (in this case ‘Happy’), and an initial detailed geometry extracted using FDS (Chen et al., 2019) (shown in column ‘Reconstruction’ and row ‘Detailed Shading’) as input, our method hallucinates facial geometric details consistent with the target expression. The hallucinated details are added to the smooth proxy geometry (marked as ‘Proxy Shading’, also extracted using FDS (Chen et al., 2019)), to give a detailed geometry with facial details consistent with the target expression (in column ‘Happy’ and row ‘Detailed Shading’). The detailed geometry is then rendered using Neural Rendering to give the final image (first row of the column labelled ‘Happy’). A zoom-in of one of the predicted facial details and its render is shown in column ‘Predicted Facial Detail’ and ‘Render of Predicted Facial Detail’ respectively. (Electronic zoom recommended)
references.authors: Niki Aifanti Christos Papachristou Anastasios Delopoulos
references.title: The mug facial expression database
references.venue: WIAMIS, pp. 1–4,
references.authors: S Athar Z Shu D Samaras
references.title: Self-supervised deformation modeling for facial expression editing
references.venue: In IEEE FG,
references.authors: Volker Blanz Thomas Vetter
references.title: A morphable model for the synthesis of 3d faces
references.authors: James Booth Epameinondas Antonakos Stylianos Ploumpis George Trigeorgis Yannis Panagakis Stefanos Zafeiriou
references.title: 3d face morphable models" in-the-wild
references.venue: In CVPR. IEEE,
references.authors: Anpei Chen Zhang Chen Guli Zhang Kenny Mitchell Jingyi Yu
references.title: Photo-realistic facial details synthesis from single image
references.venue: In ICCV, 2019
references.authors: Yunjey Choi Minje Choi Munyoung Kim Jung-Woo Ha Sunghun Kim Jaegul Choo
references.title: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation
references.venue: In CVPR,
references.authors: Yunjey Choi Youngjung Uh Jaejun Yoo Jung-Woo Ha
references.title: Stargan v2: Diverse image synthesis for multiple domains
references.venue: In CVPR,
references.authors: Pengfei Dou Shishir K Shah Ioannis A Kakadiaris
references.title: End-to-end 3d face reconstruction with deep neural networks
references.venue: In CVPR,
references.authors: P. Ekman W Friesen
references.title: Facial action coding system: A technique for the measurement of facial movement
references.authors: Yao Feng Haiwen Feng Michael J. Black Timo Bolkart
references.title: Learning an animatable detailed 3D face model from in-the-wild images. 2021
references.venue: URL https://doi.org/10.1145/3450626.3459936
references.authors: Kyle Genova Forrester Cole Aaron Maschinot Aaron Sarna Daniel Vlasic William T. Freeman
references.title: Unsupervised training for 3d morphable model regression
references.venue: In CVPR,
references.authors: Thomas Gerig Andreas Morel-Forster Clemens Blumer Bernhard Egger Marcel Luthi Sandro Schönborn Thomas Vetter
references.title: Morphable face models-an open framework
references.venue: In IEEE FG,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In NeurIPS,
references.authors: Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Sepp Hochreiter
references.title: Gans trained by a two time-scale update rule converge to a local nash equilibrium
references.venue: NeurIPS,
references.authors: Xun Huang Serge Belongie
references.title: Arbitrary style transfer in real-time with adaptive instance normalization
references.venue: In ICCV,
references.authors: Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A Efros
references.title: Image-to-image translation with conditional adversarial networks
references.venue: In CVPR,
references.authors: Aaron S Jackson Adrian Bulat Vasileios Argyriou Georgios Tzimiropoulos
references.title: Large pose 3d face reconstruction from a single image via direct volumetric cnn regression
references.venue: In ICCV,
references.authors: Tero Karras Samuli Laine Timo Aila
references.title: A style-based generator architecture for generative adversarial networks
references.venue: In CVPR,
references.authors: Hyeongwoo Kim Michael Zollhöfer Ayush Tewari Justus Thies Christian Richardt Christian Theobalt
references.title: Inversefacenet: Deep monocular inverse face rendering
references.venue: In CVPR,
references.authors: Haibin Ling Kazunori Okada
references.title: Diffusion distance for histogram comparison
references.venue: In CVPR,
references.authors: Lars Mescheder Andreas Geiger Sebastian Nowozin
references.title: Which training methods for gans do actually converge? 2018
references.authors: Albert Pumarola Antonio Agudo Aleix M Martinez Alberto Sanfeliu Francesc Moreno-Noguer
references.title: Ganimation: One-shot anatomically consistent facial animation
references.venue: International Journal of Computer Vision,
references.authors: Elad Richardson Matan Sela Roy Or-El R. Kimmel
references.title: Face reconstruction from a single image
references.venue: In CVPR,
references.authors: Edgar Schonfeld Bernt Schiele Anna Khoreva
references.title: A u-net based discriminator for generative adversarial networks
references.venue: In CVPR,
references.authors: Florian Schroff Dmitry Kalenichenko James Philbin
references.title: Facenet: A unified embedding for face recognition and clustering
references.venue: In CVPR,
references.authors: Matan Sela Elad Richardson Ron Kimmel
references.title: Unrestricted facial geometry reconstruction using image-toimage translation
references.venue: In ICCV,
references.authors: Z. Shu E. Yumer S. Hadap K. Sunkavalli E. Shechtman D. Samaras
references.title: Neural face editing with intrinsic image disentangling
references.venue: In CVPR,
references.authors: Ayush Tewari Michael Zollhöfer Pablo Garrido Florian Bernard Hyeongwoo Kim Patrick Pérez Christian Theobalt
references.title: Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz
references.venue: In CVPR,
references.authors: Ayush Tewari Florian Bernard Pablo Garrido Gaurav Bharaj Mohamed Elgharib Hans-Peter Seidel Patrick Pérez Michael Zöllhofer Christian Theobalt
references.title: Fml: Face model learning from videos
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Justus Thies M. Zollhöfer M. Nießner
references.title: Deferred neural rendering
references.venue: ACM Transactions on Graphics (TOG), 2019
references.authors: Luan Tran Xiaoming Liu
references.title: Nonlinear 3d face morphable model
references.venue: In IEEE Computer Vision and Pattern Recognition (CVPR), Salt Lake City,
references.authors: Luan Tran Feng Liu Xiaoming Liu
references.title: Towards high-fidelity nonlinear 3d face morphable model
references.venue: Proceeding of IEEE Computer Vision and Pattern Recognition,
references.authors: Anh Tun Trn Tal Hassner Iacopo Masi Eran Paz Yuval Nirkin Gérard Medioni
references.title: Extreme 3d face reconstruction: Seeing through occlusions
references.venue: In CVPR,
references.authors: Job Van Der Schalk Skyler T Hawk Agneta H Fischer Bertjan Doosje
references.title: Moving faces, looking places: validation of the amsterdam dynamic facial expression set (adfes)
references.venue: Emotion, 11(4):907,
references.authors: Yulun Zhang Kunpeng Li Kai Li Lichen Wang Bineng Zhong Yun Fu
references.title: Image super-resolution using very deep residual channel attention networks
references.venue: In ECCV,
references.authors: Jun-Yan Zhu Taesung Park Phillip Isola Alexei A Efros
references.title: Unpaired image-to-image translation using cycle-consistent adversarial networks
references.venue: In ICCV,
references.authors: Wenbin Zhu HsiangTao Wu Zeyu Chen Noranart Vesdapunt Baoyuan Wang
references.title: Reda:reinforced differentiable attribute for 3d face reconstruction
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
sections.heading: 1 INTRODUCTION
sections.text: Modelling the geometry of the human face continues to attract great interest in the computer vision and computer graphics communities. Strong PCA-based priors make 3D morphable models (3DMMs) (Blanz et al., 1999) robust, but at the same time over-regularize them. Thus, they fail to capture fine facial details, such as the wrinkles on the forehead when the eyebrows are raised or bumps on the cheeks when one smiles. Additionally, the lack of diversity in the texture space of most available 3DMMs makes it very hard to generate realistic renderings that capture the large variations of color and texture we observe in human faces. Recent methods (Tewari et al., 2019; 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020; Booth et al., 2017; Dou et al., 2017; Jackson et al., 2017; Kim et al., 2018; Genova et al., 2018) address this by learning richer shape and expression spaces using a variety of data modalities such as in-the-wild images (Tewari et al., 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020) and videos (Tewari et al., 2019). However, despite using more expressive shape and expression spaces, these models still fail to capture fine details in geometry.
  Recent methods that accurately estimate facial geometric details from single images (Chen et al., 2019; Tun Trn et al., 2018; Feng et al., 2021), while being unable to hallucinate and photo-realistically render novel details under expression change, have nonetheless enabled the large scale annotation of unpaired image data with facial geometric details. Thus, it is now possible to train facial detail hallucination methods using unsupervised adversarial losses (Choi et al., 2018; Pumarola et al., 2020). Similarly, Neural Rendering (Thies et al., 2019) has made it possible to render 3D geometries with photo-realism via the use of high-dimensional Neural Texture Maps (NTMs). However, NTMs are able to store fine details of the output image, causing the rendered details to be completely independent of the geometric details. They do not change even if the geometric details do, making neural rendering unsuitable for rendering facial geometric details. In this paper we introduce FaceDet3D, a generative model that hallucinates facial geometric details for any target expression and renders them realistically. The model is made up of two components: 1) A detail hallucination network that infers plausible geometric facial details of a person as their expression changes, from a single image of that person. 2) A rendering network that overcomes the aforementioned shortcoming of neural rendering and explicitly conditions the rendered facial details on the details of the 3D face geometry. Thus, the rendered details change only when the facial geometric details change. This conditioning is achieved through the use of the novel Augmented Wrinkle Loss and the Detailed Shading Loss during training. Our method is trained using only a large scale in-the-wild image dataset and a much smaller video dataset captured in controlled conditions, without any 3D data as supervision for the target expression geometry. An exhaustive evaluation shows that once trained, our method is able to generate plausible facial details for any desired anatomically consistent facial expression and render it photo-realistically.
sections.heading: 2 RELATED WORK
sections.text: We next describe the most related methods in facial geometry estimation, geometric facial detail estimation and facial expression editing and reanimation.
  Geometric Facial Details Estimation. Over the past few years there has been significant improvement in the realism of 3D face geometries estimated from single images. In Richardson et al. (2017), a CNN (CoarseNet) first regresses a rough geometry of the face, facial details are then estimated using another CNN (FineNet) using the coarse depth map and input images. In Sela et al. (2017), the regressed correspondence and depth maps are registered onto a template mesh, which is further refined to generate the detailed facial geometry. In Tun Trn et al. (2018) facial details are modelled with bump maps on top of a 3DMM base. Similarly, in Facial Details Synthesis (FDS)(Chen et al., 2019), details are represented as vertex displacements of an underlying 3DMM, trained using a combination of 3D data and in-the-wild images. These methods, however, can only estimate the facial details of the expression manifested in an image, but cannot predict novel facial details for a different expression, which is the motivation of our method. Geometric Facial Details Animation. In DECA (Feng et al., 2021), the authors train a network to regress the detailed geometry using a latent detail code. This code enables transfer of details from a target image to the source image. In contrast, the details generated by our method are directly conditioned on expression parameters and action units (Ekman & Friesen, 1978). Further, unlike our method, DECA (Feng et al., 2021) is unable photorealistically render the geometric details, leading to sub-par animation.
  Facial Expression Editing. The success of image-to-image translation networks (Isola et al., 2017), adversarial training (Goodfellow et al., 2014) and cycle-consistency losses (Zhu et al., 2017) have led to novel expression editing methods that use large-scale in-the-wild training datasets. In (Shu et al., 2017), an unsupervised face model disentangles the input face into albedo, normals and shading. Expressions are then edited by traversals in the disentangled latent space. In Choi et al. (2018), expressions are edited via adversarial losses coupled with cycle consistency. In (Pumarola et al., 2020), a network edits input images to target expressions, represented as Action Units (Ekman & Friesen, 1978). Athar et al. (2020) extends this work by explicitly modeling skin motion followed by texture hallucination. In Choi et al. (2020), template images are used for editing. While these methods give photo-realistic results, they are restricted to the 2D image space and cannot be used to manipulate 3D viewpoint.
  3 FACEDET3D
  3.1 PROBLEM FORMULATION
  Let Ix ∈ RH×W×3 be a face image with an expression x represented by Action Units (Ekman & Friesen, 1978). Its shape and expression parameters in the Basel Face Model (BFM) (Gerig et al., 2018) space are {αs,αe}. A vertex displacement UV map D(Ix) ∈ RHD×WD×3 encodes the facial details of the person shown in Ix with the expression x. We want to: (1) Generate facial details, represented by a vertex displacement map D(Iy), of the same person with expression y; (2) Render an image of the geometry with the generated facial geometric details. We use FDS (Chen et al., 2019) to extract the texture map, initial detail map and geometry of the input image Ix:
  T (Ix),D(Ix),αs,αe ← FDS(Ix) , (1) where T (Ix) ∈ RHT ×WT×3 is the texture map. Next, we use the detail hallucination network, DetH (·), to hallucinate the plausible facial detail map of the person in Ix for expression y, conditioned on D(Ix), as follows:
  D̃(Iy) = DetH (D(Ix),x,y,Age(Ix),FaceID(Ix)) , (2)
  where y is the target AU, x is the input AU, Age(Ix) are features extracted from an age prediction network and FaceID(Ix) is the facial embedding of Ix extracted using (Schroff et al., 2015). Note that we do not have access to the ground truth image Iy, thus we hallucinate a plausible detail map of Iy (i.e D̃(Iy)) using DetH . Once we have D̃(Iy), we use it to displace the vertices along their normal direction giving us the detailed geometry. Finally, we render this detailed face geometry using a rendering network R (·):
  Ĩy = R ( T (Ix), D̃(Iy),αs, α̂e,y, c, l, γ ) , (3)
  where α̂e are the target expression parameters, c are the desired camera parameters, γ is the albedo PCA-space parameters of BFM(Gerig et al., 2018), and l are the lighting parameters.
sections.heading: 3.2 DETAIL HALLUCINATION
sections.text: DetH takes the input detail map,D(Ix), the input expression AU x, the shape parameters αs, the face embedding FaceID(Ix), and the age features Age(Ix) and extracts features from each of them. There features are concatenated in the channel dimension and passed though the Decoder which receives the target action unit y via Adaptive Instance Normalization (Huang & Belongie, 2017) layers. The Decoder gives as output a hallucination D(Iy)H and a mask D(Iy)M which are combined together with the input detail map, D(Ix), to give D̃(Iy) as follows:
  D̃(Iy)=D(Iy)M D(Iy)H+(19D(Iy)M ) D(Ix) . (4)
  The masking mechanism ensures the preservation of the details that are not meant to be changed with the expression. A schematic of the network is shown in Fig 2.
sections.heading: 3.2.1 TRAINING LOSSES
sections.text: Due to the lack of a large scale in-the-wild dataset of paired data with expression change or 3D data, we cannot perform regression using ground-truth geometric facial details, D(Iy) of the image Iy. Therefore, we instead use an adversarial training paradigm along with cycle-consistency losses similar to (Athar et al., 2020; Pumarola et al., 2020) to hallucinate the plausible facial geometric details D̃(Iy) and to ensure the hallucination’s fidelity to the target expression and input features. In order to speed up convergence, we weakly supervise the adversarial training using randomly sampled frames from videos of the MUG (Aifanti et al., 2010) and the ADFES datasets (Van Der Schalk et al.,
  2011), ensuring that the sampling is sparse enough such that there is significant change in expression with frames sampled from each video. We leave the full exposition of all the standard losses and regularizations to the supplementary.
  Expression Adversarial Loss. In order to ensure the hallucinated facial geometric details, D̃(Iy), are consistent with the target expression y, as encoded by AUs, we use an expression discriminator DExp. Given D(Ix) of some image Ix manifesting expression x, DExp, outputs the following
  DExp(D(Ix)) = {r, x̂} , (5)
  where r is a realism score and x̂ is the predicted AU. For brevity, we will use DExp(D(Ix)) and DExp(D) interchangeably. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the R1 gradient penalty (Mescheder et al., 2018) to train DExp. We use a UNet based discriminator (Schonfeld et al., 2020) in order to discriminate on pixel level. In addition, DExp is trained to minimize the error of the predicted AU
  LDExpAU = ED∼PD [ ||[DAUExp(D) 9 x||22 ] , (6)
  where DAUExp is the AU output head of DExp. The Details Hallucination Network, DetH , in addition to be trained to minimize adversarial loss, is also trained to minimize the expression loss:
  LDetHAU = E Ix,{y} ||DAUExp(DetH (·)) 9 y||22 (7)
  where DetH (·) is to be read as in Eq. (2) and y is the target AU. Superresolution Losses. The detail maps generated by FDS (Chen et al., 2019) are of resolution 4096× 4096 and thus cannot be used directly for training due to GPU-memory constraints. To get around this, we train DetH on detail maps downsampled to 256× 256. Simultaneously, we finetune a superresolution network, RCAN (Zhang et al., 2018), to super-resolve downsampled 256 × 256 patches of D(Ix) by a factor of 4
  LRCANSR = L1 ( RCAN ( D(Ix)P256 ) ,D(Ix)P1024 ) , (8)
  where D(Ix)P1024 is a randomly sampled patch of resolution 1024 × 1024 from the full-resolution detail map D(Ix) and D(Ix)P256 is its downsampled version. During inference, we use RCAN twice on the predicted detail map D̃(Iy) to upsample it to 4096× 4096:
  D̃(Iy)HR = RCAN ( RCAN ( D̃(Iy) )) . (9)
  In the interest of brevity, we will use D̃(Iy) in lieu of D̃(Iy)HR in the remainder of this text.
sections.heading: 3.3 RENDERING NETWORK
sections.text: The rendering network, R , consists of two subnetworks: (1) The Neural Texture prediction network NTex φ and (2) The image rendering network Tex2Im . Neural Texture Prediction. NTex φ predicts the Neural Textures given the texture map T (Ix) of image Ix:
  NTM = NTex φ(T (Ix)); NTM ∈ R F×H×W , (10)
  where NTM is the predicted neural texture map with F channels. Unlike in (Thies et al., 2019), where the NTM is identity specific, NTex φ can be used on any T (Ix), regardless of identitym to generate its corresponding neural texture map. Through training, NTex φ learns to extract the appropriate high-dimensional texture features from T (Ix) such that NTM can be used to generate a realistic render of the person in Ix in any desired expression and view. Image rendering. The image rendering network, Tex2Im , consists of two branches, the low-res image renderer gθ and the detail renderer gω . The low-res image renderer generates identity-specific image textures that are invariant to the predicted geometric details, such as the skin-tone or eye color. The detail renderer gω renders the facial geometric details obtained from the detail hallucination network, DetH , and adds them to the low-res image generated by gθ. The separation of the image rendering network into two branches allows each branch to focus on its respective task and leads to high-quality renderings. The low-res image renderer, gθ, takes as input the NTM sampled using the UV map rasterized by the proxy geometry, GP = {0 × D̃(Iy),αs, α̂e}, i.e the geometry without any details, along with the shaded albedo also rasterized by GP :
  ÎLRy = gθ (Sample(NTM,UV-Raster(GP , c)),y,γ , l) (11)
  where c are the camera parameters, y is the target AU, γ is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and l are the lighting parameters. Since gθ only uses inputs dependent on GP it generates the image textures that are invariant to details predicted by DetH .
  The detail renderer, gω , takes as input the NTM sampled using the UV map rasterized by the detailed geometry, GD = {D̃(Iy),αs, α̂e} along with the shaded albedo also rasterized by GD:
  ÎDy = gθ(Sample(NTM,UV-Raster(GD, c)),y, γ, l) (12)
  where c are the camera parameters, y is the target AU, γ is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and l are the lighting parameters. The final output image is calculated as:
  Îy = Î LR y + Î D y . (13)
  Since all of the detail invariant textures are already generated by gθ in ˆILRy , the detail renderer, gω, can solely focus on realistically rendering the details hallucinated by the details hallucination network DetH .
sections.heading: 3.3.1 TRAINING LOSSES
sections.text: The renderings generated by R must: 1) faithfully render the geometric details onto the RGB space and 2) be realistic. Neural Rendering (Thies et al., 2019) is designed to address 2) as the highdimensional neural texture map is able store the fine details of the output texture. Consequently, the details on the rendered image become entirely conditional on the input texture map, T (Ix), and ignore the detailed geometryGD. This significantly hurts 1) causing the details on the rendered image to remain unchanged even if the facial geometric details change due to changes in facial expression. In order to ensure the output renderings satisfy 2) we use, along with the branched architecture discussed in Sect 3.3, an Augmented Wrinkle Loss (AugW) and the Detailed Shading Loss (DSL) to ensure the geometric details are faithfully rendered onto the output image. Additionally, R is also trained with a photometric loss and an Expression Adversarial Loss in order to maintain photometric and expression consistency. We leave the full exposition of all the standard losses and regularizations to the supplementary. Augmented Wrinkle Loss (AugW). In order to enforce the rendering of geometric details onto the rendered image we add ‘fake’ wrinkles to an image Ix and force R to generate the same. Given the the detailed geometry of Ix, GD = {D(Ix),αs,αe}, a geometry with ‘fake’ details G∗D = {D(I∗z),αs,αe} using the geometric details from some random image I∗z and the lighting l of Ix, the ‘fake’ wrinkles are added as follows:
  Shading(Ix) = LSph(GD, l);Shading ∗(Ix) = LSph(G ∗ D, l)
  I∗ x = Shading∗(Ix)×
  ( Ix
  Shading(Ix)
  ) (14)
  where LSph is the spherical harmonic lighting function and l are the coefficients of the first 9 spherical harmonics. The artificially wrinkled image I∗
  x is now re-rendered using R
  Î∗x = R (T (Ix), G∗D,x, c, l) LAugW = LapLoss(Î∗x, I∗x) .
  (15)
  where, LapLoss is the Laplacian Loss (Ling & Okada, 2006). In order to faithfully reconstruct I∗ x , R is forced to rely on the detailed geometry G∗D, since the input texture map T (Ix), and consequently the neural texture map, contain no information about the ‘fake’ wrinkles. Detailed Shading Loss (DSL). In addition to the Augmented Wrinkle Loss, we also try to predict the shading of the detailed facial geometry from the output rendering Îx
  ˆShading(Îx) = fθ(Îx) LDSL = LapLoss( ˆShading(Îx),Shading∗(Ix)) , (16)
  where fθ is a small convolutional network (CNN) with only two layers and the shading Shading∗(Ix) is calculated as in Eq. (14). We calculate this loss only over the skin region. Since, fθ is a small CNN with limited representational capacity, the details must be quite visible on the rendered image Îx in order for them to be picked up by fθ to generate an accurate shading ˆShading(Îx). Expression Adversarial Loss. In order to ensure that the rendered output conforms to the target expression we use an expression adversarial loss. Given a rendered image, Îx = R (T (Ix), GD,x, c, l),
  manifesting the expression encoded by AU x an expression discriminator, DRGBExp , outputs
  DRGBExp (Îx) = {r, x̂} , (17)
  where r is a realism score and x̂ is the predicted AU. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the R1 gradient penalty (Mescheder et al., 2018) to train DExp. In addition, DRGBExp is trained to minimize the predicted AU error
  LD RGB Exp AU = EIx∼PI [ ||[DRGB,AUExp (Ix) 9 x|| 2 2 ] , (18)
  where DRGB,AUExp is the AU output head of D RGB Exp . The Rendering Network, R , in addition to be trained to minimize adversarial loss, is also trained to minimize the AU loss
  LRAU = EIx ||D RGB,AU Exp (R (·)) 9 y||22 , (19)
  where R (·) is to be read as in Eq. (3).
sections.heading: 4 RESULTS
sections.text: We train the detail hallucination network, DetH , and the rendering network R , on 9,000 images from the FFHQ dataset (Karras et al., 2019). Additionally, 3,000 frames were sampled from the MUG (Aifanti et al., 2010) and the ADFES (Van Der Schalk et al., 2011) datasets to speed-up training of the detail hallucination network DetH . Due to memory constraints, DetH and R are trained independently. Upon publication we will release the code. Below, we show the FaceDet3D’s results on expression change, results on view consistency, a comparison with DECA (Feng et al., 2021) and Ablation studies.
  Expression change. We first demonstrate the detail hallucination and rendering results with expression change. Fig 4 shows the result of changing the expression of some input image to a variety of expressions. The first column of Fig 4 is the input image, the second column shows the image reconstructed using the detail map D(Ix) predicted by FDS (Chen et al., 2019). In the subsequent columns D(Ix) is used as input to generate D̃(Iy) where y = {Anger,Disgust,Happy,Surprise}. The first row shows the final rendered image, i.e the output of R and the second row shows the shaded geometry with the hallucinated details, i.e GD. The hallucinated details and their corresponding rendering are marked out with dashed red and blue rectangles. We zoom in the red rectangles of the last column. As seen in the second row of Fig 4, the details hallucinated are consistent with the manifested expression. For example, ‘Anger’ and ‘Disgust’ (3rd and 4th column of Fig 4 respectively) show consistent wrinkling around the forehead and the nose while ‘Happy’ (column 6 Fig 4) shows consistent wrinkling around the cheeks. Zooming in the last column of Fig 4 highlights the realism of the hallucinated details produced by the rendering network, R . Further, in Fig 7 we show the utility of hallucinating details as the expression changes. The first row of Fig 7 shows the image rendered with
  the hallucinated details from DetH as the expression changes. The second row of images shows the images rendered without details, this is done by setting the hallucinated detail map to zero. As seen by comparing the skin appearance marked with the red rectangles, in the first row the skin changes realistically as the expression of the person changes due to the changing facial geometric details, while the skin in the second row remains unchanged, significantly hurting realism. In the supplementary we show further results on expression animation and encourage the reader to inspect them.
  Reconstruction Fear Happy
  Render without details
  Detailed Shading
  DisgustInput
  DECA. The row label is the expression being animated, the first column is the input image, the second is the detailed geometry hallucinated by FaceDet3D for the expression being animated, the
  3rd column is the render generated by FaceDet3D of its hallucinated detailed geometry, the fourth column is the detailed geometry generated by DECA for the expression being animated and the fifth column is render generated by DECA. Note that DECA’s results have the full head because they use the FLAME 3DMM while we use the BFM 3DMM. By comparing the details in the second and the fourth column of Fig 6 one can see that FaceDet3D generates more concentrated and higher quality details. For example, DECA always generates wrinkles on the forehead regardless of the expression being animated, thus the details generated for ‘Happy’ (row 1) are inconsistent with the expression. Similarly, a comparison between the third and fifth column of Fig 6 shows that the renders of FaceDet3D are significantly more realistic than those of DECA. In Table 1, we see that the FaceID distance (using FaceNet (Schroff et al., 2015)) and the FID score, across a variety of expression edits, of the renders of FaceDet3D is significantly lower than the renders of DECA. Further, the blue boxes shown in columns 2 and 3 of Fig 6 show that the hallucinated details are faithfully and photo-realistically reproduced in the render.
  Ablation Studies. We examine the utility of the Augmented Wrinkle Loss (AugW) and the Detailed Shading Loss (DSL) in rendering the facial geometric details to the image space. Fig 8 shows the results of training R with and without the AugW and DP losses. As seen by comparing the results in rows 2 and 3 of Fig 8, without those losses R fails to render the hallucinated geometric facial details. Next, we ablate using a single network in Tex2Im (‘NR Single Branch’) versus using two networks as described in Sect 3.3 (‘NR Double Branch’) versus using Ordinary Rendering using the provided texture space of BFM(Gerig et al., 2018). We calculate the FID score (Heusel et al., 2017), and the FaceID distance using FaceNet (Schroff et al., 2015) across a variety of expression edits. As can be seen in Table 2, using two networks, one that outputs the detail-invariant textures and the other that renders the details, gives better results as compared to using a single network. It not only produces more realistic images (as measured by the FID score) but also more closely preserves the identity across a variety of expression edits. As it is expected, ordinary rendering performs the worst as its
  PCA space cannot capture the rich details of the human face across a variety of input identities and expression edits.
sections.heading: 5 CONCLUSION AND FUTURE WORK
sections.text: We have presented FaceDet3D, a method for hallucinating, from a single image, plausible facial geometric details as the expression changes and render them photo-realistically. The details hallucination network is trained using adversarial losses and weak supervision without any ground-truth 3D data. The rendering network is constrained using the Augmented Wrinkle Loss and Detailed Shading Loss, forcing it to use cues from the detailed geometry to render the details, ensuring their fidelity and consistency across a variety of expressions and views. The detail hallucination network relies on the detail map estimated by FDS (Chen et al., 2019) as input to predict the plausible details of the target expression, therefore it cannot handle occlusions such as glasses or make-up very well as FDS (Chen et al., 2019) fails in those conditions. In future work, we plan to incorporate explicit disentanglement of lighting in order to enable greater control over the final rendering along with explicit modelling occlusions and joint training of DetH and R .