summaries: This paper proposes a pipeline to induce geometrical deformations due to expressions on a person’s 3D face. The existing works either uses single image based 3D reconstruction that manifest the expression present in image or learn a neural representation of the deformation (latent code) which can be used to transfer the deformation due to expression from one face to other. This paper uses direct expression parameters and action units instead of a latent code and predict the deformation on 3D face for an expression instead of manifesting it from image (unlike reconstruction methods). To this end, the authors use an existing robust single image based 3D face deformation method (Chen et al.) to initialise the texture map and geometry as a displacement map of a 3D face (using Basel Face Model) from an image.  Because there is no ground truth of the new expression, they train it using adversarial loss. They produce the deformation using the Action Units(AU) as input and tries to produce those AU from the output displacement map. For rendering, the method uses the Neural Texture which is trained with the texture map produced by the Chen et al. The rendering has two components. One is the coarse level rendering which utilises the shape and expression parameters of the Basel Face model and the detail rendering consisting of the information of the displacement map due to the AU obtained through this method. Both are combined to get the final appearance.  They compared the result with DECA (Siggraph 21) which is a FLAME 3D model based method and shows that the identity, shape and expression rendering is better in the proposed method.

summaries: They propose FaceDet3D to generate facial expressions with details. To do this, they propose a Detail Hallucination Network to generate the target detail map from the source detail map along with Expression Adversarial Loss [1] and Superresolution Losses. Then they propose a Rendering Network. In addition to Photometric Loss and Expression Adversarial Loss [1],  they add Augmented Wrinkle Loss and Detailed Shading Loss. The result is an image with target expression containing details (mainly wrinkles), comparison with DECA (a previous work) shows they improve FID and FaceID by large margin.  [1] Conditional Image Synthesis With Auxiliary Classifier GANs

summaries: The paper proposes to improve the quality of the 3D reconstructed faces by taking into account the facial expression. Using a set of generative losses some of the details are recovered or hallucinated. The visual examples provided confirm the improvements, however better numerical evaluations and comparisons are required.

summaries: 1. This paper proposes a new method FaceDet3D that can generate geometric facial details from a single image given any desired target expression. The method contains two components: first pass the input image to the Detail Hallucination Network to infer the geometric facial details of the subject as there expression changes; then a rendering network is used to generate the detailed rendering result using the 3D face geometry information.  2. The two component models are trained separately using a large scale in-the-wild images and a small video dataset. During the training of the rendering network, the novel Augmented Wrinkle Loss and Detailed Shading Loss were used.  3. The authors further evaluated the method and compared it to DECA to show that the proposed method could generate plausible facial details for any desired expressions and could render the result photo-realistically. 4. Ablation studies were also done to prove the effectiveness of the components in the proposed method.

abstractText: Facial Expressions induce a variety of high-level details on the 3D face geometry. For example, a smile causes the wrinkling of cheeks or the formation of dimples, while being angry often causes wrinkling of the forehead. Morphable Models (3DMMs) of the human face fail to capture such fine details in their PCA-based representations and consequently cannot generate such details when used to edit expressions. In this work, we introduce FaceDet3D, a method that generates from a single image geometric facial details that are consistent with any desired target expression. The facial details are represented as a vertex displacement map and used then by a Neural Renderer to photo-realistically render novel images of any single image in any desired expression and view. Proxy Shading Input Reconstruction Happy Predicted Facial Detail Reconstructed Facial Geometry Detailed Shading Render of Reconstructed Facial Geometry Render of Predicted Facial Detail Figure 1: Facial detail hallucination and rendering. Given a single input image, a target expression (in this case ‘Happy’), and an initial detailed geometry extracted using FDS (Chen et al., 2019) (shown in column ‘Reconstruction’ and row ‘Detailed Shading’) as input, our method hallucinates facial geometric details consistent with the target expression. The hallucinated details are added to the smooth proxy geometry (marked as ‘Proxy Shading’, also extracted using FDS (Chen et al., 2019)), to give a detailed geometry with facial details consistent with the target expression (in column ‘Happy’ and row ‘Detailed Shading’). The detailed geometry is then rendered using Neural Rendering to give the final image (first row of the column labelled ‘Happy’). A zoom-in of one of the predicted facial details and its render is shown in column ‘Predicted Facial Detail’ and ‘Render of Predicted Facial Detail’ respectively. (Electronic zoom recommended)

authors:
Niki Aifanti
Christos Papachristou
Anastasios Delopoulos

title: The mug facial expression database

venue: WIAMIS, pp. 1–4,

authors:
S Athar
Z Shu
D Samaras

title: Self-supervised deformation modeling for facial expression editing

venue: In IEEE FG,

authors:
Volker Blanz
Thomas Vetter

title: A morphable model for the synthesis of 3d faces

authors:
James Booth
Epameinondas Antonakos
Stylianos Ploumpis
George Trigeorgis
Yannis Panagakis
Stefanos Zafeiriou

title: 3d face morphable models" in-the-wild

venue: In CVPR. IEEE,

authors:
Anpei Chen
Zhang Chen
Guli Zhang
Kenny Mitchell
Jingyi Yu

title: Photo-realistic facial details synthesis from single image

venue: In ICCV, 2019

authors:
Yunjey Choi
Minje Choi
Munyoung Kim
Jung-Woo Ha
Sunghun Kim
Jaegul Choo

title: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation

venue: In CVPR,

authors:
Yunjey Choi
Youngjung Uh
Jaejun Yoo
Jung-Woo Ha

title: Stargan v2: Diverse image synthesis for multiple domains

authors:
Pengfei Dou
Shishir K Shah
Ioannis A Kakadiaris

title: End-to-end 3d face reconstruction with deep neural networks

authors:
P. Ekman
W Friesen

title: Facial action coding system: A technique for the measurement of facial movement

authors:
Yao Feng
Haiwen Feng
Michael J. Black
Timo Bolkart

title: Learning an animatable detailed 3D face model from in-the-wild images. 2021

authors:
Kyle Genova
Forrester Cole
Aaron Maschinot
Aaron Sarna
Daniel Vlasic
William T. Freeman

title: Unsupervised training for 3d morphable model regression

authors:
Thomas Gerig
Andreas Morel-Forster
Clemens Blumer
Bernhard Egger
Marcel Luthi
Sandro Schönborn
Thomas Vetter

title: Morphable face models-an open framework

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In NeurIPS,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

venue: NeurIPS,

authors:
Xun Huang
Serge Belongie

title: Arbitrary style transfer in real-time with adaptive instance normalization

venue: In ICCV,

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A Efros

title: Image-to-image translation with conditional adversarial networks

authors:
Aaron S Jackson
Adrian Bulat
Vasileios Argyriou
Georgios Tzimiropoulos

title: Large pose 3d face reconstruction from a single image via direct volumetric cnn regression

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

authors:
Hyeongwoo Kim
Michael Zollhöfer
Ayush Tewari
Justus Thies
Christian Richardt
Christian Theobalt

title: Inversefacenet: Deep monocular inverse face rendering

authors:
Haibin Ling
Kazunori Okada

title: Diffusion distance for histogram comparison

authors:
Lars Mescheder
Andreas Geiger
Sebastian Nowozin

title: Which training methods for gans do actually converge? 2018

authors:
Albert Pumarola
Antonio Agudo
Aleix M Martinez
Alberto Sanfeliu
Francesc Moreno-Noguer

title: Ganimation: One-shot anatomically consistent facial animation

venue: International Journal of Computer Vision,

authors:
Elad Richardson
Matan Sela
Roy Or-El
R. Kimmel

title: Face reconstruction from a single image

authors:
Edgar Schonfeld
Bernt Schiele
Anna Khoreva

title: A u-net based discriminator for generative adversarial networks

authors:
Florian Schroff
Dmitry Kalenichenko
James Philbin

title: Facenet: A unified embedding for face recognition and clustering

authors:
Matan Sela
Elad Richardson
Ron Kimmel

title: Unrestricted facial geometry reconstruction using image-toimage translation

authors:
Z. Shu
E. Yumer
S. Hadap
K. Sunkavalli
E. Shechtman
D. Samaras

title: Neural face editing with intrinsic image disentangling

authors:
Ayush Tewari
Michael Zollhöfer
Pablo Garrido
Florian Bernard
Hyeongwoo Kim
Patrick Pérez
Christian Theobalt

title: Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz

authors:
Ayush Tewari
Florian Bernard
Pablo Garrido
Gaurav Bharaj
Mohamed Elgharib
Hans-Peter Seidel
Patrick Pérez
Michael Zöllhofer
Christian Theobalt

title: Fml: Face model learning from videos

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Justus Thies
M. Zollhöfer
M. Nießner

title: Deferred neural rendering

venue: ACM Transactions on Graphics (TOG), 2019

authors:
Luan Tran
Xiaoming Liu

title: Nonlinear 3d face morphable model

venue: In IEEE Computer Vision and Pattern Recognition (CVPR), Salt Lake City,

authors:
Luan Tran
Feng Liu
Xiaoming Liu

title: Towards high-fidelity nonlinear 3d face morphable model

venue: Proceeding of IEEE Computer Vision and Pattern Recognition,

authors:
Anh Tun Trn
Tal Hassner
Iacopo Masi
Eran Paz
Yuval Nirkin
Gérard Medioni

title: Extreme 3d face reconstruction: Seeing through occlusions

authors:
Job Van Der Schalk
Skyler T Hawk
Agneta H Fischer
Bertjan Doosje

title: Moving faces, looking places: validation of the amsterdam dynamic facial expression set (adfes)

authors:
Yulun Zhang
Kunpeng Li
Kai Li
Lichen Wang
Bineng Zhong
Yun Fu

title: Image super-resolution using very deep residual channel attention networks

venue: In ECCV,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

authors:
Wenbin Zhu
HsiangTao Wu
Zeyu Chen
Noranart Vesdapunt
Baoyuan Wang

title: Reda:reinforced differentiable attribute for 3d face reconstruction

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

heading: 1 INTRODUCTION

text:
Modelling the geometry of the human face continues to attract great interest in the computer vision and computer graphics communities. Strong PCA-based priors make 3D morphable models (3DMMs) (Blanz et al., 1999) robust, but at the same time over-regularize them. Thus, they fail to capture fine facial details, such as the wrinkles on the forehead when the eyebrows are raised or bumps on the cheeks when one smiles. Additionally, the lack of diversity in the texture space of most available 3DMMs makes it very hard to generate realistic renderings that capture the large variations of color and texture we observe in human faces. Recent methods (Tewari et al., 2019; 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020; Booth et al., 2017; Dou et al., 2017; Jackson et al., 2017; Kim et al., 2018; Genova et al., 2018) address this by learning richer shape and expression spaces using a variety of data modalities such as in-the-wild images (Tewari et al., 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020) and videos (Tewari et al., 2019). However, despite using more expressive shape and expression spaces, these models still fail to capture fine details in geometry.
Recent methods that accurately estimate facial geometric details from single images (Chen et al., 2019; Tun Trn et al., 2018; Feng et al., 2021), while being unable to hallucinate and photo-realistically render novel details under expression change, have nonetheless enabled the large scale annotation of unpaired image data with facial geometric details. Thus, it is now possible to train facial detail hallucination methods using unsupervised adversarial losses (Choi et al., 2018; Pumarola et al., 2020). Similarly, Neural Rendering (Thies et al., 2019) has made it possible to render 3D geometries with photo-realism via the use of high-dimensional Neural Texture Maps (NTMs). However, NTMs are able to store fine details of the output image, causing the rendered details to be completely independent of the geometric details. They do not change even if the geometric details do, making neural rendering unsuitable for rendering facial geometric details. In this paper we introduce FaceDet3D, a generative model that hallucinates facial geometric details for any target expression and renders them realistically. The model is made up of two components: 1) A detail hallucination network that infers plausible geometric facial details of a person as their expression changes, from a single image of that person. 2) A rendering network that overcomes the aforementioned shortcoming of neural rendering and explicitly conditions the rendered facial details on the details of the 3D face geometry. Thus, the rendered details change only when the facial geometric details change. This conditioning is achieved through the use of the novel Augmented Wrinkle Loss and the Detailed Shading Loss during training. Our method is trained using only a large scale in-the-wild image dataset and a much smaller video dataset captured in controlled conditions, without any 3D data as supervision for the target expression geometry. An exhaustive evaluation shows that once trained, our method is able to generate plausible facial details for any desired anatomically consistent facial expression and render it photo-realistically.

heading: 2 RELATED WORK

text:
We next describe the most related methods in facial geometry estimation, geometric facial detail estimation and facial expression editing and reanimation.
Geometric Facial Details Estimation. Over the past few years there has been significant improvement in the realism of 3D face geometries estimated from single images. In Richardson et al. (2017), a CNN (CoarseNet) first regresses a rough geometry of the face, facial details are then estimated using another CNN (FineNet) using the coarse depth map and input images. In Sela et al. (2017), the regressed correspondence and depth maps are registered onto a template mesh, which is further refined to generate the detailed facial geometry. In Tun Trn et al. (2018) facial details are modelled with bump maps on top of a 3DMM base. Similarly, in Facial Details Synthesis (FDS)(Chen et al., 2019), details are represented as vertex displacements of an underlying 3DMM, trained using a combination of 3D data and in-the-wild images. These methods, however, can only estimate the facial details of the expression manifested in an image, but cannot predict novel facial details for a different expression, which is the motivation of our method. Geometric Facial Details Animation. In DECA (Feng et al., 2021), the authors train a network to regress the detailed geometry using a latent detail code. This code enables transfer of details from a target image to the source image. In contrast, the details generated by our method are directly conditioned on expression parameters and action units (Ekman & Friesen, 1978). Further, unlike our method, DECA (Feng et al., 2021) is unable photorealistically render the geometric details, leading to sub-par animation.
Facial Expression Editing. The success of image-to-image translation networks (Isola et al., 2017), adversarial training (Goodfellow et al., 2014) and cycle-consistency losses (Zhu et al., 2017) have led to novel expression editing methods that use large-scale in-the-wild training datasets. In (Shu et al., 2017), an unsupervised face model disentangles the input face into albedo, normals and shading. Expressions are then edited by traversals in the disentangled latent space. In Choi et al. (2018), expressions are edited via adversarial losses coupled with cycle consistency. In (Pumarola et al., 2020), a network edits input images to target expressions, represented as Action Units (Ekman & Friesen, 1978). Athar et al. (2020) extends this work by explicitly modeling skin motion followed by texture hallucination. In Choi et al. (2020), template images are used for editing. While these methods give photo-realistic results, they are restricted to the 2D image space and cannot be used to manipulate 3D viewpoint.
3 FACEDET3D
3.1 PROBLEM FORMULATION
Let Ix ∈ RH×W×3 be a face image with an expression x represented by Action Units (Ekman & Friesen, 1978). Its shape and expression parameters in the Basel Face Model (BFM) (Gerig et al., 2018) space are {αs,αe}. A vertex displacement UV map D(Ix) ∈ RHD×WD×3 encodes the facial details of the person shown in Ix with the expression x. We want to: (1) Generate facial details, represented by a vertex displacement map D(Iy), of the same person with expression y; (2) Render an image of the geometry with the generated facial geometric details. We use FDS (Chen et al., 2019) to extract the texture map, initial detail map and geometry of the input image Ix:
T (Ix),D(Ix),αs,αe ← FDS(Ix) , (1) where T (Ix) ∈ RHT ×WT×3 is the texture map. Next, we use the detail hallucination network, DetH (·), to hallucinate the plausible facial detail map of the person in Ix for expression y, conditioned on D(Ix), as follows:
D̃(Iy) = DetH (D(Ix),x,y,Age(Ix),FaceID(Ix)) , (2)
where y is the target AU, x is the input AU, Age(Ix) are features extracted from an age prediction network and FaceID(Ix) is the facial embedding of Ix extracted using (Schroff et al., 2015). Note that we do not have access to the ground truth image Iy, thus we hallucinate a plausible detail map of Iy (i.e D̃(Iy)) using DetH . Once we have D̃(Iy), we use it to displace the vertices along their normal direction giving us the detailed geometry. Finally, we render this detailed face geometry using a rendering network R (·):
Ĩy = R ( T (Ix), D̃(Iy),αs, α̂e,y, c, l, γ ) , (3)
where α̂e are the target expression parameters, c are the desired camera parameters, γ is the albedo PCA-space parameters of BFM(Gerig et al., 2018), and l are the lighting parameters.

heading: 3.2 DETAIL HALLUCINATION

text:
DetH takes the input detail map,D(Ix), the input expression AU x, the shape parameters αs, the face embedding FaceID(Ix), and the age features Age(Ix) and extracts features from each of them. There features are concatenated in the channel dimension and passed though the Decoder which receives the target action unit y via Adaptive Instance Normalization (Huang & Belongie, 2017) layers. The Decoder gives as output a hallucination D(Iy)H and a mask D(Iy)M which are combined together with the input detail map, D(Ix), to give D̃(Iy) as follows:
D̃(Iy)=D(Iy)M D(Iy)H+(19D(Iy)M ) D(Ix) . (4)
The masking mechanism ensures the preservation of the details that are not meant to be changed with the expression. A schematic of the network is shown in Fig 2.

heading: 3.2.1 TRAINING LOSSES

text:
Due to the lack of a large scale in-the-wild dataset of paired data with expression change or 3D data, we cannot perform regression using ground-truth geometric facial details, D(Iy) of the image Iy. Therefore, we instead use an adversarial training paradigm along with cycle-consistency losses similar to (Athar et al., 2020; Pumarola et al., 2020) to hallucinate the plausible facial geometric details D̃(Iy) and to ensure the hallucination’s fidelity to the target expression and input features. In order to speed up convergence, we weakly supervise the adversarial training using randomly sampled frames from videos of the MUG (Aifanti et al., 2010) and the ADFES datasets (Van Der Schalk et al.,
2011), ensuring that the sampling is sparse enough such that there is significant change in expression with frames sampled from each video. We leave the full exposition of all the standard losses and regularizations to the supplementary.
Expression Adversarial Loss. In order to ensure the hallucinated facial geometric details, D̃(Iy), are consistent with the target expression y, as encoded by AUs, we use an expression discriminator DExp. Given D(Ix) of some image Ix manifesting expression x, DExp, outputs the following
DExp(D(Ix)) = {r, x̂} , (5)
where r is a realism score and x̂ is the predicted AU. For brevity, we will use DExp(D(Ix)) and DExp(D) interchangeably. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the R1 gradient penalty (Mescheder et al., 2018) to train DExp. We use a UNet based discriminator (Schonfeld et al., 2020) in order to discriminate on pixel level. In addition, DExp is trained to minimize the error of the predicted AU
LDExpAU = ED∼PD [ ||[DAUExp(D) 9 x||22 ] , (6)
where DAUExp is the AU output head of DExp. The Details Hallucination Network, DetH , in addition to be trained to minimize adversarial loss, is also trained to minimize the expression loss:
LDetHAU = E Ix,{y} ||DAUExp(DetH (·)) 9 y||22 (7)
where DetH (·) is to be read as in Eq. (2) and y is the target AU. Superresolution Losses. The detail maps generated by FDS (Chen et al., 2019) are of resolution 4096× 4096 and thus cannot be used directly for training due to GPU-memory constraints. To get around this, we train DetH on detail maps downsampled to 256× 256. Simultaneously, we finetune a superresolution network, RCAN (Zhang et al., 2018), to super-resolve downsampled 256 × 256 patches of D(Ix) by a factor of 4
LRCANSR = L1 ( RCAN ( D(Ix)P256 ) ,D(Ix)P1024 ) , (8)
where D(Ix)P1024 is a randomly sampled patch of resolution 1024 × 1024 from the full-resolution detail map D(Ix) and D(Ix)P256 is its downsampled version. During inference, we use RCAN twice on the predicted detail map D̃(Iy) to upsample it to 4096× 4096:
D̃(Iy)HR = RCAN ( RCAN ( D̃(Iy) )) . (9)
In the interest of brevity, we will use D̃(Iy) in lieu of D̃(Iy)HR in the remainder of this text.

heading: 3.3 RENDERING NETWORK

text:
The rendering network, R , consists of two subnetworks: (1) The Neural Texture prediction network NTex φ and (2) The image rendering network Tex2Im . Neural Texture Prediction. NTex φ predicts the Neural Textures given the texture map T (Ix) of image Ix:
NTM = NTex φ(T (Ix)); NTM ∈ R F×H×W , (10)
where NTM is the predicted neural texture map with F channels. Unlike in (Thies et al., 2019), where the NTM is identity specific, NTex φ can be used on any T (Ix), regardless of identitym to generate its corresponding neural texture map. Through training, NTex φ learns to extract the appropriate high-dimensional texture features from T (Ix) such that NTM can be used to generate a realistic render of the person in Ix in any desired expression and view. Image rendering. The image rendering network, Tex2Im , consists of two branches, the low-res image renderer gθ and the detail renderer gω . The low-res image renderer generates identity-specific image textures that are invariant to the predicted geometric details, such as the skin-tone or eye color. The detail renderer gω renders the facial geometric details obtained from the detail hallucination network, DetH , and adds them to the low-res image generated by gθ. The separation of the image rendering network into two branches allows each branch to focus on its respective task and leads to high-quality renderings. The low-res image renderer, gθ, takes as input the NTM sampled using the UV map rasterized by the proxy geometry, GP = {0 × D̃(Iy),αs, α̂e}, i.e the geometry without any details, along with the shaded albedo also rasterized by GP :
ÎLRy = gθ (Sample(NTM,UV-Raster(GP , c)),y,γ , l) (11)
where c are the camera parameters, y is the target AU, γ is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and l are the lighting parameters. Since gθ only uses inputs dependent on GP it generates the image textures that are invariant to details predicted by DetH .
The detail renderer, gω , takes as input the NTM sampled using the UV map rasterized by the detailed geometry, GD = {D̃(Iy),αs, α̂e} along with the shaded albedo also rasterized by GD:
ÎDy = gθ(Sample(NTM,UV-Raster(GD, c)),y, γ, l) (12)
where c are the camera parameters, y is the target AU, γ is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and l are the lighting parameters. The final output image is calculated as:
Îy = Î LR y + Î D y . (13)
Since all of the detail invariant textures are already generated by gθ in ˆILRy , the detail renderer, gω, can solely focus on realistically rendering the details hallucinated by the details hallucination network DetH .

heading: 3.3.1 TRAINING LOSSES

text:
The renderings generated by R must: 1) faithfully render the geometric details onto the RGB space and 2) be realistic. Neural Rendering (Thies et al., 2019) is designed to address 2) as the highdimensional neural texture map is able store the fine details of the output texture. Consequently, the details on the rendered image become entirely conditional on the input texture map, T (Ix), and ignore the detailed geometryGD. This significantly hurts 1) causing the details on the rendered image to remain unchanged even if the facial geometric details change due to changes in facial expression. In order to ensure the output renderings satisfy 2) we use, along with the branched architecture discussed in Sect 3.3, an Augmented Wrinkle Loss (AugW) and the Detailed Shading Loss (DSL) to ensure the geometric details are faithfully rendered onto the output image. Additionally, R is also trained with a photometric loss and an Expression Adversarial Loss in order to maintain photometric and expression consistency. We leave the full exposition of all the standard losses and regularizations to the supplementary. Augmented Wrinkle Loss (AugW). In order to enforce the rendering of geometric details onto the rendered image we add ‘fake’ wrinkles to an image Ix and force R to generate the same. Given the the detailed geometry of Ix, GD = {D(Ix),αs,αe}, a geometry with ‘fake’ details G∗D = {D(I∗z),αs,αe} using the geometric details from some random image I∗z and the lighting l of Ix, the ‘fake’ wrinkles are added as follows:
Shading(Ix) = LSph(GD, l);Shading ∗(Ix) = LSph(G ∗ D, l)
I∗ x = Shading∗(Ix)×
( Ix
Shading(Ix)
) (14)
where LSph is the spherical harmonic lighting function and l are the coefficients of the first 9 spherical harmonics. The artificially wrinkled image I∗
x is now re-rendered using R
Î∗x = R (T (Ix), G∗D,x, c, l) LAugW = LapLoss(Î∗x, I∗x) .
(15)
where, LapLoss is the Laplacian Loss (Ling & Okada, 2006). In order to faithfully reconstruct I∗ x , R is forced to rely on the detailed geometry G∗D, since the input texture map T (Ix), and consequently the neural texture map, contain no information about the ‘fake’ wrinkles. Detailed Shading Loss (DSL). In addition to the Augmented Wrinkle Loss, we also try to predict the shading of the detailed facial geometry from the output rendering Îx
ˆShading(Îx) = fθ(Îx) LDSL = LapLoss( ˆShading(Îx),Shading∗(Ix)) , (16)
where fθ is a small convolutional network (CNN) with only two layers and the shading Shading∗(Ix) is calculated as in Eq. (14). We calculate this loss only over the skin region. Since, fθ is a small CNN with limited representational capacity, the details must be quite visible on the rendered image Îx in order for them to be picked up by fθ to generate an accurate shading ˆShading(Îx). Expression Adversarial Loss. In order to ensure that the rendered output conforms to the target expression we use an expression adversarial loss. Given a rendered image, Îx = R (T (Ix), GD,x, c, l),
manifesting the expression encoded by AU x an expression discriminator, DRGBExp , outputs
DRGBExp (Îx) = {r, x̂} , (17)
where r is a realism score and x̂ is the predicted AU. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the R1 gradient penalty (Mescheder et al., 2018) to train DExp. In addition, DRGBExp is trained to minimize the predicted AU error
LD RGB Exp AU = EIx∼PI [ ||[DRGB,AUExp (Ix) 9 x|| 2 2 ] , (18)
where DRGB,AUExp is the AU output head of D RGB Exp . The Rendering Network, R , in addition to be trained to minimize adversarial loss, is also trained to minimize the AU loss
LRAU = EIx ||D RGB,AU Exp (R (·)) 9 y||22 , (19)
where R (·) is to be read as in Eq. (3).

heading: 4 RESULTS

text:
We train the detail hallucination network, DetH , and the rendering network R , on 9,000 images from the FFHQ dataset (Karras et al., 2019). Additionally, 3,000 frames were sampled from the MUG (Aifanti et al., 2010) and the ADFES (Van Der Schalk et al., 2011) datasets to speed-up training of the detail hallucination network DetH . Due to memory constraints, DetH and R are trained independently. Upon publication we will release the code. Below, we show the FaceDet3D’s results on expression change, results on view consistency, a comparison with DECA (Feng et al., 2021) and Ablation studies.
Expression change. We first demonstrate the detail hallucination and rendering results with expression change. Fig 4 shows the result of changing the expression of some input image to a variety of expressions. The first column of Fig 4 is the input image, the second column shows the image reconstructed using the detail map D(Ix) predicted by FDS (Chen et al., 2019). In the subsequent columns D(Ix) is used as input to generate D̃(Iy) where y = {Anger,Disgust,Happy,Surprise}. The first row shows the final rendered image, i.e the output of R and the second row shows the shaded geometry with the hallucinated details, i.e GD. The hallucinated details and their corresponding rendering are marked out with dashed red and blue rectangles. We zoom in the red rectangles of the last column. As seen in the second row of Fig 4, the details hallucinated are consistent with the manifested expression. For example, ‘Anger’ and ‘Disgust’ (3rd and 4th column of Fig 4 respectively) show consistent wrinkling around the forehead and the nose while ‘Happy’ (column 6 Fig 4) shows consistent wrinkling around the cheeks. Zooming in the last column of Fig 4 highlights the realism of the hallucinated details produced by the rendering network, R . Further, in Fig 7 we show the utility of hallucinating details as the expression changes. The first row of Fig 7 shows the image rendered with
the hallucinated details from DetH as the expression changes. The second row of images shows the images rendered without details, this is done by setting the hallucinated detail map to zero. As seen by comparing the skin appearance marked with the red rectangles, in the first row the skin changes realistically as the expression of the person changes due to the changing facial geometric details, while the skin in the second row remains unchanged, significantly hurting realism. In the supplementary we show further results on expression animation and encourage the reader to inspect them.
Reconstruction Fear Happy
Render without details
Detailed Shading
DisgustInput
DECA. The row label is the expression being animated, the first column is the input image, the second is the detailed geometry hallucinated by FaceDet3D for the expression being animated, the
3rd column is the render generated by FaceDet3D of its hallucinated detailed geometry, the fourth column is the detailed geometry generated by DECA for the expression being animated and the fifth column is render generated by DECA. Note that DECA’s results have the full head because they use the FLAME 3DMM while we use the BFM 3DMM. By comparing the details in the second and the fourth column of Fig 6 one can see that FaceDet3D generates more concentrated and higher quality details. For example, DECA always generates wrinkles on the forehead regardless of the expression being animated, thus the details generated for ‘Happy’ (row 1) are inconsistent with the expression. Similarly, a comparison between the third and fifth column of Fig 6 shows that the renders of FaceDet3D are significantly more realistic than those of DECA. In Table 1, we see that the FaceID distance (using FaceNet (Schroff et al., 2015)) and the FID score, across a variety of expression edits, of the renders of FaceDet3D is significantly lower than the renders of DECA. Further, the blue boxes shown in columns 2 and 3 of Fig 6 show that the hallucinated details are faithfully and photo-realistically reproduced in the render.
Ablation Studies. We examine the utility of the Augmented Wrinkle Loss (AugW) and the Detailed Shading Loss (DSL) in rendering the facial geometric details to the image space. Fig 8 shows the results of training R with and without the AugW and DP losses. As seen by comparing the results in rows 2 and 3 of Fig 8, without those losses R fails to render the hallucinated geometric facial details. Next, we ablate using a single network in Tex2Im (‘NR Single Branch’) versus using two networks as described in Sect 3.3 (‘NR Double Branch’) versus using Ordinary Rendering using the provided texture space of BFM(Gerig et al., 2018). We calculate the FID score (Heusel et al., 2017), and the FaceID distance using FaceNet (Schroff et al., 2015) across a variety of expression edits. As can be seen in Table 2, using two networks, one that outputs the detail-invariant textures and the other that renders the details, gives better results as compared to using a single network. It not only produces more realistic images (as measured by the FID score) but also more closely preserves the identity across a variety of expression edits. As it is expected, ordinary rendering performs the worst as its
PCA space cannot capture the rich details of the human face across a variety of input identities and expression edits.

heading: 5 CONCLUSION AND FUTURE WORK

text: We have presented FaceDet3D, a method for hallucinating, from a single image, plausible facial geometric details as the expression changes and render them photo-realistically. The details hallucination network is trained using adversarial losses and weak supervision without any ground-truth 3D data. The rendering network is constrained using the Augmented Wrinkle Loss and Detailed Shading Loss, forcing it to use cues from the detailed geometry to render the details, ensuring their fidelity and consistency across a variety of expressions and views. The detail hallucination network relies on the detail map estimated by FDS (Chen et al., 2019) as input to predict the plausible details of the target expression, therefore it cannot handle occlusions such as glasses or make-up very well as FDS (Chen et al., 2019) fails in those conditions. In future work, we plan to incorporate explicit disentanglement of lighting in order to enable greater control over the final rendering along with explicit modelling occlusions and joint training of DetH and R .


summaries: This paper proposes a new model ADVAE, which uses a sequence of latent variables which are constructed using cross-attention, which are then used to condition the inference model. The findings show that certain latent variables correlated with different syntactic roles (measured using a dependency parser).  The work claims that the latent variables, in this case, are able to disentangle content effectively, which I am inclined to agree with, however, I feel that the experimental setup is lacking to solidly support this hypothesis (which I detail in the “cons” and “questions” section).

summaries: The paper proposes a method for unsupervised disentanglement of text components and shows its ability to identify semantic roles. To this end, a neural network is trained to compress the input into a fixed number of independent latent variables which are regularized to be standard Gaussians via the VAE framework. The inference network consists of a Transformer encoder-decoder network, where the decoder inputs correspond to the latent variables, which cross-attend to the outputs of the Transformer encoder, i.e., the encoded sentence. The idea behind this architecture is that attention-based seq2seq architectures align source and target sequences with each other.  The model is evaluated on disentanglement of semantic roles. To this end, they investigate how resampling of individual latent variables impacts the semantic roles in the text generated by the decoder, and how syntactic roles are aggregated into latent vectors via attention. They find that their proposed architecture is more successful at disentangling semantic roles into the latent variables than standard VAEs.

summaries: This paper propose a framework to obtain the disentanglement of syntactic roles as latent variables for sentence representations. The model is an attention-driven VAE which maps syntactic roles to separate latent variables using an encoder-decoder framework. In the second part of the paper, the authors introduce an evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder, which includes syntactic role extraction, latent variable influence on decoder, encoder influence on latent variables and disentanglement metrics.

summaries: ## Summary  - This paper proposes a probabilistic model called Attention-Driven Variational Autoencoder (ADVAE). This model is another instance of $\beta$-VAE whose encoder and encoders are composed of Transformers rather than previous neural architectures such as RNN. - The authors aim to disentangle the semantics of latent variables according to some syntactic roles (e.g., nouns and verbs) defined by syntax. To achieve this goal, they suggest employing the combination of Transformer and the existing $\beta$-VAE framework which is known to be effective for disentangling the role of each latent variable in VAE. - Moreover, this work presents a new way of quantifying syntactic disentanglement between latent variables, relying on the information obtained from the attention matrices of the Transformer architecture. - The experiments show that the proposed method is quantitatively better than the normal VAE, and that swapping the value of a specific latent variable can impact the generation of the target word (decided by the syntactic role of the latent variable we choose).

abstractText: Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,. . . ) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that i) disentanglement of syntactic roles can be induced without supervision, ii) ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, iii) realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available1.

authors:
Dzmitry Bahdanau
Kyung Hyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings,

authors:
Yu Bao
Hao Zhou
Shujian Huang
Lei Li
Lili Mou
Olga Vechtomova
Xinyu Dai
Jiajun Chen

title: Generating sentences from disentangled syntactic and semantic spaces. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics

venue: Proceedings of the Conference,

authors:
Melika Behjati
James Henderson

title: Inducing meaningful units from character sequences with slot attention, 2021

authors:
Samuel R. Bowman
Gabor Angeli
Christopher Potts
Christopher D. Manning

title: A large annotated corpus for learning natural language inference

venue: In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,

authors:
Samuel R Bowman
Luke Vilnis
Oriol Vinyals
Andrew M Dai
Rafal Jozefowicz
Samy Bengio

title: Generating sentences from a continuous space

venue: CoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning,

authors:
Mingda Chen
Qingming Tang
Sam Wiseman
Kevin Gimpel

title: A multi-task approach for disentangling syntax and semantics in sentence representations

authors:
Tian Qi Chen
Xuechen Li
Roger Grosse
David Duvenaud

title: Isolating sources of disentanglement in variational autoencoders

venue: In 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings,

authors:
Pengyu Cheng
Martin Renqiang Min
Dinghan Shen
Christopher Malon
Yizhe Zhang
Yitong Li
Lawrence Carin

title: Improving Disentangled Text Representation Learning with InformationTheoretic Guidance

authors:
Kyunghyun Cho
Bart van Merrienboer
Caglar Gulcehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using rnn encoder-decoder for statistical machine

authors:
Kevin Clark
Urvashi Khandelwal
Omer Levy
Christopher D Manning

title: What Does BERT Look at? An Analysis of BERT’s Attention

venue: In BlackBoxNLP@ACL,

authors:
Marie-Catherine de Marneffe
Christopher D. Manning

title: The Stanford typed dependencies representation

venue: In Coling 2008: Proceedings of the workshop on Cross-Framework and CrossDomain Parser Evaluation,

authors:
Marie-Catherine De Marneffe
Christopher D Manning

title: Stanford typed dependencies manual

venue: Technical report,

authors:
Jacob Devlin
Ming Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

authors:
Andrea Dittadi
Frederik Träuble
Francesco Locatello
Manuel Wuthrich
Vaibhav Agrawal
Ole Winther
Stefan Bauer
Bernhard Schölkopf

title: On the transfer of disentangled representations in realistic settings

venue: In International Conference on Learning Representations,

authors:
Wenyu Du
Zhouhan Lin
Yikang Shen
Timothy J O’Donnell
Yoshua Bengio
Yue Zhang

title: Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach

authors:
Chris Dyer
Adhiguna Kuncoro
Miguel Ballesteros
Noah A Smith

title: Recurrent neural network grammars. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016

authors:
John Hewitt
Christopher D. Manning

title: A structural probe for finding syntax in word representations

venue: In Proceedings of the

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: B-VAE: Learning basic visual concepts with a constrained variational framework

venue: 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings,

authors:
Irina Higgins
Arka Pal
Andrei A. Rusu
Loic Matthey
Christopher P Burgess
Alexander Pritzel
Matthew Botvinick
Charles Blundell
Alexander Lerchner. Darla

title: Improving zero-shot transfer in reinforcement learning, 2018

authors:
Matthew Honnibal
Ines Montani

title: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing

authors:
Jennifer Hu
Jon Gauthier
Peng Qian
Ethan Wilcox
Roger P Levy

title: A Systematic assessment of syntactic generalization in neural language models

authors:
James Y. Huang
Kuan-Hao Huang
Kai-Wei Chang

title: Disentangling semantics and syntax in sentence embeddings with pre-trained language models

venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1372–1379,

authors:
Kuan-Hao Huang
Kai-Wei Chang

title: Generating syntactically controlled paraphrases without using annotated parallel pairs

venue: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,

authors:
Ganesh Jawahar
Benoı̂t Sagot
Djamé Seddah
Ganesh Jawahar Benoˆıt
Benoˆıt Sagot

title: What does BERT learn about the structure of language?(ACL2019)

authors:
Vineet John
Lili Mou
Hareesh Bahuleyan
Olga Vechtomova

title: Disentangled representation learning for non-parallel text style transfer

venue: In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference,

authors:
Hyunjik Kim
Andriy Mnih

title: Disentangling by factorising

venue: 35th International Conference on Machine Learning, ICML 2018,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A Method for Stochastic Optimization

venue: In 3rd International Conference on Learning Representations,

authors:
Diederik P. Kingma
Max Welling

title: Auto-encoding variational bayes

venue: In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations,

authors:
Jordan Kodner
Nitish Gupta

title: Overestimation of syntactic representation in neural language models

authors:
Artur Kulmizev
Vinit Ravishankar
Mostafa Abdou
Joakim Nivre

title: Do neural language models show preferences for syntactic formalisms

authors:
Mike Lewis
Yinhan Liu
Naman Goyal
Marjan Ghazvininejad
Abdelrahman Mohamed
Omer Levy
Ves Stoyanov
Luke Zettlemoyer

title: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019

authors:
Chunyuan Li
Xiang Gao
Yuan Li
Baolin Peng
Xiujun Li
Yizhe Zhang
Jianfeng Gao

title: Optimus: Organizing sentences via pre-trained modeling of a latent space, 2020a

authors:
Juncen Li
Robin Jia
He He
Percy Liang

title: Delete, retrieve, generate: a simple approach to sentiment and style transfer

venue: URL https://www.aclweb.org/anthology

authors:
Zhiyuan Li
Jaideep Vitthal Murkute
Prashnna Kumar Gyawali
Linwei Wang

title: Progressive Learning and Disentanglement of Hierarchical Representations. arXiv, 2 2020b. ISSN 23318422

authors:
Nelson F. Liu
Matt Gardner
Yonatan Belinkov
Matthew E. Peters
Noah A. Smith

title: Linguistic knowledge and transferability of contextual representations

authors:
Francesco Locatello
Dirk Weissenborn
Thomas Unterthiner
Aravindh Mahendran
Georg Heigold
Jakob Uszkoreit
Alexey Dosovitskiy
Thomas Kipf

title: Object-centric learning with slot attention, 2020

authors:
Minh Thang Luong
Hieu Pham
Christopher D Manning

title: Effective approaches to attention-based neural machine translation

venue: Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing,

authors:
Rebecca Marvin
Tal Linzen

title: Targeted syntactic evaluation of language models

venue: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Martha Palmer
Daniel Gildea
Paul Kingsbury

title: The proposition bank: An annotated corpus of semantic roles

venue: Computational linguistics,

authors:
Tiago Pimentel
Josef Valvoda
Rowan Hall Maudslay
Ran Zmigrod
Adina Williams
Ryan Cotterell

title: Information-theoretic probing for linguistic structure

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609–4622,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J. Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer, 2020

authors:
Anna Rogers
Olga Kovaleva
Anna Rumshisky

title: A Primer in BERTology: What we know about how BERT

venue: works. arXiv,

authors:
Michal Rolinek
Dominik Zietlow
Georg Martius

title: Variational autoencoders pursue pca directions (by accident)

venue: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,

authors:
Sashank Santhanam
Samira Shaikh

title: Emotional neural language generation grounded in situational contexts

venue: In Proceedings of the 4th Workshop on Computational Creativity in Language Generation,

authors:
Florian Schmidt
Stephan Mandt
Thomas Hofmann

title: Autoregressive text generation beyond feedback loops

venue: EMNLP-IJCNLP

authors:
Yikang Shen
Shawn Tan
Alessandro Sordoni
Aaron Courville

title: Ordered neurons: Integrating tree structures into recurrent neural networks

venue: In 7th International Conference on Learning Representations,

authors:
Aditya Siddhant
Melvin Johnson
Henry Tsai
Naveen Arivazhagan
Jason Riesa
Ankur Bapna
Orhan Firat
Karthik Raman

title: Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation, 2019

title: UDPipe 2.0 prototype at CoNLL 2018 UD shared task

venue: In Proceedings of the CoNLL

authors:
Ian Tenney
Dipanjan Das
Ellie Pavlick

title: BERT rediscovers the classical NLP pipeline. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention Is All You Need. In NeurIPS, number Nips, 2017

authors:
Ralph Weischedel
Martha Palmer
Mitchell Marcus
Eduard Hovy
Sameer Pradhan
Lance Ramshaw
Nianwen Xue
Ann Taylor
Jeff Kaufman
Michelle Franchini
Mohammed El-Bachouti
Robert Belvin
Ann Houston

title: URL https://hdl.handle.net /11272.1/AB2/MKJJ2R

venue: OntoNotes Release

authors:
Chen Wu
Prince Zizhuang Wang
William Yang Wang

title: On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

venue: URL

authors:
Peng Xu
Jackie Chi Kit Cheung
Yanshuai Cao

title: On Variational Learning of Controllable Representations for Text without Supervision

venue: The 37th International Conference on Machine Learning (ICML 2020),

authors:
Xinyuan Zhang
Yi Yang
Siyang Yuan
Dinghan Shen
Lawrence Carin

title: Syntax-Infused Variational Autoencoder for Text Generation

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Shengjia Zhao
Jiaming Song
Stefano Ermon

title: Learning hierarchical features from deep generative models

venue: 34th International Conference on Machine Learning, ICML 2017,

authors:
De Marneffe
Manning

title: We set the number of importance samples

heading: 1 INTRODUCTION

text:
A disentangled representation of data describes information as a combination of separate understandable factors. This separation provides better transparency, but also better transfer performance (Higgins et al., 2018; Dittadi et al., 2021). When it comes to disentanglement, Variational Autoencoders (VAEs; Kingma & Welling, 2014) were extensively proven effective (Higgins et al., 2017; Chen et al., 2018; Rolinek et al., 2019). and were used throughout several recent works (Chen et al., 2019; Li et al., 2020b; John et al., 2020). In NLP, disentanglement has been mostly performed to separate the semantics (or content) in a sentence from characteristics such as style and structure in order to generate paraphrases (Chen et al., 2019; John et al., 2020; Bao et al., 2020; Huang & Chang, 2021; Huang et al., 2021). We show in our work that the information in the content itself can be separated with a VAE-based model. In contrast to the aforementioned works, we use neither supervision nor input syntactic information for this separation. We demonstrate this ability by controlling the lexical realization of core syntactic roles. For example, the subject in a sentence can be encoded separately and controlled to generate the same sentence with another subject. Our framework includes a model and an evaluation protocol aimed at measuring the disentanglement of syntactic roles.
The model we introduce is an Attention-Driven VAE (ADVAE), which we train on the SNLI raw text dataset (Schmidt et al., 2020). It draws its inspiration from attention-based machine translation models (Bahdanau et al., 2015; Luong et al., 2015). Such models translate sentences between languages with
1URL to be disclosed upon publication. Our code is provided as supplemental material during the submission process.
different underlying structures and can be inspected to show a coherent alignment between spans from both languages. Our ADVAE uses Transformers (Vaswani et al., 2017), an attention-based architecture, to map sentences from a language to independent latent variables, then map these variables back to the same sentences. Although ADVAE could be used to study other attributes, we motivate it (§4.1) and therefore study it for the alignment of syntactic roles with latent variables.
Evaluating disentanglement with regard to spans is challenging. After training the model and only for evaluation, we use linguistic information (from an off-the-shelf dependency parser) to first extract syntactic roles from sentences, and then study their relation to latent variables. To study this relation on the ADVAE decoder, we repeatedly i) generate a sentence from a sampled latent vector ii) perturb this latent vector at a specific location iii) generate a sentence from this new vector and observe the difference. On the encoder side, we study the attention values to see whether each latent variable is focused on a particular syntactic role in input sentences. The latter procedure is only possible through the way our ADVAE uses attention to produce latent variables. To the best of our knowledge, we are the first to use this transparency mechanism to obtain quantitative results for a latent variable model.
We first justify our focus on syntactic roles in §3, then we go over our contribution, which is threefold: i) We introduce the ADVAE, a model that is designed for unsupervised disentanglement of syntactic roles, and that enables analyzing the interaction between latent variables and observations through the values of attention (§4), ii) We design an experimental protocol for the challenging assessment of disentanglement over realizations of syntactic roles, based on perturbations on the decoder side and attention on the encoder side (§5), iii) Our empirical results show that our architecture disentangles syntactic roles better than standard sequence VAEs and Transformer VAEs and that it is capable of controlling realizations of syntactic roles separately during generation (§6).

heading: 2 RELATED WORKS

text:
Linguistic information in neural models Accounting for linguistic information provided better inductive bias in the design of neural NLP systems during recent years. For instance, successful attempts at capturing linguistic information with neural models helped improve grammar induction (RNNG; Dyer et al., 2016), constituency parsing and language modeling (ON-LSTM; Shen et al., 2019, ONLSTM-SYD; Du et al., 2020), as well as controlled generation (SIVAE; Zhang et al., 2019). Many ensuing works have also dived into the linguistic capabilities of the resulting models, the types of linguistic annotations that emerge best in them, and syntactic error analyses (Hu et al., 2020; Kodner & Gupta, 2020; Marvin & Linzen, 2020; Kulmizev et al., 2020). Based on the Transformer architecture, the self-supervised model BERT (Devlin et al., 2019) has also been subject to studies showing that the linguistic information it captures is organized among its layers in a way remarkably close to the way a classical NLP pipeline works (Tenney et al., 2020). Furthermore, (Clark et al., 2019), showed that many attention heads in BERT specialize in dependency parsing. We refer the reader to (Rogers et al., 2020) for an extensive review of Bert-related studies. However, such studies most often rely on structural probes (Jawahar et al., 2019; Liu et al., 2019; Hewitt & Manning, 2019) to explain representations, probes which are not without issues, as shown by Pimentel et al. (2020). In that regard, the generative capabilities and the attention mechanism of our model offer an alternative to probing: analysis is performed directly on sentences generated by the model and on internal attention values.
Disentanglement in NLP The main line of work in this area revolves around using multitask learning to separate concepts in neural representations (e.g. style vs content (John et al., 2020), syntax vs semantics (Chen et al., 2019; Bao et al., 2020)). Alternatively, Huang & Chang (2021) and Huang et al. (2021) use syntactic trees as inputs to separate syntax from semantics, and generate paraphrases without a paraphrase corpus. Towards less supervision, Cheng et al. (2020) only uses style information to separate style from content in representations. Literature on unsupervised disentanglement in NLP remains sparse. Examples are the work of Xu et al. (2020) on categorical labels (sentiment and topic), and that of Behjati & Henderson (2021) on representing morphemes using character-level Seq2Seq models. The work of Behjati & Henderson (2021) is closest to ours as it uses Slot Attention (Locatello et al., 2020), which, like ADVAE, is a cross-attention-based representation technique. Our contribution depart from previous work since i) syntactic parses are not used as learning signals but as a way to interpret our model, and ii) cross-attention enables our model to link a fixed number of latent variables to text spans.

heading: 3 SYNTACTIC ROLES AND DEPENDENCY PARSING

text:
We present in Figure 1 an example sentence with its dependency parse2, its Part-of-Speech (PoS) tags, and a flat predicative structure with PropBank-like semantic roles (Palmer et al., 2005). Dependency parsing yields a tree, where edges are labeled with syntactic roles (or relations or functions) such as nominal subject (nsubj). The lexical realizations of these syntactic functions are textual spans and correspond to syntactic constituents. For instance, the lexical realization of the direct object (dobj) of the verb holds in this sentence is the span his nice guitar, with guitar as head. In short, the spans corresponding to subtrees consist of tokens that are more dependent of each other than of the rest of the sentence. As a consequence, and because a disentanglement model seeks independent substructures in the data, we expect such a model to converge to representations that display separation in realizations of frequent syntactic roles.
In our work, we focus3 within the same framework. on nominal subjects, verbal roots of sentences, and direct or prepositional objects. These are core (as opposed to oblique; see Nivre et al. (2016) for details on the distinction) syntactic roles, since they directly relate to the predicative structure. In fact in most cases, as illustrated in Figure 1, the verbal root of a sentence is its main predicate, the nominal subject its agent (ARG0) and the direct or prepositional object its patient (ARG1).

heading: 4 MODEL DESCRIPTION

text: The usual method to obtain sentence representations from Transformer models uses only a Transformer encoder either by taking an average of the token representations or by using the representation of a special token (e.g [CLS] in BERT(Devlin et al., 2019)). Recently, the usage of both Transformer encoders and decoders has also been explored in order to obtain representations whether by designing classical Autoencoders (Lewis et al., 2019; Siddhant et al., 2019; Raffel et al., 2020), or VAEs (Li et al., 2020a). Our model, the ADVAE, differs from these models in that it uses targets in Machine Translation (MT) Transformers (i.e an encoder and a decoder) to produce sentence representations. Producing representations with Cross-Attention has been introduced by Locatello et al. (2020) as part of the Slot Attention modules in the context of unsupervised object discovery. However, in contrast to Locatello et al. (2020), we simply use Cross-Attention as it is found in Vaswani et al. (2017), i.e. without normalizing attention weights over the query axis, or using GRUs(Cho et al., 2014) to update representations. As will be shown through our experiments, this is sufficient to disentangle syntactic roles. We explain the observation that motivates our work in §4.1, we then describe in §4.2 the minimal changes we apply to MT Transformers, and finally, we present the objective we use in §4.3. The parallel between our model and MT Transformers is illustrated in Figure 2.

heading: 4.1 THE INTUITION BEHIND OUR MODEL

text:
Consider s = (sj)1≤j≤Ns and t = (tj)1≤j≤Nt , two series of tokens forming respectively a sentence in a source language and a sentence in a target language. Given s, attention-based translation models
2Following the ClearNLP constituent to dependency conversion, close to Stanford Dependencies de Marneffe & Manning (2008). See https://github.com/clir/clearnlp-guidelines/blob/master/ md/components/dependency_conversion.md.
3Future research that takes interest in the finer-grained disentanglement of content may simply study a larger array of syntactic roles. Using our current system we display results including all syntactic roles in Appendix G.
are capable of yielding t while also providing information about the alignment between the groups of tokens (of different sizes) in both sentences (Bahdanau et al., 2015; Luong et al., 2015)). This evidence suggests that attention-based architectures are capable of factoring information from groups of words according to a source structure, and redistributing it according to a target structure.
The aim of our design is to use, as a target, a set of NZ independent latent variables that will act as fixed placeholders for the information in sentences. We stress that NZ is fixed and independent of the input sentence size Ns. Combining Transformers, an attention-based MT model, and the VAE framework for disentanglement, our ADVAE is intended to factor information from independent groups of words into separate latent variables. In the following sections, we will refer to this set of independent latent variables as the latent vector z = (zi)1≤i≤NZ and to each zi as a latent variable.

heading: 4.2 MODEL ARCHITECTURE

text:
Inference model: This is the inference model qφ (encoder in Fig. 2.b) for our latent variables z = (zi)1≤i≤NZ . It differs from an MT Transformer in two ways. First it uses as input a sentence s, and NZ learnable vectors (eenczi )1≤i≤NZ instead of the target tokens t used in translation. These learnable vectors will go through Cross-Attention without Self-Attention. We stress that these learnable vectors are input-independent. Second its output is not used to select a token from a vocabulary but rather passed to a linear layer (resp. a linear layer followed by a softplus non-linearity) to yield the mean parameters (µi)1≤i≤NZ (resp. the standard deviation parameters (σi)1≤i≤NZ ) to parameterize the diagonal Gaussian distributions (q(i)φ (zi|s))1≤i≤NZ . The Transformer Decoder is therefore replaced in Fig 2.b by a Transformer Encoder that uses Cross-attention to factor information from the sentence. The distribution of the whole latent vector is simply the product of Gaussians qφ(z1, . . . , zNZ |s) = ∏NZ i q (i) φ (zi|s).
Generation model: Our generation model consists of an autoregressive decoder (Fig. 2.c) pθ(s|z1, . . . , zNZ ) = ∏Ns j pθ(sj |s<j , z1, . . . , zNZ ) where s<i is the series of tokens preceding
si, and a prior assuming independent standard Gaussian variables, i.e. p(z1, . . . , zNZ ) = ∏NZ i p(zi).
Each latent variable zi is concatenated with an associated learnable vector edeczi (z-Identifier in Fig. 2.c) instead of going through positional encoding. From there on, the latent variables are used like source tokens in an MT Transformer.

heading: 4.3 OPTIMIZATION OBJECTIVE

text:
We train our ADVAE using the β-VAE (Higgins et al., 2017) objective, which is the Evidence Lower-Bound (ELBo) with a controllable weight on its Kullback-Leibler (KL) term:
log pθ(s) ≥ E(z)∼qφ(z|s) [log pθ(s|z)]− βDKL[qφ(z|s)||p(z)] (1) In Eq. 1, s is a sample from our dataset, z is our latent vector and the distributions pθ(s) =∫ pθ(s|z)p(z)dz and qφ(z|s) are respectively the generation model and the inference model. We use a standard Gaussian distribution as prior p(z) and a diagonal Gaussian distribution as the approximate inference distribution qφ(z|s). The weight β is used (as in Chen et al., 2018, Xu et al., 2020, Li et al., 2020b) to control disentanglement, but also to find a balance between the expressiveness of latent variables and the generation quality.

heading: 5 EVALUATION PROTOCOL

text:
In order to quantify disentanglement, we first measure the interaction between latent variables and syntactic roles. To do so, we extract core syntactic roles from sentences according to the procedure we describe in §5.1. Subsequently, for the ADVAE decoder, we repeatedly perturb latent variables and measure their influence on the realizations of the syntactic roles in generated sentences (§5.2). For the ADVAE encoder, we use attention to determine the syntactic role that participates most in producing the value of each latent variable (§5.3).
Given these metrics, we measure disentanglement taking inspiration from the Mutual Information Gap (MIG; Chen et al., 2018) in §5.4. MIG consists in measuring the difference between the first and second latent variables with the highest mutual information with regard to a target factor. It is intended to quantify the extent to which a target factor is concentrated in a single variable. This metric assumes knowledge of the underlying distribution of the target information in the dataset.However, there is no straightforward or agreed-upon way to set this distribution for text spans, and therefore to calculate MIG in our case. As a workaround, we use the influence metrics defined in §5.2 and §5.3 as a replacement for mutual information to quantify disentanglement.

heading: 5.1 SYNTACTIC ROLE EXTRACTION

text:
We use the Spacy4 dependency parser (Honnibal & Montani, 2017) trained on Ontonotes5 (Weischedel et al., 2013). For each sentence the realization of verb is the root of the dependency tree if its POS tag is VERB. Realizations of subj (subject), dobj (direct object), and pobj (prepositional object) are spans of subtrees whose roots are labelled resp. nsubj, dobj, and pobj.
In the rare cases where multiple spans answer the requirement for a syntactic role, we take the first one as the subsequent spans are most often part of a subordinate clause. A realization of a syntactic role in R = {verb, subj, dobj, pobj} is empty if no node in the dependency tree satisfies its extraction condition.5

heading: 5.2 LATENT VARIABLE INFLUENCE ON DECODER

text:
Intuitively, we repeatedly compare the text generated from a sampled latent vector to the text generated using the same vector where only one latent variable is resampled. Thus we can isolate the effect of each latent variable on output text and gather statistics.
More precisely, we sample T dec latent vectors (z(l))1≤l≤Tdec = (z (l) i )1≤l≤Tdec,1≤i≤NZ . Then for each zl, and for each i we create an altered version z̃(li) = (z̃(li)i′ )1≤i′≤NZ where we resample only the ith latent variable (i.e. ∀i′ 6= i, z̃(li)i′ = z (l) i′ ).
Generating the corresponding sentences6 with pθ(s|z) yields a list of original sentences (s(l))1≤l≤Tdec , and a matrix of sentences displaying the effect of modifying each latent variable
4https://spacy.io/models/en#en_core_web_sm 5Examples of syntactic role extractions can be found in Appendix D. 6Throughout this work, we use greedy sampling (sampling the maximum-probability word at each step), for
all generated sentences.
(s̃(li))1≤l≤Tdec,1≤i≤NZ . For each syntactic role r ∈ R, we will denote the realization extracted from a sentence s with ρr(s).
To measure the influence of a variable zi on the realization of a syntactic role r, denoted Γdecri , we estimate the probability that a change in this latent variable incurs a change in the span corresponding to the syntactic role. We first discard, for the influence on a role r, sentence pairs (s(l), s̃(li)) where it appears or disappears, because the presence of a syntactic role is a property of its parent word, (e.g. the presence or absence of a dobj is controlled by the transitivity of the verb) hence not directly connected to the representation of the role r itself. As they are out of the scope of our work, we report measures of these structural changes (diathesis) in Appendix C, and leave their extensive study to future works. We denote the remaining number of samples T ′decri .
In the following, we use operator 1{.}, which is equal to 1 when the boolean expression it contains is true and to 0 when it is false. This process yields a matrix Γdec of shape (|R|, NZ) which summarizes interactions in the decoder between syntactic roles and latent variables:
Γdecri = T ′decri∑ l=1 1{ρr(s(l)) 6= ρr(s̃(li))} T ′decri
(2)

heading: 5.3 ENCODER INFLUENCE ON LATENT VARIABLES

text:
We compute this on a held out set of size T enc of sentences (s(l)j )1≤l≤T enc,1≤j≤N s(l) . Each sentence s(l) of sizeNs(l) generates an attention matrix (a (l) ij )1≤i≤NZ ,1≤j≤Ns(l) . Attention values are available in the Transformer Encoder with cross-attention computing the inference model7, and quantify the degree to which each latent variable embedding eenczi draws information from each token sj to form the value of zi.
For the encoder, we consider the influence of a syntactic role on a latent variable to be the probability for the attention values of the latent variable to reach their maximum on the index of a token in that syntactic role’s realization. The indices of tokens belonging to a syntactic role r in a sentence s(l) are denoted argr(s(l)). For each syntactic role r and sentence s(l), we discard inputs where this syntactic role cannot be found, and denote the remaining number of samples T ′encr . The resulting measure of influence of syntactic role r on variable zi is denoted Γencri . The whole process yields matrix Γ
enc of shape (|R|, NZ) which summarizes interactions in the encoder between syntactic roles and latent variables:
Γencri = T ′encr∑ l=1 1{arg max j (a (l) ij ) ∈ argr(s (l))} T ′encr (3)

heading: 5.4 DISENTANGLEMENT METRICS

text:
For Γ∗ (either Γdec or Γenc) each line corresponds to a syntactic role in the data. The disentanglement metric for role r is the following:
∆Γ∗r = Γ ∗ rm1 − Γ ∗ rm2 (4)
s.t. m1 = arg max 1≤i≤NZ Γ∗ri, m2 = arg max 1≤i≤NZ ,j 6=m1 Γ∗ri
We calculate total disentanglement scores for syntactic roles using Γdec, Γenc as follows: Ddec = ∑ r∈R ∆Γencr , Denc = ∑ r∈R ∆Γencr (5)
In summary, the more each syntactic role’s information is concentrated in a single variable, the higher the values of Ddec and Denc. However, similar to MIG, these metrics do not say whether variables
7For simplicity, attention values are averaged over attention heads and transformer layers. This also allows drawing conclusions with regard to the tendency of the whole attention network, and not just particular specialized heads as was done in Clark et al. (2019). Nevertheless, we display per-layer results in Appendix J.
capturing our concepts of interest are distinct. Therefore, we also report the number of distinct variables that capture the most each syntactic role (i.e the number of distinct values of m1 in Eq. 4 when looping over r). This is referred to as NΓenc for the encoder and NΓdec for the decoder.

heading: 6 EXPERIMENTS

text:
Dataset Previous unsupervised disentanglement works (Higgins et al., 2017; Kim & Mnih, 2018; Li et al., 2020b) tend to use relatively homogeneous and low complexity data. The data has low complexity if it varies along clear factors which correspond to what the model aims to disentangle. Similarly, we use a dataset where samples exhibit low variance in terms of syntactic structure while providing a high diversity of realizations for the syntactic roles composing the sentences, which is an adequate test-bed for unsupervised disentanglement of syntactic roles’ realizations. This dataset is the plain text from the SNLI dataset (Bowman et al., 2015) extracted8 by Schmidt et al. (2020). The SNLI data is a collection of premises (on average 8.92 ± 2.66 tokens long) made for Natural Language Inference. We use 90K samples as a training set, 5K for development, and 5K as a test set.
Setup Our objective is to check whether the architecture of our ADVAE induces better syntactic role disentanglement. We compare it to standard Sequence VAEs (Bowman et al., 2016) and to a Transformer-based baseline that doesn’t use cross-attention. Instead of cross-attention, this second baseline uses mean-pooling over the output of a Transformer encoder for encoding. For decoding, it uses the latent variable as a first token in a Transformer decoder, as is done for conditional generation with GPT-2 Santhanam & Shaikh (2019). These comparisons are performed using the same β-VAE objectives, and the decoder disentanglement scores as metrics. Training specifics and hyper-parameter settings are detailed in Appendix E. For each of the two baselines, the latent variables we vary during the decoder’s evaluation are the mono-dimensional components of its latent vector. It is easier to pack information about the realizations of multiple syntactic roles into Dz dimensions than into a single dimension. Consequently, the single dimensions we study for the baselines should be at an advantage to separate information into different variables.
Scoring disentanglement on the encoder side will not be possible for the baselines above as it requires attention values. To establish that our model effectively tracks syntactic roles, we compare it to a third baseline that locates each syntactic role through its median position across the dataset. This baseline is fairly strong on a language where word order is rigid (i.e configurational language) such as English. We refer to this Position Baseline as PB.
The scores are given for different values of β (Eq. 1). Raising β lowers the expressiveness of latent variables, but yields better disentanglement (Higgins et al., 2017). Following Xu et al. (2020), we set β to low values to avoid posterior collapse. In our case, we observed that the models do not collapse for β < 0.5. Therefore, we display results for β ∈ {0.3, 0.4}. We stop at 0.3 as lower values for β result in poorer generation quality. For our model we report performance for instances with NZ = 4 (ours-4) and NZ = 8 (ours-8).
Results The global disentanglement metrics are reported in Table 1.9
On the decoder side, the Sequence VAE exhibits disentanglement scores in the range of those reported for our model for β = 0.3, and higher for β = 0.4. However, NΓdec shows that it struggles to factor the realizations of different syntactic roles in different latent variables, and the higher score shown for β = 0.4 is accompanied by a lower tendency to separate the information from different syntactic roles. The Transformer VAE baseline assigns different latent variables to the different syntactic role (high NΓdec), but suffers from very low specialization for these latent variables (low Ddec). In contrast, our model is consistently able to separate 3 out of 4 syntactic roles, and while a higher β raises its Ddec, it does not decrease its NΓdec . As ours-8 has more latent variables, this encourages the model to further split the information in each syntactic role between more latent variables10. The fact that ADVAEs perform better than both Sequence VAEs and classical Transformer VAEs shows that its disentanglement capabilities are due to the usage of Cross-Attention to obtain latent variables, and not only to the usage of Transformers. On the encoding side, our models consistently score above the baseline, showing that our latent variables actively follow the syntactic roles.
In Figures 3 and 4, we display the influence matrices Γenc and Γdec for an instance of our ADVAE with NZ = 4 as heatmaps. The vertical axes correspond to the latent variables. As can be seen, our model successfully associates latent variables to verbs and subjects but chooses not to separate direct objects and prepositional objects into different latent variables. Upon further inspection of the same heatmaps for the VAE baseline, it appears that it most often uses a single latent variable for verb and subj, and another for dobj and pobj.
One can also notice in Figures 3 and 4, that the encoder matrix is sparser than the decoder matrix (which is consistent with the higher encoder disentanglement scores in Table 1). This is to be expected as the decoder pθ(s|z) adapts the realizations of syntactic roles to each other after they are sampled separately from p(z). The reason for this is that the language modeling objective requires some coherence between syntactic roles (conjugating verbs with subjects, changing objects that are semantically inadequate for a verb, etc). This co-adaptation, contradicts the independence of our latent variables. It will be further discussed in the following paragraph.
10Results for a larger grid of Nz values are reported in Appendix K, and show that latent variables still clearly relate to syntactic roles, but in groups.
Changing the Realizations of Syntactic Roles Here, we display of few qualitative examples of how the realizations of syntactic roles can be separately changed using an instance of our ADVAE.
As a first example, we generate a sentence from a random latent vector, then resample for each syntactic role the corresponding disentangled latent variable to observe the change on the subsequently generated altered sentence. The results of this manipulation are in Table 211. As can be seen, some examples exhibit changes that only affect the target syntactic role (example 1). However, the model often produces co-adaptations that go past the target syntactic role either for semantic soundness (example 2, resampled verb adapts the object), or simply for lack of generalization from the SNLI data used for training.
A second example we display is a swap of syntactic role realizations between sentences. A few examples are given in Table 3. Similar to Table 2, the model often yields the expected result. Coadaptation is best seen here, as taking a syntactic role to a sentence with which it is incompatible results in unexpected changes (example 3).
Further investigations As this is a first step in this research direction, we conducted this study on a dataset of relatively regular sentences. Running similar experiments on a dataset with more complicated and diverse sentence structures such as in Yelp (Appendix B) results in the same comparative patterns. However, disentanglement scores are much lower. This calls for future iterations to improve upon ADVAE and our evaluation protocol to better model structure in order to scale to User Generated Content (UGC). Our experiments also enabled underlining an inherent issue to syntactic role disentanglement: co-adaptation. The independence between our latent variables causes the decoder pθ(s|z) to correct the incoherence between independently sampled syntactic role realizations. Using structured latent variables to learn relations between syntactic roles seems to be the natural solution to this problem. An investigation of a hierarchical version of the ADVAE (Appendix A) showed, however, that a drop-in replacement of the independent prior with a structured prior is not sufficient in order to absorb co-adaptation into the latent variable model. Our future works will, therefore, also include the investigation of training techniques that can achieve improved results with structured latent variables.

heading: 7 CONCLUSION

text:
We introduce a framework to study the disentanglement of syntactic roles and show that it is possible to learn a representation of sentences that exhibits separation in the realizations of these syntactic functions without supervision. Our framework includes: i) Our model, the ADVAE, which maps syntactic roles to separate latent variables more often than standard Sequence VAEs and with better concentration than standard Transformer VAEs, and allows for the use of attention to study the interaction between latent variables and spans, ii) An evaluation protocol to quantify disentanglement between latent variables and spans both in the encoder and in the decoder.
Our study constitutes a first step in a promising process towards unsupervised explainable modeling and fine-grained control over the predicate-argument structure of sentences. Although we focused on syntactic roles realizations, this architecture as well as the evaluation method are generic and could be applied to other tasks. The architecture could be used at the document level (e.g. disentangling discourse relations), while the evaluation protocol could be applied to other spans such as constituents.
11More Examples are available in Appendix H

heading: A A HIERARCHICAL VERSION OF OUR ADVAE

text:
As we stated, our ADVAE aims to factor sentences into independent latent variables. However, given the dependency structure of sentences, realizations of syntactic roles are known to be interdependent to some degree in general. Therefore one may think that a structured latent variable model would be better suited to model the realizations of syntactic roles. In fact, such a model could absorb the language modeling co-adaptation between syntactic roles. For instance, instead of sampling an object and a verb from p(z) that are inadequate, then co-adapting them through pθ(s|z), a structured pθ(z) could produce an adequate object for the verb. For this experiment, rather than using an independent prior p(z), we use a structured prior pθ(z) = p(z0) ∏L l=1 pθ(z
l|zl−1) where p(z0) is a standard Gaussian, and all subsequent L−1 hierarchy levels are parameterized by learned conditional diagonal Gaussians. The model used for each pθ(zl|zl−1) is shown in Figure 5 below:
We display the results for L = 2 and L = 3 in Table 4. For both models, we set NZ to 4.
The results show lower mean disentanglement scores, and high standard deviations compared to the standard version of our ADVAE. By inspecting individual training instances of this hierarchical model, we found that some instances achieve disentanglement with close scores to those of the standard ADVAE, while others completely fail (which results in the high variances observed in Table 4). Unfortunately, hierarchical latent variable models are notoriously difficult to train (Zhao et al., 2017). Our independent latent variable model is therefore preferable to the structured one due to these empirical results. More advanced hierarchical latent variable training techniques (such as Progressive Learning and Disentanglement (Li et al., 2020b)) may, however, provide better results.

heading: B EXPERIMENTING WITH THE YELP DATASET

text:
We investigated the behavior of our ADVAE of on the user-generated reviews from the Yelp dataset used in Li et al. (2018) using the same procedure we used for SNLI. The length of sentences from this dataset (8.88± 3.64) is similar to the length of sentences from the SNLI dataset. Similar to the experiments in the main body of the paper, we display the disentanglement scores in Table 5, and the influence metrics of one of the instances of our model as heatmaps in Figures 6 and 7.
Table 5: Disentanglement results for the Yelp dataset
Model β Denc NΓenc Ddec NΓdec
Sequence VAE 0.3 - - 0.44(0.09) 2.20(0.45)0.4 - - 1.21(0.06) 2.25(0.50) PB - 0.33(-) 2.00(-) - -
ours-4 0.3 0.48(0.07) 2.00(0.00) 0.18(0.02) 2.50(0.58)0.4 0.54(0.04) 3.00(0.00) 0.23(0.03) 2.40(0.55) ours-8 0.3 0.44(0.04) 3.80(0.45) 0.17(0.04) 2.80(0.84)0.4 0.57(0.26) 3.40(0.55) 0.15(0.10) 2.40(0.89)
Figure 6: Encoder influence heatmap for Yelp(Γenc).
Figure 7: Decoder influence heatmap for Yelp(Γdec).
Although the results show similar trends, they are weaker than what we obtained for SNLI. Given the difference between SNLI and Yelp (displayed in Appendix D) there are two clear reasons for this decrease. The first is that Yelp is a dataset where it is harder to locate the syntactic roles. This is illustrated by the fact that the PB baseline obtains a much lower score. The second is that our syntactic role extraction heuristics are tailored for regular sentences with verbal roots, which subjects the evaluation metrics on Yelp to a considerable amount of noise. Nevertheless, the comparisons between a VAE, an ADVAE, and PB retain the same conclusions, but with lower margins and some overlapping standard deviations.
Through manual inspection of examples, we observed that the various structural characteristics (enumerations, sentences with nominal roots, presence of coordinating conjunctions, etc) were captured by different variables. This indicates that future iterations of our model need to provide ways to separate structural information from content-related information.

heading: C MEASURING THE EFFECT OF LATENT VARIABLES ON THE STRUCTURE OF SENTENCES

text: In Figure 8, for each latent variable and each syntactic role, we report the probability that resampling the latent variable causes the appearance/disappearance of the syntactic role. The instance we use here is the same as the one we use for the heatmaps in the main body of the paper. According to the heatmaps in Figures 3 and 4, latent variable 3 is the one associated with the verb. As can be seen in the present heatmap in Figure 8, this same variable is the one that has the most influence on the appearance/disappearance of direct and prepositional objects, and this is a pattern that proved to be consistent across our different runs. This constitutes empirical justification for our choice of discarding these cases from our decoder influence metrics.

heading: D EXAMPLE SENTENCES FROM YELP AND SNLI AND THEIR CORRESPONDING SYNTACTIC EXTRACTIONS

text: Table 6 shows some samples from SNLI and Yelp reviews. Samples from Yelp Reviews exhibit a clearly higher structural diversity. On the other hand, most SNLI samples are highly similar in structure. Our syntactic role extraction heuristics were tailored for sentences with verbal roots. As a result, it can be seen that they struggle with sentences with nominal roots as well as other forms of irregular utterances present in Yelp. For SNLI, our extractions mostly yield the expected results, allowing for a reliable global assessment of our models.

heading: E TRAINING DETAILS AND HYPER-PARAMETER SETTINGS

text:
Our ADVAE’s hyper-parameters Our model has been set to be large enough to reach a low reconstruction error during the initial reconstruction phase of the training. We use 2-layer Transformers with 4 attention heads and a hidden size of 192. Contrary to Vanilla VAEs, our model seems to perform better with high values of NZ . Therefore, we set our latent vector to a size of 768, and divide it into 96-dimensional variables for our NZ = 8 model and to 192-dimensional latent variables for our NZ = 4 model. No automated hyper-parameter selection has been done afterward.
Sequence VAE hyper-parameters As is usually done for this baseline (Xu et al., 2020), we set both the encoder and the decoder to be 2-layer LSTMs. We run this model for hidden LSTM sizes in [256, 512], and latent vector sizes in [16, 32]. The results for the model scoring the highest Ddec are then reported. Even though selection has been done according to Ddec, we checked the remaining instances of our baselines and they also yielded low NΓdec values.
Transformer VAE hyper-parameters We set the hidden sizes and number of layers for this baseline similarly to ADVAE, since it is also a Transformer. We run this model for latent vector sizes in [16, 32] and display the highest scoring model, as is done for the Sequence VAE.
Training phases All our models are trained using ADAM(Kingma & Ba, 2015) with a batch size of 128 and a learning rate of 2e-4 for 20 epochs. The dropout is set to 0.3. To avoid posterior collapse, we train all our models for 3000 steps with β = 0 (reconstruction phase), then we linearly increase β to its final value for the subsequent 3000 steps. Following Bowman et al. (2016), we also use word-dropout. We set its probability to 0.1.
Evaluation For the evaluation, T dec is set to 2000, and T enc is equal to the size of the test set.

heading: F DISENTANGLEMENT SCORES FOR EACH SYNTACTIC ROLE

text: The full disentanglement scores are reported in Table 7 for the decoder, and in Table 8 for the encoder.

heading: G DISENTANGLEMENT HEATMAPS OVER THE ENTIRE RANGE OF SYNTACTIC ROLES AND POS TAGS

text:
We report decoder and encoder heatmaps for all the syntactic roles following the Stanford Dependencies (SD; De Marneffe & Manning, 2008) annotation scheme of Ontonotes, which was used to train our Spacy2 parser, in Figures 9 and 10. For the sake of extensiveness and to make sure we did not draw results from some parser biases, we also report the same heatmaps but using UDPipe 2.0 (Straka, 2018), which uses UD type annotations12, in Figures 13 and 14. Finally, we also report
12A widely adopted annotation scheme derived from Stanford Dependencies.
heatmaps for interaction with PoS Tags extracted with Spacy2 in Figures 11 and 12. As was done in the main body of the paper, the span corresponding to each syntactic role (in both annotation schemes) was taken to be the series of words included in its corresponding subtree. In contrast, the span corresponding to each PoS tag was just taken to be the tagged word. Results from UD parsing extraction lead to the same conclusions as from our initial SD results.
The instance of our ADVAE for which we display the above heatmaps is the same one for which we display the heatmaps in Figures 3 and 4 in the main body of the paper. As shown in those Figures, it mostly uses variable 3 for verbs, variables 2 for subjects, and variable 1 for objects. The remaining variable (0) also seems to capture some interaction with objects. The heatmaps show that our ADVAE tends to group syntactic roles into latent variables in a way that aligns with the predicative structure of sentences. In fact, variable 2 displays the highest influence on the PoS tag VERB as well as its surroundings as a predicate argument such as adverbs and adverbial phrases. Similarly, latent variable 2 displays a high influence on subjects (nominal or clausal), numeral modifiers, adjectival modifiers, and auxiliaries (for conjugation). Moreover, Variable 1 highly influences the direct and prepositional objects, which we study in the main body of the paper, but also diverse clausal modifiers and obliques which often play similar roles to direct and prepositional objects in a predicate structure.

heading: H ADDITIONAL EXAMPLES OF RESAMPLED REALIZATIONS FOR EACH SYNTACTIC ROLE

text: Table 9 contains a wide array of examples where the latent variable corresponding to each syntactic role is resampled.

heading: I RECONSTRUCTION AND KULLBACK-LEIBLER VALUES ACROSS EXPERIMENTS

text:
The values for the reconstruction loss, the DKL divergence, and the upper bound on perplexity concerning the experiments in the main body of the paper are reported in Table 10. Since our models are VAE-based, one can only obtain the upper bound on the perplexity and not its exact value. These upper bound values are obtained using an importance sampling-based estimate of the negative log-likelihood, as was done in Wu et al. (2020). We set the number of importance samples to 10.
It can be seen that the behavior of ADVAEs is very different from classical Sequence VAEs and Transformer VAEs. On the plus side, they are capable of sustaining much more information in their latent variables as shown by their higher DKL, and they do better at reconstruction. The upper bound estimate of their perplexity is however higher. A high DKL makes it more difficult for the importance sampling-based perplexity estimate to reach the true value of the model’s perplexity. This may be the reason behind the higher values observed for ADVAEs.

heading: J LAYER-WISE ENCODER ATTENTION

text:
In the main body of the paper, we use attention values that are averaged throughout the network. We hereby display the encoder heatmaps obtained by using attention values from the first layer (Fig. 15), the second layer (Fig. 16), or an average on both layers (Fig. 17) for comparison.
As can be seen, the first layer alone provides the most sparse heatmap, and thus, the clearest correspondence between syntactic roles and latent variables. This corroborates the claims of Tenney et al. (2020) about syntax being most prominently processed in the early layers of Transformers.
K ADVAE RESULTS FOR A LARGER GRID OF Nz VALUES
We display in Table 11 the quantitative results of ADVAE on SNLI for Nz in {2, 4, 6, 8}. For ours-2, it is normal that it only separates syntactic role realizations into a maximum of 2 latent variables, as seen from the values of NΓenc and NΓenc , since 2 is its total number of latent variables.
As observed in the main body of the paper, the increase of the number of latent variables used in ADVAE leads to dispatching the influence on the realization of a single syntactic role to multiple latent variables. This in turn,leads to for Denc and Ddec that we observe . In Figures 18 and 19, we
respectively display the encoder and decoder heatmaps of ADVAE with 16 latent variables. As can be seen in these figures, each latent variable still highly specializes in a specific syntactic role. This is seen more clearly on the encoder heatmap due to co-adaptation harming the clarity of the decoder heatmap. This specialization seems to be shared among groups (e.g. variables 4 and 8 specialize in the subject, as indicated by the green squares on the figure). This causes the difference of influence between the most influential variable and the second most influential one to be low, and thus decreases the values of Denc and Ddec.
Table 11: Disentanglement quantitative results on SNLI for a larger grid of Nz values.
Model β Denc NΓenc Ddec NΓdec
ours-2 0.3 2.01(0.07) 2.00(0.00) 0.92(0.06) 2.00(0.00)0.4 0.33(0.15) 1.60(0.55) 0.13(0.09) 1.20(0.45) ours-4 0.3 1.30(0.09) 3.00(0.00) 0.68(0.22) 2.80(0.45)0.4 1.46(0.33) 3.00(0.00) 0.81(0.05) 3.00(0.00) ours-8 0.3 1.36(0.13) 3.40(0.89) 0.60(0.10) 3.00(0.00)0.4 1.44(0.79) 3.40(0.55) 0.63(0.35) 2.80(0.45)
ours-16 0.3 0.60(0.31) 3.60(0.55) 0.33(0.30) 2.60(0.55)0.4 0.65(0.16) 3.40(0.55) 0.56(0.28) 2.60(0.55)
Figure 19: Decoder influence heatmap for ADVAE with 16 latent variables on SNLI (Γdec). Squares with similar colors highlight groups of latent variables that relate to the same syntactic role.