summaries: In this paper, the authors propose an image attribute editing method by manipulating the GAN latent vector. Specifically, this paper uses a pre-trained GAN to synthesize images, a pre-trained regressor to get the image attributes, and trains a network T to find meaningful latent-space directions. It then edits image attributes by modifying the input latent vector, described as z' = z + T(z)ε. The experimental results show that the proposed method performs better than other selected methods to some degree.

summaries: This paper presents a new approach for the semantic image editing task by allowing the controllable transformation on the latent space. Authors proposed to integrate an attribute regression network for training the transformation functions. The local transformation T is learned from a simple MLP conditioned on the latent vector z. Two outputs of the regression module for the original latent vector z and the transformed one z+T*epsilon are used to minimize the cross-entropy loss. Experiments validate the effectiveness of the proposed method in terms of manipulation quality.

summaries: The paper proposes a new simple, yet powerful and alternative method of editing the semantic attributes of images generated using pre-trained GAN models as well as a pre-trained regressors. The approach allows for the manipulation of single or multiple various image attributes, while preserving the identity of the original image in contrast to the baseline method of Shen et. al 2019. The method focuses on the manipulation of the latent space, in contrast to the popular image space editing methods.

abstractText: Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of transformation functions, and apply a content loss and an adversarial loss that encourages the maintenance of image identity and photo-realism. We propose quantitative evaluation strategies for measuring controllable editing performance, unlike prior work, which primarily focuses on qualitative evaluation. Our model permits better control for both singleand multipleattribute editing while preserving image identity and realism during transformation. We provide empirical results for both natural and synthetic images, highlighting that our model achieves state-of-the-art performance for targeted image manipulation.

authors:
Rameen Abdal
Yipeng Qin
Peter Wonka

title: Image2stylegan: How to embed images into the stylegan latent space

venue: In ICCV,

authors:
Binod Bhattarai
Tae-Kyun Kim

title: Inducing optimal attribute representations for conditional gans

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale gan training for high fidelity natural image synthesis

authors:
Q. Cao
L. Shen
W. Xie

title: Vggface2: A dataset for recognising faces across pose and age

venue: In IEEE FG,

authors:
Yunjey Choi
Minje Choi
Munyoung Kim

title: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation

authors:
Yunjey Choi
Youngjung Uh
Jaejun Yoo

title: Stargan v2: Diverse image synthesis for multiple domains

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza

title: Generative adversarial nets

venue: In NeurIPS,

authors:
Erik Härkönen
Aaron Hertzmann
Jaakko Lehtinen

title: Ganspace: Discovering interpretable gan controls

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou

title: Image-to-image translation with conditional adversarial networks

authors:
Ali Jahanian
Lucy Chai
Phillip Isola

title: On the”steerability” of generative adversarial networks

title: The relativistic discriminator: a key element missing from standard gan

authors:
Tero Karras
Timo Aila
Samuli Laine

title: Progressive growing of gans for improved quality, stability, and variation

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

venue: In CVPR,

authors:
Tero Karras
Samuli Laine
Miika Aittala

title: Analyzing and improving the image quality of stylegan

authors:
Marek Kowalski
Stephan J Garbin
Virginia Estellers
Tadas Baltrušaitis
Matthew Johnson
Jamie Shotton

title: Config: Controllable neural face image generation

authors:
Pierre-Yves Laffont
Zhile Ren
Xiaofeng Tao

title: Transient attributes for high-level understanding and editing of outdoor scenes

authors:
H.Y. Lee
H.Y. Tseng
Q. Mao

title: Drit++: Diverse image-to-image translation via disentangled representations

authors:
Bowen Li
Xiaojuan Qi
Thomas Lukasiewicz
Others

title: Controllable text-to-image generation

authors:
Y. Li
C. Fang
J. Yang

title: Universal style transfer via feature transforms

authors:
Ziwei Liu
Ping Luo
Xiaogang Wang

title: Large-scale celebfaces attributes (celeba) dataset

authors:
F. Luan
S. Paris
E. andothers Shechtman

title: Deep photo style transfer

venue: PCVPR,

authors:
Taesung Park
Ming-Yu Liu
Ting-Chun Wang
Jun-Yan Zhu

title: Semantic image synthesis with spatially-adaptive normalization

authors:
Antoine Plumerault
Hervé Le Borgne
Céline Hudelot

title: Controlling generative models with continuous factors of variations

authors:
Olga Russakovsky
Jia Deng
Hao Su

title: Imagenet large scale visual recognition challenge

authors:
Yujun Shen
Bolei Zhou

title: Closed-form factorization of latent semantics in gans

authors:
Yujun Shen
Jinjin Gu
Xiaoou Tang

title: Interpreting the latent space of gans for semantic face editing

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

authors:
Yuri Viazovetskyi
Vladimir Ivashkin
Evgeny Kashin

title: Stylegan2 distillation for feed-forward image manipulation

authors:
Andrey Voynov
Artem Babenko

title: Unsupervised discovery of interpretable directions in the gan latent space

authors:
Sheng-Yu Wang
Oliver Wang
Richard Zhang

title: Cnn-generated images are surprisingly easy to spot.

venue: now. CVPR,

authors:
Sheng-Yu Wang
Oliver Wang

title: Detecting photoshopped faces by scripting photoshop

authors:
Ting-Chun Wang
Ming-Yu Liu
Jun-Yan Zhu

title: High-resolution image synthesis and semantic manipulation with conditional gans

authors:
Po-Wei Wu
Yu-Jing Lin
Che-Han Chang

title: Relgan: Multi-domain image-to-image translation via relative attributes

authors:
Han Zhang
Ian Goodfellow
Dimitris Metaxas

title: Self-attention generative adversarial networks

venue: PMLR,

authors:
Bolei Zhou
Agata Lapedriza
Aditya Khosla

title: Places: A 10 million image database for scene recognition

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola

title: Unpaired image-to-image translation using cycleconsistent adversarial networks

authors:
Jun-Yan Zhu
Richard Zhang
Deepak Pathak

title: Toward multimodal image-to-image translation

heading: 1 INTRODUCTION

text:
Semantic image editing is the task of transforming a source image to a target image while modifying desired semantic attributes, e.g., to make an image taken during summer look like it was captured in winter. The ability to semantically edit images is useful for various real-world tasks, including artistic visualization, design, photo enhancement, and targeted data augmentation. To this end, semantic image editing has two primary goals: (i) providing continuous manipulation of multiple attributes simultaneously and (ii) maintaining the original image’s identity as much as possible while ensuring photo-realism.
Existing GAN-based approaches for semantic image editing can be categorized roughly into two groups: (i) image-space editing methods directly transform one image to another across domains (Choi et al., 2018; 2020; Isola et al., 2017; Lee et al., 2020; Wu et al., 2019; Zhu et al., 2017a;b), usually using variants of generative adversarial nets (GANs) (Goodfellow et al., 2014). These approaches often have high computational cost, and they primarily focus on binary attribute (on/off) changes, rather than providing continuous attribute editing abilities. (ii) latent-space editing methods focus on discovering latent variable manipulations that permit continuous semantic image edits. The chosen latent space is most often the latent space of GANs. Both unsupervised and (self-)supervised latent space editing methods have been proposed. Unsupervised latent-space editing methods (Härkönen et al., 2020; Voynov & Babenko, 2020) are often less effective at providing semantically meaningful directions and all too often change image identity during an edit. Current (self-)supervised methods (Jahanian et al., 2019; Plumerault et al., 2020) are limited to geometric edits such as rotation and scale. To our knowledge, only one supervised approach has been proposed (Shen et al., 2019) – developed to discover semantic latent-space directions for binary attributes. As we show, this method suffers from entangled attributes and often does not preserve image identity during manipulation.
Contributions. We propose a latent-space editing framework for semantic image manipulation that fulfills the aforementioned primary goals. Specifically, we use a GAN and employ a joint sampling strategy trained to edit multiple attributes simultaneously. To disentangle attribute transformations
Original Inversion + Night + Winter + Clouds + Sunny
B as el in e
O ur s
in the latent space of GANs, we integrate a regressor to predict the attributes that an image exhibits. The regressor also permits precise control of the manipulation degree and is easily extended to multiple attributes simultaneously. In addition, we incorporate a perceptual loss (Li et al., 2019) and an adversarial loss that helps preserve image identity and photo-realism during manipulation.
We compare our method to several popular frameworks, from existing image-to-image translation methods (Choi et al., 2020; Wu et al., 2019; Zhu et al., 2017a) to latent space transformation-based approaches (Shen et al., 2019; Voynov & Babenko, 2020). We mention that prior work primarily uses qualitative evaluation like the one in Fig. 1. In contrast, we propose a quantitative evaluation to measure controllability. Both qualitative and quantitative results provide evidence that our approach outperforms prior work in terms of quality of the semantic image manipulation while maintaining image identity.

heading: 2 RELATED WORK

text:
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have significantly improved realistic image generation in recent years (Brock et al., 2018; Jolicoeur-Martineau, 2019; Karras et al., 2017; 2019a;b; Park et al., 2019; Zhang et al., 2018). For this, a GAN formulates a 2-player non-cooperative game between two deep nets: (i) a generator that produces an image given a random noise vector in the latent space, sampled from a known prior distribution, usually a normal or a uniform distribution; (ii) a discriminator whose input is both synthetic and real data, which is to be differentiated.
Semantic image editing seeks to automate image manipulation of semantics. Encouraged by the success of deep nets, recent works have applied deep learning methods for semantic image editing tasks such as style transfer (Li et al., 2017; Luan et al., 2017), image-to-image translation (Choi et al., 2018; 2020; Isola et al., 2017; Lee et al., 2020; Wang et al., 2018; Wu et al., 2019; Zhu et al., 2017a) and discovering semantically meaningful directions in a GAN latent space (Härkönen et al., 2020; Jahanian et al., 2019; Plumerault et al., 2020; Shen et al., 2019; Voynov & Babenko, 2020). Note that our task is an extended version of semantic image editing that requires more comprehensive control to satisfy user-desired operations. Therefore, most aforementioned approaches do not meet
the requirements. Nonetheless, we categorize the approaches most relevant to our task into two groups: (i) image editing via manipulation in image space, and (ii) image editing via latent space navigation.
Image-space editing using GANs directly manipulates an image for targeted editing. Early work (Isola et al., 2017) used GANs to implement semantic translations between two image domains with paired data, e.g., day to night. Follow-up work focused on multi-modal (Bhattarai & Kim, 2020; Choi et al., 2018; 2020; Lee et al., 2020; Wang et al., 2018; Wu et al., 2019) and unpaired image domain translation (Zhu et al., 2017a). In this case, they primarily consider binary (on/off) attribute changes, regardless of whether the process is dynamic, for example, day to night.
Latent-space editing via GANs has received an increasing amount of recent interest. Most prior work focused on identifying semantically meaningful directions in the latent space of GANs, so that shifting latent vectors towards these directions achieves the desired image manipulation. Recent papers found semantics in the latent space of GANs, such as color transformations and camera movements (Jahanian et al., 2019), or face attribute changes (Shen et al., 2019), such as smile. Other work considered unsupervised methods (Härkönen et al., 2020; Voynov & Babenko, 2020) to discover interpretable latent space directions. However, additional challenges inherent to unsupervised manipulation of the latent space arise, and have not been addressed in prior work, e.g., direction quality with regard to degree control and image identity preservation.

heading: 3 METHOD

heading: 3.1 PROBLEM STATEMENT

text: We consider controllable semantic image editing via latent space navigation in GANs. We begin with a fixed GAN model that consists of a generator G and a discriminator D. The input of G is a latent vector z ∈ Rm from a latent space Z . Here, m is the dimensionality of the latent space. GivenN attributes, we aim to discover semantically meaningful latent-space GAN directions, T = {d1, . . . ,dN}, to manipulate the attributes of synthetic images G(z) with an assigned step size ε = {ε1, . . . , εN}, where di ∈ Rm for all i ∈ {1, . . . , N}. In the end, each attribute of an edited image can be changed with the corresponding degree ε fromα = {α1, . . . , αN}, the original attributes of G(z) predicted by a regressor R.

heading: 3.2 PROPOSED APPROACH

text:
We provide an overview of our method in Fig. 2 (a), and further illustrate how our approach differs from prior work in Fig. 2 (b-d).
Overview. As shown in Fig. 2 (a), we employ a GAN model consisting of a generator G and a discriminator D (not shown due to limited space), as well as a regressor R, all of which are pretrained. Our goal is to find latent directions T that provide attribute specific image edits. At each training step, G takes z and an edited latent vector referred to as z′. We follow prior work which suggests that direction vectors in latent space of GANs permit image editing (Jahanian et al., 2019). Formally, given a transformation degree ε, we obtain the latent-space edit as z′ = z + Tε = z + ∑N i=1 εidi. G provides the recovered and edited images G(z) and G(z
′), which are separately processed by the regressor R to predict attribute values. The original attributes of G(z) are α = R(G(z)). The (pseudo-) ground truth and predicted attribute of G(z′) are α′ and α̂′, respectively, where α′ = α+ ε and α̂′ = R(G(z′)).
Intuitively, the goal of T is to transform z by adding semantically meaningful information such that the corresponding output image G(z′) exhibits attribute changes ε from α. In practice, we normalize the range of attribute values to [0,1], i.e., both α and α′ ∈ [0,1]. During training, we maintain the unit range by controlling the given ε. Formally, ε is drawn from a distribution Dε uniform in [−1, 1]N while considering the constraint that 0 ≤ α+ ε ≤ 1. Objective function. To find T we minimize the weighted objective:
min T L , λ1Lreg + λ2Ldisc + λ3Lcontent. (1)
Note, the objective is only used for optimizing T , while the other modules remain fixed. Hyperparameters λ1, λ2, and λ3 adjust the loss terms.
The regression lossLreg assesses whether T performs the transformations indicated by ε. We express the regression loss via a binary cross entropy:
Lreg = Ez∼Z,ε∼Dε [−α̂′ logα′ − (1− α̂′) log (1−α′)]. (2)
Note thatα′ is from the distribution generated by z and ε. Please refer to the appendix C.1 for more details on this distribution.
The second loss term Ldisc is computed using the discriminator D and measures the quality of the generated images, i.e.,
Ldisc = Ez′∼Z′|z [log(1−D(G(z′)))] . (3)
Here we refer to the domain of z′ via Z ′, which is conditioned on z. We write this using Z ′|z. Lastly, we use a content loss Lcontent, often also referred to as perceptual loss. It is designed to estimate the distance between two images and it is employed to maintain the image identity during the transformation. Specifically, we use the content loss term
Lcontent = Ez∼Z,z′∼Z′|z ∑
i∈Dcontent
‖Fi(G(z′))− Fi(G(z))‖22, (4)
where Fi(·) denotes a feature function which extracts intermediate features from images. Dcontent indicates the layers of a pre-trained model which are used as features. We approximate the aforementioned expectations by empirical sampling. We defer algorithm details to Appendix A.
Joint-distribution sampling and training. The regressor operates in a multi-label setting, i.e., each data possesses multiple attributes. To ensure the disentanglement of attribute edits we sample synthetic images from the entire data distribution and find all transformations at once (illustrated in Fig. 2 (b,c) right). In contrast, Shen et al. (2019) prepare training samples on the two opposing data subsets with regards to an attribute, e.g., no clouds vs. many clouds, and find the directions with N one-vs-one classifiers (sketched in Fig. 2 (b,c) left).
Transformation module T . We study two types of transformations T : (i) global and (ii) local. A global transformation T refers to a semantic latent-space transformation identical for all z during inference. This is illustrated via parallel red dashed arrows in Fig. 2 (d) left. These global directions
are commonly used (Härkönen et al., 2020; Jahanian et al., 2019; Shen et al., 2019; Viazovetskyi et al., 2020). However, a globally identical direction might not serve all data. In contrast, a local transformation T is a function of z which provides various directions for different z (in Fig. 2 (d) right, shown via non-parallel red arrows). Formally, di = f iθ(z), where f i θ is implemented via a deep net. In Sec. 4 we show that the local transformation T succeeds on failure cases of the global transformation.

heading: 4 EXPERIMENTS

text:
We introduce datasets, implementation details, evaluation metrics, and show results next.
Datasets. We evaluate the proposed approach on two types of datasets: (i) face datasets – FFHQ (Karras et al., 2019a), CelebA (Liu et al., 2018) and CelebA-HQ (Karras et al., 2017), commonly used in prior work (Härkönen et al., 2020; Karras et al., 2017; 2019a;b; Shen et al., 2019;
Viazovetskyi et al., 2020). (ii) natural scene datasets – Transient Attribute Database (Laffont et al., 2014) and MIT Places2 data (Zhou et al., 2017), which contain attributes suitable for continuous semantic image editing. We briefly introduce the scene datasets:
• Transient Attribute Database (Laffont et al., 2014): It contains 8,571 scene images with 40 attributes in 5 categories including lighting (e.g., “bright”), weather (e.g., “cloudy”) , seasons (e.g., “winter”), subjective impressions (e.g., “beautiful”), and additional attributes (e.g., “dirty”). Each attribute is annotated with a real-valued score between 0 and 1, where 0 indicates absence of the attribute.
• MIT Places2 data (Zhou et al., 2017): Using the provided category annotations (i.e., indoor/outdoor and natural/artificial), we select the natural outdoor scenes, obtaining a total of 144,543 images.
Implementation details. We choose λ1 = 10, λ2 = 0.05, λ3 = 0.05 in Eq. (1) and compute the perceptual loss using the conv1 2, conv2 2, conv3 2, conv4 2 activations in a VGG-19 network (Simonyan & Zisserman, 2014) pre-trained on the ImageNet dataset (Russakovsky et al., 2015). We train T for 50k iterations with a batch size of 4. An Adam optimizer is used with a learning rate of 1e-4. For the latent space Z in the proposed method we use the W space of StyleGAN2 and the Z space of PGGAN. For the attribute regressor R, we adopt a ResNet-50 (He et al., 2016) for attribute prediction on the Transient Attribute (Laffont et al., 2014) and the CelebA (Liu et al., 2018) data. The last fully connected layer in the ResNet-50 is replaced by a linear layer with an output dimension of 40. We train the regressors for 500 epochs and use the weights with the best validation mean squared error (MSE) on CelebA and the best test MSE on the Transient Scene Database. The GAN follows the StyleGAN2 architecture and is pretrained with 200k iterations on a union of the two natural scene datasets using a resolution of 256 × 256. The training batch size is 32 and an Adam optimizer is used with a learning rate of 2e-3. For the face datasets, we use FFHQ pre-trained weights1 of StyleGAN2, and CelebA-HQ weights of PGGAN2.
Baselines. We compare our approach to several popular image-to-image translation approaches (Choi et al., 2020; Wu et al., 2019; Zhu et al., 2017a) and latent space direction discovery methods (Shen et al., 2019; Voynov & Babenko, 2020). Since Choi et al. (2020); Shen et al. (2019); Zhu et al. (2017a) cannot deal with continuous attributes, we split the data into 2 domains using a threshold value of 0.5 for each of the 40 attributes. Afterward, the models are trained on 2 contrast domains, e.g., with or without a certain attribute. We use the official code of all baseline methods and rigorously follow their training steps. Note that work by Voynov & Babenko (2020) is unsupervised, i.e., the latent-space directions are human interpreted. To avoid bias during the selection of directions, we use the attribute regressor to automatically identify the most significant directions that can edit predetermined attributes. The details of preparing the baselines are given in Appendix C.2.
Evaluation metrics. There is no good numerical metric to evaluate image editing (Shen et al., 2019; Voynov & Babenko, 2020). In an attempt to address this concern, we automate quantitative evaluation based on a property that editing of attributes should maintain image identity. To achieve this, we employ a popular image identity recognition model3 pre-trained on the VGGface2 data (Cao et al., 2018). Cosine similarity is used to represent the similarity between paired original and edited images (Cao et al., 2018). In addition, we evaluate changes of the other independent attributes using the pre-trained regressor.

heading: 4.1 RESULTS ON NATURAL SCENE DATASETS

text:
Comparison to image-to-image translation: As shown in Fig. 3 (a-d), we observe image-to-image translation to perform poorly when editing image details (e.g., removing clouds). Moreover, sometimes artifacts (orange spots in (a) and (b)) are introduced. In contrast, our model shows overall compelling transfer performance on all these attributes.
Comparison to latent-space translation: Inspecting Fig. 1, we observe our approach to improve upon work by Shen et al. (2019) in attribute edit quality and number of artifacts. Similarly, in
1https://github.com/rosinality/stylegan2-pytorch 2https://pytorch.org/hub 3https://github.com/ox-vgg/vgg face2
Fig. 4, we notice that our method performs well in attribute editing, and has a better ability to preserve image identity when adding “night” and “snow.” We show additional analysis and results in Appendix D and E.1.
Quantitative evaluation: We follow a user study proposed by Kowalski et al. (2020) to evaluate controllability of our scene edits. Concretely, given an image I = G(z), we generate paired images (I+, I−) that have opposing values for attribute n. 40 users took the test to answer: (a) the presence and strength of the attribute n in I+, and (b) if I+ and I− are identical when ignoring the attribute. Both types of questions have options on a 5-level scale from 0 (“not at all”) to 1 (“totally”). Ideally, the response to question (a) should be 1 while we expect replies for (b) to be 0. We test performance of image editing with “clouds”, “sunrise&sunset”, “night” and “snow” attributes. A question example and statistical results over 50 image pairs are shown in Fig. 5. Statistical evidence supports our claim that the proposed approach performs well with regards to image edits while maintaining image identity.

heading: 4.2 RESULTS ON FACE DATASETS

text:
Manipulation results on StyleGAN2: The comparison on real images shown in Fig. 1 suggests that our method works well for attribute edits. Further, the synthetic image edit results in Fig. 4 indicate that our edits are disentangled, while the baselines unexpectedly add “glasses” when aging the face. To increase the task difficulty, we edit real images with multiple attribute changes simultaneously. Results are summarized in Fig. 6, which highlights the controllability of our edits.
Quantitative evaluation: We measure the changing degrees of the other independent attributes and the image identity when editing attributes with various degrees ε̂. Here ε̂ is a predicted changing degree on a target attribute byR. For a thorough comparison, we evaluate the performance on 3 seg-
ments according to the absolute value of ε̂, i.e., |ε̂| in the range of (0, 0.3], (0.3, 0.6] and (0.6, 0.9]. Evaluation on “smile,” “hair color,” and “smile+hair color” attributes are shown in Tab. 1 and Tab. 2. The “hair color” attribute includes “blond” and “black” colors where we average the results on both cases. For multiple attribute editing, “smile + hair color,” we evaluate the case that both the two targeted attributes change within the |ε̂| range. We use around 1k original images, generate 10k edited images with regard to each target attribute, and repeat the experiment 3 times. The averaged results and the standard deviations presented in Tab. 1 and Tab. 2 suggest that our model outperforms the baselines with regard to disentanglement and image identity preservation.
Manipulation results on PGGAN: For PGGAN, we use local transformations parameterized by three-layer perceptrons with leaky ReLU activations. We show the detailed MLP architecture in Appendix B.1. Editing results in Fig. 7 suggest that the proposed approach is able to continuously edit faces with various semantic attributes. We also study the differences between local and global directions in Fig. 11. Particularly, a global T (in Fig. 11 (a)) is a matrix where each column i is given by di. For a local T (in Fig. 11 (b)) di is given by a fully-connected layer with identical input and output dimensions. Fig. 11 (c) and (d) use the MLP components with the aforementioned structure. We use Lreg to train T in (c), while the total loss L is used for (d). The results in Fig. 11 indicate that a local T , with either linear or MLP structure works well for face attribute edits. Moreover, our loss L stabilizes the training process of T .

heading: 5 CONCLUSION

text:
We propose an effective approach to semantically edit images by transferring latent vectors towards meaningful latent space directions. The proposed method enables continuous image manipulations with respect to various attributes. Extensive experiments highlight that the model achieves state-ofthe-art performance for targeted image manipulation.
Acknowledgements: This work is supported in part by NSF under Grant No. 1718221, 2008387, 1934986 and MRI #1725729, NIFA award 2020-67021-32799, UIUC, Samsung, Amazon, 3M, and Cisco Systems Inc. (Gift Award CG 1377144). We thank Cisco for access to the Arcetri cluster. We thank Amazon for EC2 credits.

heading: A. ALGORITHM

text:
The overall procedure of the proposed method in Algorithm 1, taking local transformation T as an example.
Algorithm 1 Training Procedure Input: A pre-trained GAN with G, D, and input noise distribution z ∼ Z; a pre-trained regressor
R for predictions on N attributes; an initialized T ; max iteration number M . 1: for iteration m = 1, . . . ,M do 2: Sample random noise z ∼ Z , and ε 3: Compute internal attributes for a synthetic image, i.e., α = R(G(z)) 4: Compute the actual shift value δ = CLIP (α+ ε, (0, 1))−α 5: Compute the transformed latent vector z′ = z + T (z)δ 6: Compute I ′ = G(z′), α′ = α+ δ 7: Compute attribute predictions α̂′ = R(I ′) 8: Compute the loss L 9: Update T
10: end for Return: T

heading: B. METHOD DETAILS

text:
B.1. METHOD ON PGGAN
We apply a multi-layer perceptron (MLP) to parameterize a latent space path on PGGAN. Fig. 9 shows the MLP architecture that we use.
In practice, we find that normalizing the direction vector di helps preserve image identity during manipulation on PGGAN. Formally, the transformed latent vector can be written as
z′ = z + λε di ‖di‖ , (5)
where λ is a weight adjusting the direction scale. We set λ = 3.

heading: C. EXPERIMENTAL DETAILS

text:
C.1 THE EXPECTATION IN LREG
We define the regression loss Lreg as Lreg = Ez∼Z,ε∼Dε [−α̂′ logα′ − (1− α̂′) log (1−α′)], (6) where α′ is from the distribution generated by z and ε. We now discuss how we derive the distribution of α′. To begin, z is sampled from Z and ε is sampled from Dε. The other variables are generated by z and ε:
z′ = z + Tε
α′ = R(G(z)) + ε
α̂′ = R(G(z′)).
As a result, the distribution of α′ is dependent on Z and Dε.
C.2 UNSUPERVISED LATENT-SPACE EDITS OF GANS
Voynov & Babenko (2020) learn a matrix where each column is a direction. Yet, Voynov & Babenko (2020) require a human to interpret the learnt directions. To avoid bias during the selection of directions in our comparison, we use the pre-trained attribute regressor to automatically identify the most significant directions. Concretely, we first generate 100 images and edit them with target attributes of various degrees. Next, we use the regressorR on the edited images to predict attributes. We choose for an attribute edit the direction on which all the edited images have the overall highest changing response on the target attribute.

heading: D. ADDITIONAL ANALYSIS

text:
D.1 GAN INVERSION PERFORMANCE
We use an optimization-based GAN inversion method (Abdal et al., 2019) to find optimal latent vectors that can best reconstruct the real images via the generator. To examine the effect of GAN inversion performance on our approach, we terminated the GAN inversion approach at different training steps, i.e., 500 and 4,000 iterations. We show averaged reconstruction MSE loss in Tab. 3. In this case, we reconstructed 20 real face images and edited their “Smile” and the “Blond hair” attribute with 10 different degrees ε, i.e., 200 images in total for each attribute editing. Quantitative evaluation on image identity preservation and numerical changes on the other semantically independent attributes for the reconstructed face images are given in Tab. 4. The results in Tab. 4 suggest that the performance of the GAN inversion method affects our method to some degree. Visualized inversion and editing results are shown in Fig. 10. The qualitative results suggest that our method still works remarkably well on the worse inversion image.
D.2 ABLATION STUDY
We conduct an ablation study with single- and joint-distribution training strategies in our method. Specifically, single-distribution sampling refers to learn one attribute direction at a time. In contrast, joint-distribution training means to train multiple attribute directions simultaneously. Fig. 11 shows the visualized results with the two training strategies on the face and the scene dataset. We observe that the model with the single-distribution training strategy generates more unexpected changes as the manipulation degree is getting large, e.g., darker scene colors by the model trained for a single attribute, shown in Fig. 11 (b).

heading: E. ADDITIONAL RESULTS

text:
E.1 SCENE IMAGE EDITS ON STYLEGAN2
We show more results of our method on continuous scene image edits in Fig. 12 - 15. The semantically edited attributes include “clouds” (Fig. 12), “brightness” (Fig. 13), “snow” (Fig. 14), and “summer” (Fig. 15).

heading: F. BROADER IMPACT

text:
From an application perspective, our method is effective and efficient with regard to image manipulation and photo-realism, which we hope will contribute to 2D and 3D controllable image editing tasks. Moreover, usage of deep nets to learn mappings between spaces is still not well understood, e.g., mappings between low-dimensional space and image space in classification and generation tasks. We hope our method provides inspiration for representation learning and a first step for a new view with regard to deep net interpretability.
Obviously, we are aware of the dangers of automated image manipulation. Similar to deepfake tasks whose aim is to produce fabricated images and videos that appear to be real, improper use of image manipulation approaches might raise negative issues with regard to information security, property, etc. Beyond that, edited image detection techniques Wang et al. (2019a;b) are proposed recently to avoid the aforementioned issues, which promotes growth in both domains.

title: ENJOY YOUR EDITING: CONTROLLABLE GANS


summaries: The paper proposes an improvement to sequence generative adversarial networks (GAN) to cope with the common training issues of GANs. For the sake, the paper combines Gumbel-Softmax based GAN, relativistic discrimination  function with  the matching of mean representations of true and generated samples in a latent feature space. This feature statistics alignment allows to leak information from the discriminator to the generator as the used features are extracted from the discriminator network. Experimental evaluations on synthetic and real datasets show the improvement achieved by the proposed method over existing sequence generation networks.

summaries: The paper addresses the task of improving GANs for sequence generation and proposed a method based on the relativistic discriminator. The proposed method employs a Feature Statistics Alignment (FSA) paradigm to reduce the gap between real and generated data distributions. It relies on the relativistic discriminator for "coarse" differences and FSA for "fine-grained" differences between real and generated data distributions. It is evaluated on synthetic and real datasets, and it significantly outperforms the baselines. It also outperforms baselines on human evaluation based on the acceptance, grammaticality, and meaningfulness of the generated sentences.

summaries: This paper proposes a new GAN-based text generation method that incorporates feature statistics alignment and gumbel-softmax for reparameterization to deal with mode collapse and unstable training. For feature statistics alignment, the authors design two methods such as mean square and mean distance alignments. They evaluate the proposed method on a synthetic dataset, MS COCO caption, and EMNLP2017 WMT news dataset, comparing them with RL-based and non RL-based models. With extensive experiments including ablation studies, the proposed method show promising results.

abstractText: Generative Adversarial Networks (GAN) are facing great challenges in synthesizing sequences of discrete elements, such as mode dropping and unstable training. The binary classifier in the discriminator may limit the capacity of learning signals and thus hinder the advance of adversarial training. To address such issues, apart from the binary classification feedback, we harness a Feature Statistics Alignment (FSA) paradigm to deliver fine-grained signals in the latent high-dimensional representation space. Specifically, FSA forces the mean statistics of the fake data distribution to approach that of real data as close as possible in a finite-dimensional feature space. Experiments on synthetic and real benchmark datasets show the superior performance in quantitative evaluation and demonstrate the effectiveness of our approach to discrete sequence generation. To the best of our knowledge, the proposed architecture is the first that employs feature alignment regularization in the Gumbel-Softmax based GAN framework for sequence generation.

authors:
Peter Anderson
Xiaodong He
Chris Buehler
Damien Teney
Mark Johnson
Stephen Gould
Lei Zhang

title: Bottom-up and top-down attention for image captioning and visual question answering

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Samy Bengio
Oriol Vinyals
Navdeep Jaitly
Noam Shazeer

title: Scheduled sampling for sequence prediction with recurrent neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Tong Che
Yanran Li
R. Zhang
R. Devon Hjelm
W. Li
Y. Song
Yoshua Bengio

title: Maximumlikelihood augmented discrete generative adversarial

venue: networks. ArXiv,

authors:
Liqun Chen
Shuyang Dai
Chenyang Tao
Dinghan Shen
Zhe Gan
H. Zhang
Yizhe Zhang
L. Carin

title: Adversarial text generation via feature-mover’s

venue: distance. ArXiv,

authors:
Xinlei Chen
Hao Fang
Tsung-Yi Lin
Ramakrishna Vedantam
Saurabh Gupta
Piotr Dollár
C Lawrence Zitnick

title: Microsoft coco captions: Data collection and evaluation

venue: server. arXiv preprint arXiv:1504.00325,

authors:
William Fedus
Ian Goodfellow
Andrew M Dai

title: Maskgan: Better text generation via filling in the

venue: arXiv preprint arXiv:1801.07736,

authors:
Ian J. Goodfellow
Jean Pouget-Abadie
M. Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron C. Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In NIPS,

authors:
Xiaodong Gu
Kyunghyun Cho
Jung-Woo Ha
Sunghun Kim

title: Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder

venue: arXiv preprint arXiv:1805.12352,

authors:
Ishaan Gulrajani
F. Ahmed
Martı́n Arjovsky
Vincent Dumoulin
Aaron C. Courville

title: Improved training of wasserstein gans

authors:
Jiaxian Guo
S. Lu
Han Cai
W. Zhang
Y. Yu
J. Wang

title: Long text generation via adversarial training with leaked

venue: information. ArXiv,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Ari Holtzman
Jan Buys
Li Du
Maxwell Forbes
Yejin Choi

title: The curious case of neural text degeneration

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparameterization with gumbel-softmax

title: The relativistic discriminator: a key element missing from standard

venue: gan. ArXiv,

authors:
Pei Ke
Fei Huang
Minlie Huang
Xiaoyan Zhu

title: Araml: A stable adversarial training framework for text generation

title: Convolutional neural networks for sentence classification

venue: arXiv preprint arXiv:1408.5882,

authors:
Matt J. Kusner
José Miguel Hernández-Lobato

title: Gans for sequences of discrete elements with the gumbel-softmax

venue: distribution. ArXiv,

authors:
Mike Lewis
Yinhan Liu
Naman Goyal
Marjan Ghazvininejad
Abdelrahman Mohamed
Omer Levy
Ves Stoyanov
Luke Zettlemoyer

title: Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension

authors:
Jiwei Li
Will Monroe
Tianlin Shi
Sébastien Jean
Alan Ritter
Dan Jurafsky

title: Adversarial learning for neural dialogue generation

venue: arXiv preprint arXiv:1701.06547,

authors:
Kevin Lin
Dianqi Li
Xiaodong He
M. Sun
Zhengyou Zhang

title: Adversarial ranking for language

venue: generation. ArXiv,

authors:
Chris J. Maddison
A. Mnih
Y. Teh

title: The concrete distribution: A continuous relaxation of discrete random variables

venue: ArXiv,

authors:
Weili Nie
Nina Narodytska
Ankit B. Patel

title: Relgan: Relational generative adversarial networks for text generation

venue: In ICLR,

authors:
Kishore Papineni
Salim Roukos
Todd Ward
Wei-Jing Zhu

title: Bleu: a method for automatic evaluation of machine translation

venue: In Proceedings of the 40th annual meeting of the Association for Computational Linguistics,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

venue: arXiv preprint arXiv:1910.10683,

authors:
Tim Salimans
Ian J. Goodfellow
W. Zaremba
Vicki Cheung
A. Radford
Xi Chen

title: Improved techniques for training

venue: gans. ArXiv,

authors:
Adam Santoro
Ryan Faulkner
David Raposo
Jack Rae
Mike Chrzanowski
Theophane Weber
Daan Wierstra
Oriol Vinyals
Razvan Pascanu
Timothy Lillicrap

title: Relational recurrent neural networks

venue: In Advances in neural information processing systems,

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms

venue: arXiv preprint arXiv:1707.06347,

authors:
Thomas Scialom
Paul-Alexis Dray
Sylvain Lamprier
Benjamin Piwowarski
Jacopo Staiano

title: Coldgans: Taming language gans with cautious sampling strategies

authors:
Yonghui Wu
Mike Schuster
Zhifeng Chen
Quoc V Le
Mohammad Norouzi
Wolfgang Macherey
Maxim Krikun
Yuan Cao
Qin Gao
Klaus Macherey

title: Google’s neural machine translation system: Bridging the gap between human and machine translation

venue: arXiv preprint arXiv:1609.08144,

authors:
Lantao Yu
W. Zhang
J. Wang
Y. Yu

title: Seqgan: Sequence generative adversarial nets with policy gradient

venue: In AAAI,

authors:
Yizhe Zhang
Zhe Gan
K. Fan
Z. Chen
Ricardo Henao
Dinghan Shen
L. Carin

title: Adversarial feature matching for text generation

authors:
Wangchunshu Zhou
Tao Ge
Ke Xu
Furu Wei
M. Zhou

title: Self-adversarial learning with comparative discrimination for text

authors:
Yaoming Zhu
S. Lu
L. Zheng
Jiaxian Guo
W. Zhang
J. Wang
Y. Yu

title: Texygen: A benchmarking platform for text generation models

venue: The 41st International ACM SIGIR Conference on Research Development in Information Retrieval,

heading: 1 INTRODUCTION

text:
Unsupervised sequence generation is the cornerstone for a plethora of applications, such as machine translation (Wu et al., 2016), image captioning (Anderson et al., 2018), and dialogue generation (Li et al., 2017). The most common approach to autoregressive sequence modeling is maximizing the likelihood of each token in the sequence given the previous partial observation. However, using maximum likelihood estimation (MLE) for sequence modeling is inherently prone to the exposure bias problem (Bengio et al., 2015), which results from the discrepancy between the training and inference stage: the generator predicts the next token conditioned on its previously generated ones during inference but conditioned on its prefix ground-truth tokens during training, yielding accumulative mismatch along with the increment of generated sequence length.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) can serve as an alternative to models trained by MLE, which have achieved promising results in generating sequences of discrete elements, in particular, language sequences (Kusner & Hernández-Lobato, 2016; Yu et al., 2017; Lin et al., 2017; Guo et al., 2018; Fedus et al., 2018; Nie et al., 2019; de Masson d’Autume et al., 2019; Zhou et al., 2020; Scialom et al., 2020). GANs consist of two competing networks: a discriminator that is trained to distinguish the generated samples from real data, and a generator that aims to generate high-quality samples to fool the discriminator.
Although having succeeded in avoiding exposure bias issues, GANs still suffer from some intrinsic problems, such as mode dropping, reward sparsity, and training instability. To enrich the informativeness of the discriminator’s training signal, several approaches have been proposed by measuring the latent features, such as feature distribution matching (Zhang et al., 2017; Chen et al., 2018) and comparative discriminators (Lin et al., 2017; Zhou et al., 2020). Zhang et al. (2017) and Chen et al. (2018) leveraged feature matching mechanism by minimizing the kernel-based moment-matching metric, such as Maximum Mean Discrepancy and Earth-Mover’s Distance, between encoded features. However, merely adopting feature matching in lieu of the original learning signal may lack some guiding feedback at the initial stage of training.
Another approach is to compare the finite latent features with comparative discriminators like ranker and relativistic discriminator. Lin et al. (2017) maintained that the binary classification in the discriminator network limits the learning capacity of tasks because the diversity and richness are circumscribed by the degenerated distribution. RankGAN (Lin et al., 2017) replaced the binary clas-
sifier with a pairwise feature ranker by comparing the similarities between sample features in the latent space. SAL (Zhou et al., 2020) classified the encoded features of constructed pairwise training examples into three categories, i.e., better / worse / indistinguishable. Nevertheless, adopting a comparative discriminator could provide the fine-grained signals for updating the generator network and may require some coarse credits for further improvements.
In this work, we propose to improve the GANs for sequence generation by jointly considering both the feature statistics matching and relativistic discriminator to serve as fine-grained and coarse learning signals respectively. We leverage the Feature Statistics Alignment (FSA) paradigm to embed the latent feature representations in a finite feature space and force the distribution of generated samples to approach the real data distribution by minimizing the distance between their respective feature representation centroids. Intuitively, matching the mean feature representations of fake and real samples could make the two data distributions closer. Besides, the relativistic discriminator (Jolicoeur-Martineau, 2019) is employed to measure the comparative information between generated and real sequences and empirically to show the effectiveness during the model training.
Our experimental results illustrate the effectiveness of FSA techniques and large batch size to alleviate the gradient vanishing problem and stabilize the training process in comparison with the vanilla Gumbel-Softmax GANs. Besides, our models could generate discrete text sequences with high quality in terms of the semantic coherence and grammatical correctness of language, as evaluated with crowdsourcing. Furthermore, we empirically demonstrate that the proposed architecture overshadows most existing models in terms of quantitative and qualitative evaluation. To the best of our knowledge, the proposed framework is the first to adopt the statistics feature alignment paradigm in the Gumbel-Softmax based GAN framework for discrete sequence generation.

heading: 2 ADVERSARIAL SEQUENCE GENERATION

text:
Adversarial sequence generation has attracted broad attention for its properties to solve the exposure bias issue suffered with maximum likelihood estimation (MLE) for generating language sequences. Based on the game theory, its goal is to train a generator network G(z;θ(G)) that produces samples from the data distribution pdata(x) by decoding the randomly initialized noise z (i.e., standard normal distribution) into the sequence x = G(z;θ(G)), where the training signal is provided by the discriminator networkD(x;φ(D)) that is trained to distinguish between the samples drawn from the real data distribution pdata and those produced by the generator. The minimax objective of adversarial training is formulated as:
min θ(G) max φ(D)
Ex∼pdata [ logD(x;φ(D)) ] + Ez∼pz [ log ( 1−Dφ(D)(G(z;θ(G))) )] . (1)
Despite the impressive results of GANs in the sequence generation (Yu et al., 2017; Gulrajani et al., 2017; Scialom et al., 2020), there are still several fundamental issues in the GAN training: (a) Training instability, which arises from the intrinsic nature of minimax games in GANs; (b) Mode dropping, which is the fact that GANs only generate samples with limited patterns in the real data distribution instead of attending to diverse patterns (Chen et al., 2018); (c) Reward sparsity, which is because that it is easier to train the discriminator than the generator, making it difficult to acquire the instructive feedback (Zhou et al., 2020).
Due to the non-differentiability of gradients caused by sampling operations between the generator and discriminator for sequence generation, the majority of previous works have resorted to reinforcement learning (RL) heuristics with Monte Carlo search to collect the credits from the discriminator. The usage of RL may further deteriorate the instability of model training and exacerbate the reward sparsity problem. Gumbel-Softmax relaxation has proven to be an alternative to RL techniques (Kusner & Hernández-Lobato, 2016; Nie et al., 2019). How to efficiently train GANs with the Gumbel-Softmax trick still remains under-explored. Therefore, we utilize the Gumbel-Softmax reparameterization instead of conventional policy gradients in our framework.

heading: 3 METHODOLOGY

text:
As illustrated in Fig. 1, standard GANs employ the real/fake binary classifier as the discriminator, which is prone to be overtrained in comparison with the generator (Salimans et al., 2016). To prevent the discriminator from overfitting and further stabilize the training process, we propose to leverage the FSA techniques and relativistic discriminator by comparing the fake and real distributions from two different aspects.
Compared with conventional GANs, the proposed framework enjoys the following advantages: (a) In the earlier training stage, the relativistic discriminator could estimate the probability that how much better the real data is in comparison with the generated samples. This could provide the relatively “coarse” credits to the generator. (b) When it comes to the later training stage and the quality of generated samples becomes high, FSA measures the “fine-grained” difference between latent features of fake and real data batches, precluding the discriminator from being overly confident. In other words, the FSA can be regarded as a kind of regularization for adversarial training.

heading: 3.1 FEATURE STATISTICS ALIGNMENT

text:
Intuitively, aligning the statistics of embedded feature representations increases the opportunities to capture the various modes of the data distribution. For brevity, we utilize the first-order mean statistics in our framework and leave the higher-order statistics for future work.
Denoting the feature extractor as Fω parameterized by ω, a minibatch of real data samples as x with the batch size of N , we propose two variants of FSA formulations, which calculate the mean squared difference and Euclidean distance between the minibatch centroids of real and fake feature representations. To reduce the parameter amount, we share the weights between the discriminator and feature extractor of FSA.
Mean Squared Alignment (MSA) We take the mean squared difference between the centroids of fake and generated distributions as the feature alignment metric:
LMSA = ∥∥Ex∼pdata[Fω(x)]− Ez∼pz[Fω(G(z;θ(G)))]∥∥22 (2)
= ∥∥ 1 N N∑ i=1 Fω(xi)− 1 N N∑ i=1 Fω(G(zi;θ (G))) ∥∥2 2 . (3)
Mean Distance Alignment (MDA) Another intuitive approach is to calculate the distance between two sample centroids, which is equivalent to the square root of MSA mathematically:
LMDA = ∥∥Ex∼pdata[Fω(x)]− Ez∼pz[Fω(G(z;θ(G)))]∥∥2 (4)
= √ LMSA. (5)
By forcing the mean statistics of fake data to be close to the real samples, the generator could receive more informative signals in the training process. It is worth noting that the large batch size is helpful to reduce the variance of small mini-batches.

heading: 3.2 RELATIVISTIC DISCRIMINATOR

text:
We consider the Relativistic Discriminator (Jolicoeur-Martineau, 2019) to take into account the relative confidence that the given real data is more realistic than the randomly sampled fake data. In the standard GAN, defining the discriminator as D(x) = sigmoid(H(x)), where H(·) represents the non-transformed layer before the final non-linearity. The objectives for the discriminator and generator in terms of the Relativistic Discriminator are defined as:
LRD = −Ex∼pdata,z∼pz [log a ( H(x)−H(G(z;θ(G))) ) ], (6)
LRG = −Ex∼pdata,z∼pz [log a ( H(G(z;θ(G)))−H(x) ) ], (7)
where a represents the activation function to be relativistic (we use sigmoid function in our experiments), RD and RG denote the loss terms for the discriminator and generator respectively.

heading: 3.3 OVERALL TRAINING OBJECTIVES

text:
Previous works using moment matching techniques to support the training on both the discriminator and generator (Zhang et al., 2017; Chen et al., 2018). However, the generator is always more difficult to train than the discriminator, resulting in the training instability and reward sparsity. To relieve these issues, we only adopt the FSA techniques to enhance the generator but keep the objective of the discriminator unchanged. This could pass more informative signals only to the generator, and also prevent the discriminator to be overtrained.
Therefore, the overall training objectives for the proposed framework are defined as: LD = LRD, (8) LG = LRG + LFSA, (9)
where LFSA takes the form of LMSA and LMDA as Eq. 2 and Eq. 4. The goal of the discriminator is to maximize the gap between the generated and real data, whereas the generator jointly considers two different aspects simultaneously: it not only competes with the discriminator by maximizing the gap in terms of the relativistic signals but takes into account the additional leaked feature information from the discriminator. The idea of leaked features from the discriminator is similar to LeakGAN (Guo et al., 2018). The FSA term on the RHS can also be regarded as a dynamic regularizer for the sequence generator.

heading: 3.4 TRAINING WITH DISCRETE SEQUENCE

heading: 3.4.1 GUMBEL-SOFTMAX DISTRIBUTION

text:
Conventional GANs for generating discrete sequences are inherently unable to backpropagate the gradient through samples due to the non-differentiable sampling from a categorical distribution. Gumbel-Softmax distribution (Jang et al., 2017; Maddison et al., 2017) was proposed to deal with this issue by smoothly annealed to approximate the categorical distribution.
Denoting the output probabilities π1, π2, · · · , π|V |, where |V | represents the output vocabulary size in the generator, the Gumbel-Max trick (Maddison et al., 2014) can be parameterized as:
yi = one hot ( argmax
i [gi + log πi]
) for i = 1, · · · , |V |, (10)
where {gi|i = 1, · · · , |V |} are i.i.d from the Gumbel(0,1) distribution, that is, gi = − log(− log ui) with ui is drawn from a standard uniform distribution Uniform(0,1). one hot represents the |V |- dimensional one hot encoding.
Gumbel-Softmax approximates the non-differential argmax operation using the softmax function:
ŷi = exp
( (log(πi) + gi)/τ )∑|V | j=1 exp ( (log(πj) + gj)/τ
) , for i = 1, · · · , |V |, (11) where τ denotes the softmax temperature to modulate the exploitation and exploration during training. When τ approaches too high, the approximation is nearly equiprobable, encouraging the generator to explore different options. In contrast, the lower τ could discourage the exploration and tend to exploit during training. In particular, when τ → 0, ŷi approaches the result of one hot operator as in Eq. 10, whereas ŷi will degenerated into a uniform distribution when τ →∞.

heading: 3.4.2 ARCHITECTURE AND ADVERSARIAL TRAINING

text:
RNN Generator Due to the free-running mode of GAN’s generator, it is unsuitable to adopt the transformer-based models due to the non-recurrence nature. Thus, we investigate the Recurrent Neural Network (RNN) based models, such as Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997), and Relational Memory Core (RMC) (Santoro et al., 2018).
CNN Discriminator & Feature Extractor We use the convolutional neural networks (CNN) architecture (Kim, 2014) as the discriminator and feature extractor for input sequences. We employ the multi-channel convolution using multiple filters with various window sizes to extract the distinct n-gram features, followed by a max-over-time pooling operation to gather the most salient features, i.e., features with the highest value for each feature map.
Adversarial Training Algorithm Alg. 1 illustrates the overall training process of the proposed framework. The Relativistic Discriminator and the generator could reach the Nash Equilibrium when the generator could fool the discriminator into accepting its output as being true. Since the discriminator is easy to be overtrained, we do not pretrain the discriminator but only pretrain the generator using MLE for few epochs.
Algorithm 1 Adversarial Training with Feature Statisitcs Alignment 1: Require: generator Gθ; discriminator Dφ ; samples of real data S; generator training step g;
discriminator training step k; the generator pretraining epochs m. 2: Pretrain Gθ using MLE on S for m epochs 3: repeat 4: for g steps do 5: Sample a minibatch from real data S 6: Generate a minibatch of samples xg ∼ Gθ 7: Update Gθ via Eq.(9) 8: end for 9: for k steps do
10: Sample a minibatch from real data S 11: Sample a minibatch from the generated data 12: Train the discriminator Dφ by Eq.(8) 13: end for 14: until convergence

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETTING

text:
Similar to (Lin et al., 2017; Guo et al., 2018; Nie et al., 2019; Zhou et al., 2020), we evaluate the proposed framework based on the Texygen benchmark platform (Zhu et al., 2018) for adversarial text generation. Experiments were conducted on the synthetic and real datasets: (a) synthetic data, which is generated by an oracle single-layer LSTM as in (Yu et al., 2017); (b) MS COCO Image Caption dataset (Chen et al., 2015); (c) EMNLP WMT 2017 News dataset (Guo et al., 2018). Table 1 summarizes the statistics of benchmark datasets for evaluation.
For the synthetic data experiments, we utilize a single-layer LSTM initialized by the standard normal distribution as the oracle model, which is used to generate 10,000 samples of length 20 and 40
respectively as the real samples. We use the negative log-likelihood (NLL) under the oracle data distribution for evaluation, termed NLLoracle. For the real data experiments, the BLEU score (Papineni et al., 2002) serves as a metric to evaluate the n-gram statistics overlapping on the whole dataset.
To measure the diversity of generated samples, the NLL of the generator (denoted as NLLgen) is used by computing the NLL of reference samples in the test set by the generator. Considering that BLEU scores always focus on the local text statistics and may be insufficient for evaluating the overall quality of texts, we conducted additional human evaluation via crowdsourcing on the generated samples of all comparison models. See Appendix A for more experimental details.
We compare the proposed framework with the MLE baseline and other state-of-the-art models, involving SeqGAN (Yu et al., 2017), RankGAN (Lin et al., 2017), LeakGAN (Guo et al., 2018), RelGAN (Nie et al., 2019), and Self-Adversarial Learning (SAL) (Zhou et al., 2020).

heading: 4.2 EXPERIMENTAL RESULTS

heading: 4.2.1 SYNTHETIC DATA

text:
Table 2 illustrates the performance of different models on NLLoracle. Since our experiments achieved better results using MSA instead of MDA on synthetic data, we only report the optimal results with MSA in Table 2. Our models outperform other prevalent adversarial models in terms of the generated sample quality, demonstrating the effectiveness of our proposed method.
In practice, we found that the LSTM generator could outperform the RMC generator for producing the sequence with the length of 20, whereas RMC generators exceed LSTMs for long sequence generation. This may be due to the fact that LSTMs may forget the long-term dependencies with the sequence length increases, but the self-attention based relational memory cell in RMC could mitigate the issues using interactive memory slots. This demonstrates that the proposed model could produce samples with high quality. As to the diversity, our method achieves competitive NLLgen score compared with baselines models (see Appendix C.1 for the detail).

heading: 4.2.2 MS COCO DATASET

text:
To further test the performance on the real data, we run and evaluate our model on MS COCO image caption datasets. The data and preprocessing remain the same as in Texygen (Zhu et al., 2018). Empirically, we found that RMC achieves better results in terms of the long sequences, and thus reports the results using the RMC generator if not otherwise specified.
Table 3 exhibits the final results of the BLEU and NLLgen scores on different comparison models. Our models reported in Table 3 set the softmax temperature as 0.1 for the model with MSA and that with MDA. Notably, our model shows the significant improvement on previous methods, consistently overshadowing the state-of-the-art models in terms of the sample quality (indicated by BLEU scores) while maintaining the diversity (indicated by NLLgen).

heading: 4.2.3 EMNLP2017 WMT NEWS DATASET

text:
Table 4 presents the considerable improvements of our model (w.r.t. quality) on EMNLP2017 WMT News dataset, with the temperature of 1 for models with MSA and MDA. The maximum length in the EMNLP2017 dataset is 51, greatly challenging the generation task.
It can be observed that only RankGAN and LeakGAN slightly outrank the MLE in terms of the NLLgen, which may be due to the fact that feature ranking or leakage information could pass more internal information from the discriminator to the generator, and thereby smoothly assist the training process of the generator. By leveraging the FSA methods, our model greatly outperforms these models and yielding the long sequences with promising qualities.
Apart from the automatic quantitative evaluation, we also conducted human evaluation on the MS COCO Image Captioning dataset. We randomly sampled 100 sentences for each model and the real data, then asked 12 different people to score them on the scale of 1-5 with anonymizing the model’s identity. Please see Appendix. C.2 for more details of human evaluation.
Our models received the highest score in comparison with other models, where the model with MSA receives higher credits than that with MDA. By manually going through the generated samples, we found that models with MSA tend to generate long sentences with higher quality but run into the mode dropping issue. In contrast, models with MDA posses a better trade-off between the diversity and quality of samples, yielding promising results in comparison with previous models.

heading: 4.3 DISCUSSION

text:
Fig. 2 (Left) illustrates the ablation test of FSA in terms of the BLEU-4 score (See Appendix D.2 for all the results). It can be observed that the increasing trend for models with FSA outreached that w/o feature alignment mechanism and our models achieve superior performance in the later training stage. The utilization of FSA leads to significant performance gains, as it could provide the consecutive “fined-grained” smoother learning signals to update the generator.
Fig. 3 demonstrates the relative importance of each component of our models with an ablation test. The usage of FSA results in the most significant performance gain, followed by Gumbel-Softmax, and large batch size. A large batch size can boost the performance due to the variance reduction, accurate estimates of feature statistics, and stabilizing effect during the adversarial training (see Fig. 2 (Right)).
As to the Gumbel-Softmax trick, we conduct experiments on various temperature values and find that a fined-tuned temperature hyperparameter could account for the second important factor to contribute to the overall performance boost. Please check Appendix D.3 for the detailed results w.r.t the Gumbel-Softmax temperature.
5 RELATED WORK
There has been a large category of GANs for sequence generation, which heavily rely on the RL paradigm. SeqGAN (Yu et al., 2017) regards the sequence generation as a Markov decision making process, estimates the rewards via Monte Carlo search, and trains the generator with policy gradient. RankGAN (Lin et al., 2017) and SAL (Zhou et al., 2020) replace the binary classifier in the discriminator as comparative discriminators to take into account the re-
lation between constructed pair samples. MaliGAN (Che et al., 2017) utilizes the information in the discriminator as an additional source of training signals on the MLE objective to reduce the variance of gradients. LeakGAN (Guo et al., 2018) leaks the intermediate feature information via a manager to guide the generator, which is inspired by hierarchical RL. ColdGAN (Scialom et al., 2020) integrates the advance of importance sampling, Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017), and nucleus sampling (Holtzman et al., 2019) for finetuning the pretrained T5 (Raffel et al., 2019) and BART (Lewis et al., 2019).
Another approach applies non-RL methods for adversarial sequence generation by either approximating the categorical sampling or directly using the continuous latent representation. TextGAN (Zhang et al., 2017) uses feature matching techniques via a kernelized discrepancy in the Reproducing Kernel Hilbert Space. FMGAN (Chen et al., 2018) proposes to match the feature distributions using a Feature-Mover’s Distance. Similar to our proposed model, both of them apply annealed soft-argmax for approximation. ARAML Ke et al. (2019) utilizes Reward Augmented Maximum Likelihood by sampling from the stationary distribution to acquire rewards. However, none of them adopt the Gumbel-Max trick to reparameterize the categorical sampling. Besides, they applied feature matching as the training objectives of both the discriminator and generator, whereas we only apply the feature statistics matching to modulate the generator. Gumbel-Softmax (GS) GAN (Kusner & Hernández-Lobato, 2016) and RelGAN (Nie et al., 2019) prove the effectiveness of Gumbel-Softmax on unsupervised sequence generation. DialogeWAR Gu et al. (2018) employs a GS GAN within the latent variable space for dialogue generation. However, improving the training of GS GANs still remains an open problem. Our model aims to promote the GS GAN with the proposed FSA and other techniques to boost the training of language GANs. We also report a list of other techniques we tried but proved to be unsuccessful or unnecessary in Appendix B.

heading: 6 CONCLUSION

text: We propose an adversarial training framework for discrete sequence generation, by leveraging the advance of Feature Statistics Alignment and Gumbel-Softmax relaxation. Our model empirically shows superior performance in terms of the quantitative and human evaluation. In the future, it would be a promising direction to extend the proposed model to conditional text generation, such as text style transfer.

heading: A EXPERIMENTAL DETAILS

text:
A.1 TRAINING DETAILS
Generator The input embedding dimension for the generator is set to 32. As for the LSTM generator, we use the single-layer LSTM with the hidden dimension of 32. Then adopt a linear transformation to get the logits at each time step, and iteratively feed the sampled output tokens into the generator at the next time step. As for the RMC generator, we follow the experimental settings as (Nie et al., 2019), setting the memory size as 256, memory slots as 1, attention head number as 2. After the one-layer RMC, a linear projection is applied to get the output logits at each step.
CNN Discriminator The input embedding dimension for the discriminator is set to 64. We adopt the filter size of {2, 3, 4, 5} with the number of 300 channels for each. A max-over-time pooling is adopted after the convolution layer. Afterward, a highway layer that is identical to SeqGAN (Yu et al., 2017) is used followed by a linear transformation with the dimension of 100. Finally, apply a linear transformation to get the final logits. The feature extractor for Feature Statistics Alignment shares the identical architecture and weights with the CNN discriminator, and the leaked feature dimension is set to 100.
Optimization We use Adam optimizer with β1 = 0.9 and β2 = 0.999. The initial learning rate for the generator was set to 1e-2 and 1e-4 for pretraining and adversarial training. We set the initial learning rate as 1e-4 for the discriminator during the adversarial training. To prevent overfitting, we clip the gradients of parameters whose L2 norm exceeds 5.
Training Procedure We conduct experiments to finetune the following experiments: the batch size of {32, 64, 128}, the Gumbel-Softmax temperature τ ∈ {1, 0.5, 0.1, 0.01.0.001}. The training steps of generator and discriminators are set to g = 1 and d = 5, respectively. The generator is pretrained for 150 epochs before adversarial training. Finally, the optimal batch size is set to 128 for both synthetic and real datasets. It is worth noting that we also test the batch size to 256, which requires too much GPU resource but do not show obvious improvement.

heading: B NEGATIVE RESULTS

text:
Here we list some approaches that we tried but proved unsuccessful:
• Using Mogrifier LSTM as the generator, which achieves similar results as vanilla LSTMs on the synthetic data.
• Using a Wasserstein loss instead of current Relativistic Discriminator. Not as stable as current solutions.
• Using the Transformer model as the discriminator. It achieves unsatisfied results with the current experimental settings.
• Using interleaved training instead of two-stage training, i.e., adversarial training after pretraining. It is unsuccessful to train the generator for 15 iterations after one iteration using MLE.
• Using top-k sampling and nucleus sampling, instead of the argmax in the Gumbel-Max trick. This does not always boost the final performance.
• Using a hinge loss on the discriminator. This did not improve over the current relativistic loss.

heading: C EVALUATION DETAILS

text:
C.1 NLLGEN ON SYNTHETIC DATA
Table 6 reports the NLLgen metric of comparison models. It can be seen that the proposed model with RMC generator achieves comparative results in comparison with baselines in terms of both short and long texts.
C.2 HUMAN EVALUATION
Acceptance (i.e. if a sentence is acceptable), grammaticality (i.e., if a sentence is grammatically correct), and meaningfulness (i.e., if a sentence makes sense) are three main standards for the text quality evaluation. Please note that any minor text formatting issues which will not negatively influence the understanding and correctness of the sentences (e.g., punctuation, capitalization, spelling errors, extra spaces) can be ignored. Please also note: a sentence consists of less than 10 words should get one point deducted. Table 7 below gives more detailed criteria.
It is worth to mention that the human evaluation is used to measure the quality of generated sentences rather than the diversity.
Table 7: The human evaluation scale from 1 to 5 with corresponding criteria and example sentences.
Scale Criterion & Example
5 - Excellent Grammatical, acceptable, and meaningful.For example, “a man is carving under yellow planes .” 4 – Good Include 1 to 2 tiny grammatical errors, and the whole sentence is mostly acceptable andmeaningful. For example, “two giraffe standing in front of them .” 3 – Fair Include major grammatical errors, but the whole sentence is still acceptable and makingsense. For example, “a kitchen with a grill roll from him .” 2 – Poor Include severe grammatical errors, and the whole sentence does not make sense, butsome parts are still acceptable. For example, “a motorcycle on a paved road on the freeway .” 1 - Unacceptable It is basically a string of words with random order and totally ungrammatical. The entiresentence does not make any sense. For example, “a city .”
C.3 HUMAN EVALUATION ANALYSIS
The model with MSA tends to generate grammatically correct sentences, and the sentences tend to be longer. For example, “A man is sitting on a motorcycle on a busy street, in a city.” Though it produces the samples with high quality by human evaluation, however, it does not solve the mode dropping collapse.
In contrast, models with MDA tend to generate a more variety of sentences rather than repeated ones. Most sentences are grammatically correct and meaningful. They follow the SVO sentence structure with Preposition Phrase (PP) placed at the acceptable position in a sentence. Even though some of the auxiliary or main action verbs are missing, the meaning of each sentence can still be understandable and making sense. There is no obvious mode dropping issues according to the generated samples of MDA.

heading: D DETAILED RESULTS OF ABLATION STUDY

text:
D.1 IMPACT OF FEATURE STATISTICS ALIGNMENT
See Fig. 4 for the results of the ablation study on FSA techniques.
D.2 IMPACT OF LARGE BATCH SIZE
Fig. 5 shows the training curve of our model on various batch sizes. It is observed that the increase of batch size could provide the performance boost, due to the variance reduction of gradients and the stability of adversarial dynamics.
D.3 IMPACT OF GUMBEL-SOFTMAX TEMPERATURE
Fig. 6 reports the BLEU scores with different temperatures on MS COCO dataset. It can be seen that a suitable τ could greatly advance the automatic evaluation scores.

heading: E GENERATED SAMPLES ON REAL DATASET

text:
E.1 GENERATED SAMPLES ON MS COCO DATASET
Table 8 displays samples of generated samples from all baseline models and references on MS COCO dataset.
Table 9 shows the randomly sampled sentences from the proposed models generated on MS COCO Dataset.
E.2 GENERATED SAMPLES ON EMNLP2017 WMT NEWS DATASET
Table 10 presents the random sampled sentences from our models generated on EMNLP2017 WMT News Dataset.
Table 9: Randomly sampled 10 samples trained on MS COCO dataset, with MDA (top row) and MSA (bottom row). We can observe that the phrases “a man” and “motorcycle” occur too many times in the samples generated by MSA-enhanced models. Thus, models with MSA lack the diversity of text whereas those with MDA perform well on both the diversity and quality.
a woman sitting on a bench next to a bicycle parked near a sidewalk . a person is riding a motorcycle on a city street with a woman standing on the back of it . a cat sitting on a motorcycle on a car seat wanting to get out the car window . a woman is sitting on a bench next to a bicycle . a man is riding a motorcycle on a city street with parked vehicles . a person is riding a motorcycle on a closed intersection with a group of people standing on the side of the road with people standing around it . a woman is leaving a bathroom with a toilet and a sink . a little girl is sitting on a bench next to a bicycle . a little girl is sitting on the back of a storefront bathroom with his orange bike next to him . a person sitting on a bench next to a bicycle parked near a wall .
a man is sitting on a motorcycle on a busy street , in a city . a man is sitting on a motorcycle on a crowded street , near a building with a bike nearby . a man is sitting on a motorcycle on a crowded street , near a bicycle . a photo of a small plane sitting on a busy runway . a cat sitting on a motorcycle on a crowded street near a parking lot . a group of people sitting on a motorcycle on a side of a road . a man is sitting on a motorcycle on a crowded street , near a street . a person sitting on a motorcycle on a sidewalk next to a bicycle . a man sitting on a motorcycle on a busy sidewalk next to a bicycle . a man is sitting on a motorcycle on a busy street , in a city .
Table 10: Randomly sampled 10 samples trained on EMNLP2017 WMT News dataset, with MDA (top row) and MSA (bottom row).
the british people would have to commit to travelling to europe and has been a priority for the first time in a decade . his priority finally becomes a hope for the police to be informed by the attack and is not on the scene . there is more than a year before the start of the day after the european ’ s first , but it was not in the last two years . it is not something that is about to be a 15 . 6 percent in the fourth quarter , according to a report from the third of the week . “ i ’ ve been a part of our last two years , ” he said in a statement from the bbc ’ s today . now that ’ s why we have to be a part of the UK . i ’ m not saying that was the first of the kind of people who were in the wrong but it is true that it is yet to be determined . he will be a key for the first time in a decade , and has helped to stop the spread of the decade - over the past year . so what if that ’ s the reason that they can be within the last two years . it was one of the most in the first quarter , but it was the first of the nearly two months since the start of the first of the day .
if you ’ re a new , and you have to be a part of the team . in a fox news , she has already been a major despite a conflict in the world . when you ’ re in the world , when they are growing . but i ’ ve been a part of the group for christmas . instead , there is no evidence to suggest that the united kingdom . the department of health and the enforcement and defense agencies last Thursday . if you ’ re the only in the world , and i ’ m sure . he will be more than it was a time when he was in the control of the water . sometimes it was a number of the people who were in the back of the way . it is an easy for trump ; the other at the same .