summaries: The authors consider the problem of generating diverse translations from a neural machine translation model. This is a very interesting problem and indeed, even the best models lack meaningful diversity when generating with beam-search. The method proposed by the authors relies on prefixing the generation with discrete latent codes. While a good general approach, it is not new (exactly the same general approach that was used in the "Discrete Autoencoders for Sequence Models" [1] paper, https://arxiv.org/abs/1801.09797, for generating diverse translations, which is not cited directly but a follow-up work is cited, though without mentioning that a previous work has tackled the same problem). Also, the authors rely on additional supervised data (namely POS tags) which has no clear motivation and seems to cause a number of problems -- why not use a purely unsupervised approach when it has already been demonstrated on the same problem? Additionally, the authors compare to a weak translation baseline on small data-sets, making it impossible to judge whether the results would hold on a larger data-set. So the following ablations and comparison to baselines are missing:

summaries: The authors propose modeling structural diversity of translations by conditioning the generation on both the source sentence and a latent encoding of the overall structure (captured by simplified part-of-speech tags). Specifically, they first train a conditional autoencoder to learn a latent code optimized towards reconstructing the tag sequence. They then prefix the inferred latent code to the target sentence before generation. A diversity metric which measures pairwise BLEU scores between beam items is also proposed. Experiments show that the latent codes lead to greater structural diversity as well as marginally improved translation results when combined with beam search.

abstractText: Planning is important for humans when producing complex languages, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the global sentence structure ahead of translation. Our approach learns discrete structural representations to encode syntactic information of target sentences. During translation, we can either let beam search to choose the structural codes automatically or specify the codes manually. The word generation is then conditioned on the selected discrete codes. Experiments show that the translation performance remains intact by learning the codes to capture pure structural variations. Through structural planning, we are able to control the global sentence structure by manipulating the codes. By evaluating with a proposed structural diversity metric, we found that the sentences sampled using different codes have much higher diversity scores. In qualitative analysis, we demonstrate that the sampled paraphrase translations have drastically different structures.

authors:
Roee Aharoni
Yoav Goldberg

title: Towards string-to-tree neural machine translation

venue: In ACL,

authors:
Mauro Cettolo
Jan Niehues
Sebastian Stüker
Luisa Bentivogli
Marcello Federico

title: Report on the 11th iwslt evaluation campaign, iwslt 2014

venue: In Proceedings of the International Workshop on Spoken Language Translation,

authors:
Bo Dai
Dahua Lin
Raquel Urtasun
Sanja Fidler

title: Towards diverse and natural image descriptions via a conditional gan

venue: IEEE International Conference on Computer Vision (ICCV),

authors:
Akiko Eriguchi
Yoshimasa Tsuruoka
Kyunghyun Cho

title: Learning to parse and translate improves neural machine translation

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Zhiting Hu
Zichao Yang
Xiaodan Liang
Ruslan Salakhutdinov
Eric P. Xing

title: Toward controlled generation of text

venue: In ICML,

authors:
Unnat Jain
Ziyu Zhang
Alexander G. Schwing

title: Creativity: Generating diverse questions using variational autoencoders

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparameterization with gumbel-softmax

authors:
Lukasz Kaiser
Aurko Roy
Ashish Vaswani
Niki Parmar
Samy Bengio
Jakob Uszkoreit
Noam Shazeer

title: Fast decoding in sequence models using discrete latent variables

authors:
Philipp Koehn
Hieu Hoang
Alexandra Birch
Chris Callison-Burch
Marcello Federico
Nicola Bertoldi
Brooke Cowan
Wade Shen
Christine Moran
Richard Zens
Chris Dyer
Ondrej Bojar
Alexandra Constantin
Evan Herbst. Moses

title: Open source toolkit for statistical machine translation

authors:
Jiwei Li
Will Monroe
Daniel Jurafsky

title: A simple, fast diverse decoding algorithm for neural generation

authors:
Chris J. Maddison
Andriy Mnih
Yee Whye Teh

title: The concrete distribution: A continuous relaxation of discrete random variables

authors:
Alexander H. Miller
Adam Fisch
Jesse Dodge
Amir-Hossein Karimi
Antoine Bordes
Jason Weston

title: Key-value memory networks for directly reading

authors:
Maria Nadejde
Siva Reddy
Rico Sennrich
Tomasz Dwojak
Marcin Junczys-Dowmunt
Philipp Koehn
Alexandra Birch

title: Predicting target language ccg supertags improves neural machine translation

venue: WMT,

authors:
Toshiaki Nakazawa
Manabu Yaguchi
Kiyotaka Uchimoto
Masao Utiyama
Eiichiro Sumita
Sadao Kurohashi
Hitoshi Isahara

title: Aspec: Asian scientific paper excerpt corpus

title: A method for unconstrained convex minimization problem with the rate of convergence o (1/k2)

venue: In Doklady an SSSR,

authors:
Graham Neubig
Yosuke Nakata
Shinsuke Mori

title: Pointwise prediction for robust, adaptable japanese morphological analysis

venue: In ACL, pp

authors:
Shrimai Prabhumoye
Yulia Tsvetkov
Ruslan Salakhutdinov
Alan W. Black

title: Style transfer through back-translation

title: The handbook of speech production

authors:
Rico Sennrich
Barry Haddow
Alexandra Birch

title: Neural machine translation of rare words with subword units

authors:
Tianxiao Shen
Tao Lei
Regina Barzilay
Tommi S. Jaakkola

title: Style transfer from non-parallel text by cross-alignment

venue: In NIPS,

authors:
Felix Stahlberg
Eva Hasler
Aurelien Waite
Bill Byrne

title: Syntactically guided neural machine

venue: translation. CoRR,

authors:
Shuangzhi Wu
Dongdong Zhang
Nan Yang
Mu Li
Ming Zhou

title: Sequence-to-dependency neural machine translation

heading: 1 INTRODUCTION

text:
When humans speak, a planning phase exists when we want to say in a particular structure or style. Linguists have found particular speech errors or behaviors that indicate that speakers are sometimes planning ahead although they may not notice it (Redford, 2015). This planning phase is considered important both at the sentence and discourse levels. For example, we may decide to use the “In order to ..., he ..., because ...” structure before telling a long story. Therefore, it is meaningful to explore the benefits for a neural language generation model to have a such a planning phase.
In this work, we are interested in letting the model plan the sentence structure before producing words. If this can be done, then we may be able to select a structure beforehand, or obtain multiple sentences with diverse structures. Such features are useful for language generation tasks such as machine translation, as users may desire to know how a sentence can be translated with different structures. Therefore, we focus on machine translation task when developing our models.
In contrast to human, a neural machine translation (NMT) model does not have a planning phase when it is asked to generate a sentence. One can argue that the planning is performed in the hidden layers. However, as a source sentence can have multiple valid translations with different structures. When we use the cross-entropy loss to train the model, it has to assign probabilities to all possible translations with different structures. Therefore, the structural information remains probabilistic but not deterministic in the neural network. In other words, the model is unaware of the “big picture” of the output sentence before translation. Consequently, we can not control the sentence structure beforehand.
To obtain diverse sentences, conventional NMT models allow one to sample translations from theN - best list produced by beam search, but the sampled translation results usually share similar syntactic structures.
One naive approach to explicitly control the output structure is to prefix translations with their structural (e.g., part-of-speech) tags. In this case, the NMT model has to predict the tags before generating words according to these tags. Thus, we can obtain structural diverse translations by using different tags. This approach is referred to as tag planning. In our experiments, we found this approach results in significant degradation in translation quality because errors in the planning phase directly
affect the correctness of the translation. To overcome this problem, we propose learning a discrete structural representation (i.e., discrete codes) that capture structural variations without negatively affecting the translation quality.
Fig. 1 shows our proposed two-phase approach for training the NMT model. In the first phase, we parse target sentences to obtain their structural tags. In this work, we use simplified POS tags to represent the coarse structure of a sentence. Next, we design a coding model that encodes information about the structural variations into discrete codes. In the second phase, we prefix each target sentence in the training data with the structural codes produced by the coding model. As shown in Fig. 1, we use a special token “〈eoc〉” to denote the end of the codes. Finally, we train an NMT model on the enhanced dataset; the model architecture does not require any modification.
To maintain translation performance, our coding model eliminates partial information from the structural tags that is unambiguous when the source sentence is given. For example, for the Spanish sentence shown in Fig. 1, it is unambiguous that the translation will contain a noun, a pronoun, and a verb. Such obvious information does not require planning, and the translation quality may be impacted seriously if it is predicted wrongly. In this example, the structural choice between “he is hired by our company” or “our company hired him” is ambiguous. In this work, the coding model is designed to capture only the ambiguous part of the sentence structure when the source sentence is known. We call this approach discrete planning, as the planning component is actually the artificially created discrete codes.
Please note that the planning component can also be a continuous latent vector, which requires a discriminator to train the model in order that the latent captures disentangled structural information. However, such a discriminator is difficult to implement. A detailed discussion is given in related work. Our approach of learning discrete structural representation has the advantage of simplicity, and no modification to the NMT model is required.
The contributions of this work can be summarized as follows:
1. We propose a novel approach to plan the sentence structure with a discrete structural representation. By carefully learning the discrete codes to capture pure structural variation, the translation quality remains intact.
2. We are able to obtain drastically different translations by manipulating the codes. To qualitatively evaluate the diversity, we propose a structural diversity metric. The evaluation results show that our approach is capable of producing sentences with highly diverse structures.

heading: 2 LEARNING STRUCTURAL CODES

text: This section discusses the motivation and methodology for learning discrete structural codes. First, we describe a naive approach that extracts the global sentence structure from part-of-speech (POS) tags. Then, we describe a coding model that learns discrete structural codes and the advantages of doing so.

heading: 2.1 EXTRACTING GLOBAL SENTENCE STRUCTURES

text:
To plan the structures of translations, we need to obtain the structural representation of a sentence. However, it is extremely difficult for a neural network to automatically realize and extract the sentence structure. Therefore, we consider utilizing the results of POS tagging or syntactic parsing. However, most parsing results including POS tags contain detailed structural information, which is not desirable. For example, “destroy” and “break down” are interchangeable, but they have different POS tags. Such minor differences do not affect the global structure of a sentence. Therefore, we need a structural representation that captures the sentence structure at a coarse level.
In this work, we extract such a global sentence structure through a simple two-step process that simplifies the POS tags:
1. Remove all tags other than “N”, “V”, “PRP”, “,” and “.”. Note that all tags begin with “N” (e.g. NNS) are mapped to “N”, and tags begin with “V” (e.g. VBD) are mapped to “V”.
2. Remove duplicated consecutive tags.
The following list gives an example of the process:
Input: He found a fox behind the wall. POS Tags: PRP VBD DT NN IN DT NN .
Step 1: PRP V N N . Step 2: PRP V N .
It should be noted that other parse results can also be potentially useful to represent the global structure; this is left for future works to explore. In our experiments, we found simplified POS tags to be effective and convenient for implementation.

heading: 2.2 CODE LEARNING

text:
Although we can directly condition the word prediction using the structural tags, we found that errors during structural prediction significantly degraded the translation performance. To mitigate their impact, we propose learning a set of discrete codes that capture only the ambiguous structural choice when the source sentence is given.
Code Encoder Given the sentence pair (X,Y ), let the structural tags of the target sentence be S1, ..., ST , the encoder computes the logits of N softmax vectors:
h̄t = LSTM(E(St), h̄t+1;φe) , (1)
[C̃1, ..., C̃N ] = softplus(f(h̄1;φh)) , (2)
where the tag sequence S1, ..., ST is firstly encoded using a backward LSTM (Hochreiter & Schmidhuber, 1997). E(·) denotes an embedding function, and f(·) denotes a linear transformation. Then, we compute a set of continuous vectors C̃1, ..., C̃N , which are latterly discretized into approximated one-hot vectorsC1, ..., CN using Gumbel-Softmax reprarameterization trick (Jang et al., 2016; Mad-
dison et al., 2016) as:
Ci = g(S, ;φ)i (3)
= softmaxτ (log C̃i + gumbel( )) . (4)
Here, τ is the temperature of the softmax. The computation of Gumbel-softmax relies on the Gumbel noise, which is defined as gumbel( ) = − log(− log( )). As both C̃i and are vectors in Eq. 4, the log(·) here is an element-wise operation. The random variable is sampled from a uniform distribution U(0, 1). In Eq. 3, g(S, ;φ) represents the encoder function, which is used to produce the codes once its parameter φ is trained.
Decoder In the decoder, we combine the information from X and C to initialize a decoder LSTM that sequentially predicts the structural tags S1, ..., ST :
st = LSTM(E(Xt), st+1; θx) , (5) h0 = f([C1, ..., CN ]; θh) + s1 , (6) ht = LSTM(E(St−1), ht−1; θd) , (7)
where [C1, ..., CN ] is a concatenation of N Gumbel-softmax code vectors. Note that only ht is computed with a forward LSTM. Finally, we predict the probability of emitting each tag St with
P (St|S<t, X,C) = softmax(fout(ht; θout)) . (8)
Loss Function The architecture of the code learning model is depicted in Fig. 2, which can be seen as a sequence auto-encoder with an extra context input X to the decoder. The loss function for optimizing the entire model has the following form given one training pair:
L(φ, θ) = E ∼U(0,1) [ − logP (S|X, g(S, ;φ); θ) ] , (9)
= E ∼U(0,1) [ T∑ t=1 − logP (St|S<t, X, g(S, ;φ); θ) ] . (10)
The codes are limited to have only a few bits of information capacity in our experiments. In order to minimize the loss function, the codes produced by the encoder g(·) has to capture the structural information contained in S that cannot be inferred by X .

heading: 2.3 NMT WITH STRUCTURAL PLANNING

text: Once the coding model is trained, the Gumbel-softmax code vectors C1, ..., CN shall be very close to one-hot vectors. We can extract N discrete codes by applying argmax on the softmax vectors, and prefix the target sentences in the training data with the codes, resulting in (X,C;Y ) pairs. We connect the codes and target sentences with a “〈eoc〉” token. To evaluate the performance of structural planning with discrete codes, we train a regular NMT model with the modified dataset. The codes are removed from the translation results after decoding. Similarly, we can evaluate the tag planning model using (X,S;Y ) pairs.

heading: 3 RELATED WORK

text:
Our work learns a discrete representation of structure and leverages it obtain diverse translations, which is related to topics including style transfer and diverse language generation. Though differing in purpose, our approach is also related to NMT models with syntactic constraint.
Style Transfer for Natural Language As our task learns the structural representation, the problem setting is closely related to style transfer for language generation tasks. Hu et al. (2017) simultaneously learns a disentangled representation along with entangled representations within the same latent vector. The disentangled representation is trained with a style-specific discriminator. Shen et al. (2017) learns a cross-aligned auto encoder to tackle the style transfer problem without parallel text data. Prabhumoye et al. (2018) utilizes back-translation to create a style agnostic representation of sentence. Then multiple style transfer decoders is trained to generate sentences in specific styles
according to the latent representation. The parameters are optimized with style classifiers using adversarial loss.
The main difference between our task and previous works in style transfer is in the objectives. In the aforementioned works, the styles have finite categories. For example, the label for sentiment style can only be “positive” or “negative”. In our task setting, the objective is to learn a representation that captures the global sentence structure, which has infinite variations.
Theoretically, we can still train a variational auto-encoder (VAE) that learns a categorical latent code or a continuous latent variable with a Gaussian prior to capture the structure. We can extend the VAE model described in Hu et al. (2017) with an enhanced discriminator qD(c|x) to map the global structure of a sentence x to the space of a latent variable c, and ensure that sentences with similar structures are mapped to same or close latent representations. However, as the structural information is highly entangled with the utterances, such a discriminator is fairly difficult to implement.
Diverse Language Generation As generating diverse text has its application in tasks such as conversation generation, several existing works are presented to improve the diversity of language generation. Similar to Hu et al. (2017), Dai et al. (2017) generates diverse image captions with a conditional GAN, whereas Jain et al. (2017) generates diverse questions with a VAE.
Some creative approaches are also proposed on this topic. Li et al. (2016) improves the diversity of generation by modifying the scoring function in beam search. A hyperparameter that controls the diversity is optimized by reinforcement learning. Xu et al. (2018) learns K shared decoders, each is conditioned on a trainable pattern embedding. In each iteration, only the decoder with lowest cross-entropy receives training signal.
Existing works on language generation care about using creative vocabulary. Both Li et al. (2016) and Xu et al. (2018) measure the diversity with corpus-level metrics. The former work evaluates the number of unique n-grams in the generated text, whereas the latter one measures the divergence of word distributions produced by different style-specific decoders. Our approach differs from these works by putting the focus on improving the structural diversity instead of choice of word.
NMT with Syntactic Constraint To our knowledge, there are no existing works control the sentence structure ahead of translation. However, our approach is related to NMT models under syntactic constraint. Stahlberg et al. (2016) syntactically guides the NMT decoding using the lattice produced by Hiero, a statistical machine translation system allowing hierarchical phrase structure. Eriguchi et al. (2017) parse a dependency tree and combine the parsing loss with the original loss, which improves the syntactic trace of translations in a soft way.
Similar to our approach, the following works also enhance the target sentence with extra structural tags. Nadejde et al. (2017) interleaves CCG supertags with normal output words in the target side. Instead of predicting words, Aharoni & Goldberg (2017) trains an NMT model to generate linearized constituent parse trees. Wu et al. (2017) proposed a model to generate words and parse actions simultaneously. The word prediction and action prediction are conditioned on each other. These works focus on improving the syntactic correctness, whereas our work focuses on structural diversity.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETTINGS

text:
We evaluate our models on two machine translation datasets: IWSLT14 German-to-English dataset (Cettolo et al., 2014) and ASPEC Japanese-to-English dataset (Nakazawa et al., 2016). These datasets contain 178K and 3M bilingual pairs, respectively. For the IWSLT14 De-En dataset, we apply the Moses toolkit (Koehn et al., 2007) to tokenize both sides of the corpus. For the ASPEC Ja-En dataset, we use the Moses toolkit to tokenize the English side and Kytea (Neubig et al., 2011) to tokenize the Japanese side.
After tokenization, we apply byte-pair encoding (Sennrich et al., 2016) to segment the texts into subwords, forcing the vocabulary size of each language to be 20K and 40K for the IWSLT dataset and ASPEC dataset, respectively.
For evaluation, we report tokenized BLEU with Moses script. We concatenate all five TED/TEDx development and test corpora to form a test set containing 6750 pairs in the IWSLT14 dataset. For the ASPEC dataset, as we truecase the English texts in the preprocessing phase, we detruecase the results before evaluating the scores. For planning models, we remove the structural codes (or tags) from the results before executing the evaluation script.

heading: 4.2 EVALUATION OF CODING MODEL

text:
Training Details: In the coding model, all hidden layers have 256 hidden units. The model is trained using Nesterov’s accelerated gradient (NAG) (Nesterov, 1983) for a maximum of 50 epochs with a learning rate of 0.25. The parameters with the lowest validation loss are selected for evaluation. During training, the temperature τ in the Gumbel-Softmax is fixed to 1.0. We also tried to anneal the Softmax temperature; however, this did not result in any performance improvement.
When training the coding model, we test different settings of code length N and the number of code types K. Following the code setting, the coding model will learn N Gumbel-Softmax vectors, each having a size of K. In this case, the information capacity of the codes will be N logK bits.
Evaluation Results: We evaluate the code quality from two aspects. First, we measure the reconstruction accuracy that indicates how accurately we can recover the original tag sequence from the codes. Second, to determine the difficulty of predicting the correct codes given the input sentence, we add an extra Softmax layer after the last hidden vector of the X encoder to predict the generated codes. Although the additional Softmax layer is trained simultaneously with the coding model, we disconnect the backpropagation path to ensure that the code prediction loss does not affect other components.
Table 1 shows the evaluation results for different code settings. We can see a clear trade-off between the reconstruction accuracy and code prediction accuracy, especially on IWSLT14 dataset. When the code has more capacity, it can recover structural tags more accurately; however, this results in lower predictability.
Generally, we want the codes to contain more accurate information about the sentence structure. However, the translation quality may be negatively impacted if it is too difficult for the NMT model to guess a good code. Therefore, we choose the code settings N = 1,K = 4 and N = 2,K = 4 for the following experiments as these provide a balanced trade-off.

heading: 4.3 EVALUATION OF NMT MODELS

text:
Model Architecture: To create a strong NMT baseline, we use two layers of bidirectional LSTM encoders with two layers of LSTM decoders in the NMT model. The hidden layers have 256 units for the IWSLT De-En task and 1000 units for the ASPEC Ja-En task. All parameters are initialized using the Xavier method (cite). We apply Key-Value Attention (Miller et al., 2016) to the first decoder layer, and we use residual connection (He et al., 2016) to combine the hidden states in two decoder layers. Dropout is applied everywhere outside of the recurrent function with a drop rate of 0.2 . Before the final Softmax layer, we insert another full-connected layer that has 600 hidden units.
Training Details: To train the NMT models, we also use the NAG optimizer with an initial learning rate of 0.25, and it is annealed by a factor of 10 if no improvement of loss value is observed in 20K iterations. One batch contains 128 sentence pairs, and the computation is distributed to eight GPUs. The best parameters are chosen on a validation set.
Note that all evaluated models share the same architecture, and they only differ in terms of the target side of training data. For the tag planning model and discrete planning model, the target sentences are prefixed by tags and codes, respectively.
Evaluation Results: Table 2 shows the resultant BLEU scores of different models, which indicates that our proposed planning approach does not degrade the translation performance in both translation tasks. In contrast, when we use the structural tags to directly control the sentence structure, a significant drop in BLEU scores is observed. Such a negative impact on translation quality is undesirable for structural planning.
In both De-En and Ja-En tasks, we can also observe a minor increase in performance over the strong baseline when using beam search (BS > 1). The improvement may result from being able to simultaneously explore drastically different candidates. A recent study (Li et al., 2016) also showed that beam search performance can be improved by increasing the diversity of candidates.
When we compare the resultant translation performance of different code settings, we found the translation scores are not worsen when using a longer code sequence (N = 2), comparing to planning with only one code (N = 1). This is benefited by avoiding encoding predictable structural information when learning the codes, thereby minimizing the influence of code prediction errors.

heading: 4.4 EVALUATION OF STRUCTURAL DIVERSITY

text:
In our approach, the codes are learned to determine the target structure together with the input. A single code is not guaranteed to be aligned to a specific style. Therefore, we can not evaluate the diversity with the divergence between word distributions as Xu et al. (2018) does. In order to qualitatively evaluate the diversity of generated translations, we propose to use a simple BLEU
discrepancy metric. Suppose Y is a list of candidate translations, we compute the discrepancy with
DP(Y ) = 1 |Y |(|Y | − 1) ∑ y∈Y ∑ y′∈Y,y′ 6=y 1−∆(y, y′), (11)
where ∆(y, y′) computes the BLEU score of two candidates. The equation computes the mean value of 1 - BLEU between all candidate pairs. We get a high discrepancy score when all candidates are very different from each other. As Eq. 11 evaluates for one data point, we further average the discrepancy scores over the test dataset.
In order to see the structural diversity, we further propose a POS-DP metric, in which ∆(y, y′) computes the BLEU scores using the POS tags of two candidates. Therefore, lexical diversity is completely omitted in the scores. The POS-DP metric is expected to reflect the structural diversity.
We report the discrepancy scores in Table 3. We first apply beam search on the baseline model with a beam size of 4 to obtain four candidate translations for each input, and then measure the baseline discrepancy scores. Then we sample candidate sentences from a discrete planning model trained with a code setting N = 1,K = 4. As there are only four possible codes, we sample four translations, each conditioned on a unique code. Then we measure the mean discrepancy on the sampled translations.
We can see that the proposed method achieves higher discrepancy scores, which means that the candidate translations are less similar to each other. A larger gap in diversity is observed when evaluating with the POS-DP metric. It indicates that the proposed method creates more diversity in sentence structures.

heading: 5 QUALITATIVE ANALYSIS

heading: 5.1 SAMPLING TRANSLATIONS WITH HIGH DIVERSITY

text:
Instead of letting the beam search to decide the best structural codes, we can also choose the codes manually. Specifically, we use beam search to obtain three code sequences with highest scores. Then for each code, we sample the translation result conditioned on it. Table 4 gives some samples of the candidate translations produced by the model when conditioning on different structural codes, compared to the candidates produced by beam search.
We can see that in both tasks, the candidate translations produced by beam search algorithm has only minor difference in grammatical structure. In contrast, the translation results sampled with discrete
planning have drastically different structures. We can observe the diversity more significantly in Japanese-English task because the word order of translations is less relevant to source sentences in this task. We can also notice that as a result of learning from structural tags, the discrete codes control the sentence structure rather than the choice of word.
The results in Table 4 show that the proposed method can be useful for sampling paraphrased translations with high diversity. Such property can also benefit other language generation tasks such as conversation generation.

heading: 5.2 TRANSLATING UNDER STRUCTURAL CONSTRAINT

text:
Sometimes, we want the NMT model to generate a translation with a specific sentence structure. However, after we encode the structural information into discrete codes, it is difficult to clearly tell what sentence structure a specific code sequence indicates. This problem can be overcome by encoding a known sentence into codes in the hope that the NMT model can generate a translation with a similar structure when conditioned on the codes.
Table 5 shows some results obtained when using a known sentence to constrain the structure of translation in the Japanese-English translation task. The translation results are sampled using the discrete codes extracted from the conditional sentences. Each translation shows a global sentence structure similar to the condition sentence, although they have different utterances and local structures. The results confirm that our coding model captures the variations among different structures.

heading: 6 CONCLUSION

text:
In this paper, we add a planning phase to neural machine translation, which plans the global sentence structure before decoding words. The structural planning is achieved by using some discrete codes to control the structure of output sentences. The discrete codes are learned to capture the structural variation of a translation. We prefix all target sentences in the corpus and use the enhanced training data to train an NMT model. During translation, the model first generates the codes, then output the words conditioned on the structural constraint imposed by the codes.
To learn the codes, we design an end-to-end neural network with a discretization bottleneck to encode the information of structural tags into some one-hot vectors. Experiments show that the proposed method can avoid degrading the translation performance. We also confirm the effect of the structural codes, by being able to sample translations with drastically different structures using different codes. We can also constrain the structure of a translation by using the codes extracted from a known sentence.
Our approach can be used in other language generation tasks, such as conversation modeling. Also, the discrete coding model can be extended to learn discrete representations of other factors of sentences. For example, one can learn the discrete codes to capture the logical order of a paragraph. In this case, the results of discourse parsing can be used instead of structural tags. By manipulating the codes in the planning phase, we may control the logical flow of the output sentences.

heading: A ANALYSIS OF CODE EFFICIENCY

text: In Fig. 3, we report the distribution of the structural codes learned for all target sentences in both tasks. In each plot, the Y-axis indicates the total number of sentences that correspond to a specific structural code. In general, the coding model is able to distribute available discrete codes to different sentences in the training data. However, we can still observe that four codes in IWSLT14 and one code in ASPEC task are not assigned to any sentence. Thus, there is room for further improving the coding model to exploit the full capacity of the codes, such as using an improved discretization method (Kaiser et al., 2018).


summaries: The paper describes an interesting tweak of the standard GAN model (inspired by IPM based GANs) where both the generator and the discriminator optimize relative realness (and fakeness) of the (real, fake) image pairs. The authors give some intuition for this tweak and ran experiments with CIFAR10 and CAT datasets. Different variants of the standard GAN and the new tweak were compared under the FID metric. The experimental setup and details are provided; and the code is made publicly available.

summaries: In this work, the authors considers a variation of GAN by consider simultaneously decrease the probability that real data is real for the generator. To include such a property, the authors propose a relativistic discriminator which estimate the probability that the given real data is more realistic than the fake data. Numerical results are performed to show that the proposed methods are effective, and the resulting GANs are relatively more stable and generate higher quality data samples than their non-relativistic counterparts.

abstractText: In standard generative adversarial network (SGAN), the discriminator D estimates the probability that the input data is real. The generator G is trained to increase the probability that fake data is real. We argue that it should also simultaneously decrease the probability that real data is real because 1) this would account for a priori knowledge that half of the data in the mini-batch is fake, 2) this would be observed with divergence minimization, and 3) SGAN would be more similar to integral probability metric (IPM) GANs. We show that this property can be induced by using a “relativistic discriminator” which estimate the probability that the given real data is more realistic than a randomly sampled fake data. We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average. We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs). We show that IPM-based GANs are a subset of RGANs which use the identity function. Empirically, we observe that 1) RGANs and RaGANs are significantly more stable and generate higher quality data samples than their non-relativistic counterparts, 2) Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and 3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization. The code is freely available on https://github.com/AlexiaJM/RelativisticGAN.

authors:
Martin Arjovsky
Léon Bottou

title: Towards principled methods for training generative adversarial networks

venue: arXiv preprint arXiv:1701.04862,

authors:
Martin Arjovsky
Soumith Chintala
Léon Bottou

title: Wasserstein generative adversarial networks

venue: In International Conference on Machine Learning,

title: Pros and cons of gan evaluation measures

venue: arXiv preprint arXiv:1802.03446,

authors:
William Fedus
Mihaela Rosca
Balaji Lakshminarayanan
Andrew M Dai
Shakir Mohamed
Ian Goodfellow

title: Many paths to equilibrium: Gans do not need to decrease adivergence at every step

venue: arXiv preprint arXiv:1710.08446,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: Advances in Neural Information Processing Systems

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron C Courville

title: Improved training of wasserstein gans

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Günter Klambauer
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a nash equilibrium

venue: arXiv preprint arXiv:1706.08500,

authors:
Yongjun Hong
Uiwon Hwang
Jaeyoon Yoo
Sungroh Yoon

title: How generative adversarial nets and its variants work: An overview of gan

venue: arXiv preprint arXiv:1711.05914,

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: arXiv preprint arXiv:1502.03167,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

title: Learning multiple layers of features from tiny images

authors:
Zinan Lin
Ashish Khetan
Giulia Fanti
Sewoong Oh

title: Pacgan: The power of two samples in generative adversarial networks

venue: arXiv preprint arXiv:1712.04086,

authors:
Mario Lucic
Karol Kurach
Marcin Michalski
Sylvain Gelly
Olivier Bousquet

title: Are gans created equal? a large-scale study

venue: arXiv preprint arXiv:1711.10337,

authors:
Xudong Mao
Qing Li
Haoran Xie
Raymond YK Lau
Zhen Wang
Stephen Paul Smolley

title: Least squares generative adversarial networks

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

venue: arXiv preprint arXiv:1802.05957,

title: Integral probability metrics and their generating classes of functions

venue: Advances in Applied Probability,

authors:
Sebastian Nowozin
Botond Cseke
Ryota Tomioka

title: f-gan: Training generative neural samplers using variational divergence minimization

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

venue: arXiv preprint arXiv:1511.06434,

authors:
Tim Salimans
Ian Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training gans

authors:
Weiwei Zhang
Jian Sun
Xiaoou Tang

title: Cat head detection-how to effectively exploit shape and texture features

venue: In European Conference on Computer Vision,

heading: 1 INTRODUCTION

text:
Generative adversarial networks (GANs) (Hong et al., 2017) form a broad class of generative models in which a game is played between two competing neural networks, the discriminator D and the generator G. D is trained to discriminate real from fake data, while G is trained to generate fake data that D will mistakenly recognize as real. In the original GAN by Goodfellow et al. (2014), which we refer to as Standard GAN (SGAN), D is a classifier, thus it is predicting the probability that the input data is real. When D is optimal, the loss function of SGAN is approximately equal to the Jensen–Shannon divergence (JSD) (Goodfellow et al., 2014).
SGAN has two variants for the generator loss functions: saturating and non-saturating. In practice, the former has been found to be very unstable, while the latter has been found to more stable (Goodfellow et al., 2014). Under certain conditions, Arjovsky & Bottou (2017) proved that, if real and fake data are perfectly classified, the saturating loss has zero gradient and the non-saturating loss has non-zero, but volatile gradient. In practice, this means that the discriminator in SGAN often cannot be trained to optimality or with a too high learning rate; otherwise, gradients may vanish and, if so, training will stop. This problem is generally more noticeable in high-dimensional setting (e.g., high resolution
images and discriminator architectures with high expressive power) given that there are enough degrees of freedom available to perfectly classify the training set.
To improve on SGAN, many GAN variants have been suggested using different loss functions and discriminators that are not classifiers (e.g., LSGAN (Mao et al., 2017), WGAN (Arjovsky et al., 2017)). Although these approaches have partially succeeded in improving stability and data quality, the large-scale study by Lucic et al. (2017) suggests that these approaches do not consistently improve on SGAN. Additionally, some of the most successful approaches, such as WGAN-GP (Gulrajani et al., 2017), are much more computationally demanding than SGAN.
Many of the recent successful GANs variants have been based on Integral probability metrics (IPMs) (Müller, 1997) (e.g., WGAN , WGAN-GP, Fisher GAN (Mroueh & Sercu, 2017), Sobolev GAN (Mroueh et al., 2017)). In IPM-based GANs, the discriminator is real-valued and constrained to a specific class of function which regularize the discriminator. See Mroueh et al. (2017) for a review of the different IPMs.
These IPM constraints have been shown to be beneficial even in non-IPM based GANs. Spectral normalization (Miyato et al., 2018) improves the stability of various GANs and it consists in making the discriminator Lipschitz-1, which is the constraint of WGAN. Similarly, the gradient penalty of WGAN-GP also provides improve the stability of SGAN (Fedus et al., 2017). Although this shows that certain IPM constraints improve the stability of GANs, it does not explain why IPM-based GANs generally provide increased stability over other metrics/divergences in GANs (e.g., JSD for SGAN, f -divergences for f -GANs (Nowozin et al., 2016)).
Note that although powerful, IPM-based GANs tend to more computationally demanding than other GANs. Certain IPM-based GANs use a gradient penalty (e.g. WGAN-GP, Sobolev GAN) which is very computationally costly and most IPM-based GANs need more than one discriminator update per generator update (WGAN-GP requires at least 5 (Gulrajani et al., 2017)). Assuming equal training time for D and G, every additional discriminator update increase training time by a significant 50%.
In this paper, we argue that non-IPM-based GANs are missing a key ingredient, a relativistic discriminator, which IPM-based GANs already possess. We show that a relativistic discriminator is necessary to make GANs analogous to divergence minimization and produce sensible predictions based on the a priori knowledge that half of the samples in the mini-batch are fake. We provide empirical evidence showing that GANs with a relativistic discriminator are more stable and produce data of higher quality.

heading: 2 BACKGROUND

heading: 2.1 GENERATIVE ADVERSARIAL NETWORKS

text:
GANs can be defined very generally in terms of the discriminator in the following way: LD = Exr∼P [ f̃1(D(xr)) ] + Ez∼Pz [ f̃2(D(G(z))) ] , (1)
and LG = Exr∼P [g̃1(D(xr))] + Ez∼Pz [g̃2(D(G(z)))] , (2)
where f̃1, f̃2, g̃1, g̃2 are scalar-to-scalar functions, P is the distribution of real data, Pz is generally a multivariate normal distribution centered at 0 with variance 1, D(x) is the discriminator evaluated at x, G(z) is the generator evaluated at z (Q is the distribution of fake data, thus of G(z)). Note that, through the paper, we refer to real data as xr and fake data as xf . Without loss of generality, we assume that both LD and LG are loss functions to be minimized.
Most GANs can be separated into two classes: non-saturating and saturating loss functions. GANs with the saturating loss are such that g̃1=−f̃1 and g̃2=−f̃2, while GANs with the non-saturating loss are such that g̃1=f̃2 and g̃2=f̃1. Saturating GANs are intuitive as they can be interpreted as alternating between maximizing and minimizing the same loss function. After training D to optimality, the loss function is generally an approximation of a divergence (e.g., Jensen–Shannon divergence (JSD) for SGAN (Goodfellow et al., 2014), f -divergences for F-GANs (Nowozin et al., 2016), and Wassertein distance for WGAN (Arjovsky et al., 2017)). Thus, training G to minimize LG can be roughly interpreted as minimizing the approximated divergence (although this is not technically true; see
Jolicoeur-Martineau (2018)). On the other hand, non-saturating GANs can be thought as optimizing the same loss function, but swapping real data with fake data (and vice-versa). In this article, unless otherwise specified, we assume a non-saturating loss for all GANs.
SGAN assumes a cross-entropy loss, i.e., f̃1(D(x)) = − log(D(x)) and f̃2(D(x)) = − log(1 − D(x)), whereD(x) = sigmoid(C(x)), and C(x) is the non-transformed discriminator output (which we call the critic as per Arjovsky et al. (2017)). In most GANs, C(x) can be interpreted as how realistic the input data is; a negative number means that the input data looks fake (e.g., in SGAN, D(x) = sigmoid(−5) = 0), while a positive number means that the input data looks real (e.g., in SGAN, D(x) = sigmoid(5) = 1).

heading: 2.2 INTEGRAL PROBABILITY METRICS

text:
IPMs are statistical divergences represented mathematically as:
IPMF (P||Q) = sup C∈F Ex∼P[C(x)]− Ex∼Q[C(x)],
where F is a class of real-valued functions. IPM-based GANs can be defined using equation 1 and 2 where f̃1(D(x)) = g̃2(D(x)) = −D(x) and f̃2(D(x)) = g̃1(D(x)) = D(x), where D(x) = C(x) (i.e., no transformation is applied). It can be observed that both discriminator and generator loss functions are unbounded and would diverge to −∞ if optimized directly. However, IPMs assume that the discriminator is of a certain class of function that does not grow too quickly which prevent the loss functions from diverging. Each IPM applies a different constraint to the discriminator (e.g., WGAN assumes a Lipschitz D, WGAN-GP assumes that D has a gradient norm equal to 1 around real and fake data).

heading: 3 MISSING PROPERTY OF SGAN

text: We argue that the key missing property of SGAN is that the probability of real data being real (D(xr)) should decrease as the probability of fake data being real (D(xf )) increase. We provide three arguments suggesting that SGAN should have this property.

heading: 3.1 PRIOR KNOWLEDGE ARGUMENT

text: Assuming a rational human was shown half real data and half fake data. If they perceived all samples shown as equally real (C(xf ) ≈ C(xr) for most xr and xf ), they would assume that each sample has probability .50 of being real. However, this is not the case for the discriminator in SGAN. If all samples looked real (C(xf ) ≈ C(xr) ≥ 3), D would assume incorrectly that they are indeed all real (D(x) ≈ 1 for all x). Of course, once trained with the labels, D would decrease D(xf ) and thus would obtain more reasonable estimates. However, if D(xr) decreased as D(xf ) increased, we would have had that D(x) ≈ .50 for all x before even retraining D. A rational human would not require retraining. IPM-based GANs implicitly account for the fact that some of the samples must be fake because they compare how realistic real data is compared to fake data. This is the behavior that we would want.

heading: 3.2 DIVERGENCE MINIMIZATION ARGUMENT

text:
In SGAN, when optimized, we have that the discriminator loss function is equal to the Jensen–Shannon divergence (JSD) (Goodfellow et al., 2014). The JSD is minimized (JSD(P||Q) = 0) when D(xr) = D(xf ) = 12 for all xr ∈ P and xf ∈ Q and maximized (JSD(P||Q) = log(2)) when D(xr) = 1, D(xf ) = 0 for all xr ∈ P and xf ∈ Q. Thus, if we were directly minimizing the divergence from maximum to minimum, we would expect D(xr) to smoothly decrease from 1 to .50 for most xr and D(xf ) to smoothly increase from 0 to .50 for most xf (Figure 1a). However, when minimizing the saturating loss in SGAN, we are only increasing D(xf ), we are not decreasing D(xr) (Figure 1b). Furthermore, we are bringing D(xf ) closer to 1 rather than .50.
This means that SGAN dynamics are very different from the minimization of the JSD. To bring SGAN closer to divergence minimization, training the generator should not only increase D(xf ) but
also decrease D(xr) (Figure 1c). Note that although specific to the JSD, similar dynamics are true for other divergences; when the divergence is maximal, D(xr) and D(xf ) are very far from one another, but they converge to the same value as the divergence approach zero. Thus, this argument applies to other divergences.

heading: 3.3 GRADIENT ARGUMENT

text:
We compare the gradients of standard GAN and IPM-based GANs for further insight. It can be shown that the gradients of the discriminator and generator in non-saturating SGAN are respectively:
∇wLGAND = −Exr∼P [(1−D(xr))∇wC(xr)] + Exf∼Qθ [D(xf )∇wC(xf )] , (3)
∇θLGANG = −Ez∼Pz [(1−D(G(z)))∇xC(G(z))JθG(z)] , (4) where J is the Jacobian.
It can be shown that the gradients of the discriminator and generator in IPM-based GANs are respectively:
∇wLIPMD = −Exr∼P[∇wC(xr)] + Exf∼Qθ [∇wC(xf )], (5)
∇θLIPMG = −Ez∼Pz [∇xC(G(z))JθG(z)], (6) where C(x) ∈ F (the class of functions assigned by the IPM). From these equations, it can be observed that SGAN leads to the same dynamics as IPM-based GANs when we have that:
1. D(xr) = 0, D(xf ) = 1 in the discriminator step of SGAN 2. D(xf ) = 0 in the generator step of SGAN. 3. C(x) ∈ F
Assuming that the discriminator and generator are trained to optimality in each step (which we sometimes do for D, but never for G) and that it is possible to perfectly distinguish real from the fake data (strong assumption, but generally true early in training); we would have that D(xr) = 1, D(xf ) = 0 in the generator step and that D(xr) = 1, D(xf ) = 1 in the discriminator step for most xr and xf (Figure 1b). Thus, the only missing assumption would be that D(xr) = 0 in the discriminator step.
Although the above scenario is not realistic (because we never train G to optimality), if all the assumptions were respected and the generator could indirectly influence D(xr), we would have that D(xr) = 0, D(xf ) = 1. Thus, SGAN would have the same gradients as IPM-based GANs. We conjecture that making SGAN more similar to IPM-based GANs could potentially improve its stability (our results will show that it does in fact improves stability).

heading: 4 METHOD

heading: 4.1 RELATIVISTIC STANDARD GAN

text:
In standard GAN, the discriminator can be defined, in term of the non-transformed layer C(x), as D(x) = sigmoid(C(x)). A simple way to make discriminator relativistic (i.e., having the output of D depends on both real and fake data) is to sample from real/fake data pairs x̃ = (xr, xf ) and define it as D(x̃) = sigmoid(C(xr)− C(xf )). We can interpret this modification in the following way: the discriminator estimates the probability that the given real data is more realistic than a randomly sampled fake data. Similarly, we can define Drev(x̃) = sigmoid(C(xf ) − C(xr)) as the probability that the given fake data is more realistic than a randomly sampled real data. An interesting property of this discriminator is that we do not need to include Drev in the loss function through log(1 − Drev(x̃)) because we have that 1−Drev(x̃) = 1−sigmoid(C(xf )−C(xr)) = sigmoid(C(xr)−C(xf )) = D(x̃); thus, log(D(x̃)) = log(1−Drev(x̃)). The discriminator and generator (non-saturating) loss functions of the Relativistic Standard GAN (RSGAN) can be written as:
LRSGAND = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xr)− C(xf )))] . (7)
LRSGANG = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xf )− C(xr)))] . (8)

heading: 4.2 RELATIVISTIC GANS

text:
More generally, we consider any discriminator defined as a(C(xr)−C(xf )), where a is the activation function, to be relativistic. This means that almost any GAN can have a relativistic discriminator. This forms a new class of models which we call Relativistic GANs (RGANs).
Most GANs can be parametrized very generally in terms of the critic:
LGAND = Exr∼P [f1(C(xr))] + Exf∼Q [f2(C(xf ))] (9)
and LGANG = Exr∼P [g1(C(xr))] + Exf∼Q [g2(C(xf ))] , (10)
where f1, f2, g1, g2 are scalar-to-scalar functions. If we use a relativistic discriminator, these GANs now have the following form:
LRGAND = E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [f2(C(xf )− C(xr))] (11)
and
LRGANG = E(xr,xf )∼(P,Q) [g1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [g2(C(xf )− C(xr))] . (12)
If one use the identity function (i.e., f1(y) = g2(y) = −y, f2(y) = g1(y) = y), this results in a degenerate case since there is no supremum/maximum. However, if one adds a constraint so that C(xr)− C(xf ) is bounded, then there is a supremum and one arrives at IPM-based GANs. Thus, although different, IPM-based GANs share a very similar loss function focused on the difference in critics.
Importantly, g1 is normally ignored in GANs because its gradient is zero since the generator does not influence it. However, in RGANs, g1 is influenced by fake data, thus by the generator.

heading: 4.3 RELATIVISTIC AVERAGE GANS

text:
The discriminator has a very different interpretation in SGAN compared to RSGAN. In SGAN, D(x) estimates the probability that x is real, while in RGANs, D(xr, xf ) estimates the probability that xr is more realistic than xf . As a middle ground, we developed an alternative to the Relativistic Discriminator, which retains approximately the same interpretation as the discriminator in SGAN while still being relativistic.
We propose the Relativistic average Discriminator (RaD) which compares the critic of the input data to the average critic of samples of the opposite type. The discriminator loss function for this approach can be formulated as:
LRaSGAND = −Exr∼P [ log ( D̄(xr) ) ) ] − Exf∼Q [ log ( 1− D̄(xf ) )] , (13)
where
D̄(x) = { sigmoid(C(x)− Exf∼QC(xf )) if x is real sigmoid(C(x)− Exr∼PC(xr)) if x is fake.
(14)
RaD has a more similar interpretation to the standard discriminator than the relativistic discriminator. With RaD, the discriminator estimates the probability that the given real data is more realistic than fake data, on average.
As before, we can generalize this approach to work with any GAN loss function using the following formulation: LRaGAND = Exr∼P [ f1 ( C(xr)− Exf∼QC(xf ) ) ) ] + Exf∼Q [f2 (C(xf )− Exr∼PC(xr))] . (15)
LRaGANG = Exr∼P [ g1 ( C(xr)− Exf∼QC(xf ) ) ) ]
+ Exf∼Q [g2 (C(xf )− Exr∼PC(xr))] . (16) We call this general approach Relativistic average GAN (RaGAN).

heading: 5 EXPERIMENTS

text:
Experiments were conducted on the CIFAR-10 dataset (Krizhevsky, 2009) and the CAT dataset (Zhang et al., 2008). Code was written in Pytorch (Paszke et al., 2017) and models were trained using the Adam optimizer (Kingma & Ba, 2014) for 100K generator iterations with seed 1 (which shows that we did not fish for the best seed, instead, we selected the seed a priori). We report the Fréchet Inception Distance (FID) (Heusel et al., 2017), a measure that is generally better correlated with data quality than the Inception Distance (Salimans et al., 2016) (Borji, 2018); lower FID means that the generated images are of better quality.
For the models architectures, we used the standard CNN described by Miyato et al. (2018) on CIFAR10 and a relatively standard DCGAN architecture (Radford et al., 2015) on CAT (see Appendix). We also provide the source code required to replicate all analyses presented in this paper (See our repository: [Anonymous until peer review is finished]).

heading: 5.1 CIFAR-10

text:
In these analyses, we compared standard GAN (SGAN), least-squares GAN (LSGAN), Wassertein GAN improved (WGAN-GP), Hinge-loss GAN (HingeGAN) (Miyato et al., 2018), Relativistic SGAN (RSGAN), Relativistic average SGAN (RaSGAN), Relativistic average LSGAN (RaLSGAN), and Relativistic average HingeGAN (RaHingeGAN) using the standard CNN architecture on stable setups (See Appendix for details on the loss functions used). Additionally, we tested RSGAN and RaSGAN with the same gradient-penalty as WGAN-GP (named RSGAN-GP and RaSGAN-GP respectively).
We used the following two known stable setups: (DCGAN setup) lr = .0002, nD = 1, β1 = .50 and β2 = .999 (Radford et al., 2015), and (WGAN-GP setup) lr = .0001, nD = 5, β1 = .50 and β2 = .9 (Gulrajani et al., 2017), where lr is the learning rate, nD is the number of discriminator updates per generator update, and β1, β2 are the ADAM momentum parameters. For optimal stability, we used batch norm (Ioffe & Szegedy, 2015) in G and spectral norm (Miyato et al., 2018) in D.
Results are presented in Table 1. We observe that RSGAN and RaSGAN generally performed better than SGAN. Similarly, RaHingeGAN performed better than HingeGAN. RaLSGAN performed on par with LSGAN, albeit sightly worse. WGAN-GP performed poorly in the DCGAN setup, but very well in the WGAN-GP setup. RasGAN-GP performed poorly; however, RSGAN-GP performed better than all other loss functions using only one discriminator update per generator update. Importantly, the resulting FID of 25.60 is on par with the lowest FID obtained for this architecture using spectral normalization, as reported by Miyato et al. (2018) (25.5). Overall, these results show that using a relativistic discriminator generally improve data generation quality and that RSGAN works very well in conjunction with gradient penalty to obtain state-of-the-art results.

heading: 5.2 CAT

text:
CAT is a dataset containing around 10k pictures of cats with annotations. We cropped the pictures to the faces of the cats using those annotations. After removing outliers (hidden faces, blurriness, etc.), the CAT dataset contained 9304 images ≥ 64x64, 6645 images ≥ 128x128, and 2011 images ≥ 256x256. The CAT dataset is particularly challenging due to its small sample size and high-resolution images; this makes it perfect for testing the stability of different GAN loss functions.
We trained different GAN loss functions on 64x64, 128x128, 256x256 images. For 256x256 images, we compared RaGANs to known stable approaches: SpectralSGAN (SGAN with spectral normalization in D) and WGAN-GP. Although some approaches were able to train on 256x256 images, they did so with significant mode collapse. To alleviate this problem, for 256x256 images, we packed the discriminator (Lin et al., 2017) (i.e., D took a concatenated pair of images instead of a single image). We looked at the minimum, maximum, mean and standard deviation (SD) of the FID at 20k, 30k, ..., 100k generator iterations; results are presented in Table 2.
Overall, we observe lower minimum FID, maximum FID, mean and standard deviation (sd) for RGANs and RaGANs than their non-relativistic counterparts (SGAN, LSGAN, RaLSGAN).
In 64x64 resolution, both SGAN and LSGAN generated images with low FID, but they did so in a very unstable matter. For example, SGAN went from a FID of 17.50 at 30k iterations, to 310.56 at 40k iterations, and back to 27.72 at 50k iterations. Similarly, LSGAN went from a FID of 20.27 at 20k iterations, to 224.97 at 30k iterations, and back to 51.98 at 40k iterations. On the other hand, RaGANs were much more stable (lower max and SD) while also resulting in lower minimum FID. Using gradient-penalty did not improve data quality; however, it reduced the SD lower than without gradient penalty, thus increasing stability further.
SGAN was unable to converge on 128x128 or bigger images and LSGAN was unable to converge on 256x256 images. Meanwhile, RaGANs were able to generate plausible images with low FID in all resolutions. Although SpectralSGAN and WGAN-GP were able to generate 256x256 images of cats, the samples they generated were of poor quality (high FID). Thus, in this very difficult setting, relativism provided a greater improvement in quality than gradient penalty or spectral normalization.

heading: 6 CONCLUSION AND FUTURE WORK

text:
In this paper, we proposed the relativistic discriminator as a way to fix and improve on standard GAN. We further generalized this approach to any GAN loss and introduced a generally more stable variant called RaD. Our results suggest that relativism significantly improve data quality and stability of
GANs at no computational cost. Furthermore, using a relativistic discriminator with other tools of the trade (spectral norm, gradient penalty, etc.) may lead to better state-of-the-art.
Future research is needed to fully understand the mathematical implications of adding relativism to GANs. Furthermore, our experiments were limited to certain loss functions using only one seed, due to computational constraints. More experiments are required to determine which relativistic GAN loss function is best over a wide-range of datasets and hyper-parameters. We greatly encourage researchers and machine learning enthusiasts with greater computing power to experiment further with our approach.

heading: APPENDICES

text:
A INTUITIVE AND MEMEFUL VISUAL REPRESENTATION OF RAGANS
B INTUITION BEHIND RAGANS
Although the relative discriminator provide the missing property that we want in GANs (i.e., G influencing D(xr)), its interpretation is different from the standard discriminator. Rather than measuring “the probability that the input data is real”, it is now measuring “the probability that the
input data is more realistic than a randomly sampled data of the opposing type (fake if the input is real or real if the input is fake)”. To make the relativistic discriminator act more globally, as in its original definition, our initial idea was the following: average the relativistic discriminator over random samples of data of the opposing type. This can be conceptualized in the following way:
P (xr is real) := Exf∼Q[P (xr is more real than xf )] = Exf∼Q[sigmoid(C(xr)− C(xf ))] = Exf∼Q[D(xr, xf )],
P (xf is real) := Exr∼P[P (xf is more real than xr)] = Exr∼P[sigmoid(C(xf )− C(xr))] = Exr∼P[D(xf , xr)],
where D(xr, xf ) = sigmoid(C(xr)− C(xf )). Then, the following loss function for D could be applied:
LD = −Exr∼P [ log ( Exf∼Q[D(xr, xf )] ) ) ] − Exf∼Q [log (1− Exr∼P[D(xf , xr)])] . (17)
The main problem with this idea is that it would require looking at all possible combinations of real and fake data in the mini-batch. This would transform the problem fromO(m) toO(m2) complexity, where m is the batch size. This is problematic; therefore, we do not use this approach.
Instead, we propose to use the Relativistic average Discriminator (RaD) which compares the critic of the input data to the average critic of samples of the opposite type (See section 4.3). This approach has O(m) complexity.

heading: C GRADIENTS

heading: C.1 SGAN

text:
∇wLGAND = −∇wExr∼P [logD(xr)]−∇wExf∼Qθ [log(1−D(xf ))] = −∇wExr∼P [ log ( eC(xr)
eC(xr) + 1
)] −∇wExf∼Qθ [ log ( 1− e C(xf )
eC(xf ) + 1 )] = −∇wExr∼P [ C(xr)− log ( eC(xr) + 1 )] −∇wExf∼Qθ [ log(1)− log ( eC(xf ) + 1
)] = −Exr∼P [∇wC(xr)] + Exr∼P [ eC(xr)
eC(xr) + 1 ∇wC(xr)
] + Exf∼Qθ [ eC(xf )
eC(xf ) + 1 ∇wC(xf ) ] = −Exr∼P [∇wC(xr)] + Exr∼P [D(xr)∇wC(xr)] + Exf∼Qθ [D(xf )∇wC(xf )] = −Exr∼P [(1−D(xr))∇wC(xr)] + Exf∼Qθ [D(xf )∇wC(xf )]
∇θLGANG = −∇θEz∼Pz [logD(G(z))] = −∇θEz∼Pz [ log ( eC(G(z))
eC(G(z)) + 1 )] = −∇θEz∼Pz [ C(G(z))− log ( eC(G(z)) + 1
)] = −Ez∼Pz [ ∇xC(G(z))JθG(z)− ( eC(G(z))
eC(G(z)) + 1
) ∇xC(G(z))JθG(z) ] = −Ez∼Pz [(1−D(G(z)))∇xC(G(z))JθG(z)]
C.2 IPM-BASED GANS
∇wLIPMD = −∇wExr∼P[C(xr)] +∇wExf∼Qθ [C(xf )] = −Exr∼P[∇wC(xr)] + Exf∼Qθ [∇wC(xf )]
∇θLIPMG = −∇θEz∼Pz [C(G(z))] = −Ez∼Pz [∇xC(G(z))JθG(z)]

heading: D SIMPLIFIED FORM OF RELATIVISTIC SATURATING AND NON-SATURATING GANS

text:
The formulation of RGANs can be simplified when we have the following two properties: (1) f2(−y) = f1(y) and (2) the generator assumes a non-saturating loss (g1(y) = f2(y) and g2(y) = f1(y)). These two properties are observed in standard GAN, LSGAN using symmetric labels (e.g., -1 and 1), IPM-based GANs, etc. With these two properties, RGANs with non-saturating loss can be formulated simply as:
LRGAN∗D = E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] (18)
and LRGAN∗G = E(xr,xf )∼(P,Q) [f1(C(xf )− C(xr))] . (19)
Assuming f2(−y) = f1(y), we have that
LRGAND = E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [f2(C(xf )− C(xr))] = E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] = 2E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] .
If g1(y) = −f1(y) and g2(y) = −f2(y) (saturating GAN), we have that
LRGAN−SG = E(xr,xf )∼(P,Q) [g1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [g2(C(xf )− C(xr))] = −E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))]− E(xr,xf )∼(P,Q) [f2(C(xf )− C(xr))] = −E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))]− E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] = −2E(xr,xf )∼(P,Q) [f1(C(xr)− C(xf ))] .
If g1(y) = f2(y) and g2(y) = f1(y) (non-saturating GAN), we have that
LRGAN−NSG = E(xr,xf )∼(P,Q) [g1(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [g2(C(xf )− C(xr))] = E(xr,xf )∼(P,Q) [f2(C(xr)− C(xf ))] + E(xr,xf )∼(P,Q) [f1(C(xf )− C(xr))] = E(xr,xf )∼(P,Q) [f1(C(xf )− C(xr))] + E(xr,xf )∼(P,Q) [f1(C(xf )− C(xr))] = 2E(xr,xf )∼(P,Q) [f1(C(xf )− C(xr))] .

heading: E TESTING THE GRADIENT ARGUMENT

text:
Previously, we argued that SGAN could be equivalent to IPM-GANs under very strict conditions and assumptions. We mentioned that although most assumptions are reasonable, the assumption that the generator is trained to optimality is unrealistic. In which case, SGAN would not be equivalent to IPM-based GANs since D(xr) would not reach 0.
As an experiment, we calculated the mini-batch average of D(xr) in the first 100 iterations of the training for the CAT dataset in 256x256. Note that SGAN becomes stuck at around 200 iterations and can never go beyond generating noise. Thus, a difference in the distribution of D(xr) could reveal something meaningful about why Relativistic GANs can converge while their non-relativistic counterparts cannot.
In those 100 iterations, we have that the distance between P and Q is maximal since G only generate noise. Thus, we can perfectly distinguish real from fake data, which is one of the assumptions. The remaining assumptions were that D and G would be trained to optimality. Although we did not train D more than once, after the discriminator step, we generally had that D(xr) ≈ 1. What we wanted to verify is whether D(xr) ≈ 0 after the generator step in Relativistic GANs, even though we did not train G enough to reach optimality.
Results are shown in Figure 2. We observe that with only one generator update per discriminator update (nG = 1), RSGAN and RaSGAN never reach an average D(xr) of 0 but the distribution is much less concentrated around 1 than with SGAN. With nG = 2, RSGAN and RaSGAN sometimes reach an averageD(xr) of 0 and they form an almost uniform distribution around [0, 1]. This suggests that with the missing property (i.e., using Relativistic GANs), SGAN can be made more similar to IPM-based GANs, but never equivalent. Thus, Relativistic Standard GANs can be seen as having a dynamic in-between SGAN and IPM-based GANs.

heading: F CIFAR-10 HARD/UNSTABLE EXPERIMENTS

text:
In these analyses, we compared SGAN, LSGAN, WGAN-GP, RSGAN, RaSGAN, RaLSGAN, and RaHingeGAN with the standard CNN architecture on unstable setups in CIFAR-10. Unless otherwise specified, we used lr = .0002, β1 = .5, β2 = .999, nD = 1, and batch norm (Ioffe & Szegedy, 2015) in G and D. We tested the following four unstable setups: (1) lr = .001, (2) β1 = .9, β2 = .9, (3) no batch norm in G or D, and (4) all activation functions replaced with Tanh in both G and D (except for the output activation function of D).
Results are presented in Table 4. We observe that RaLSGAN performed better than LSGAN in all setups. RaHingeGAN performed slightly worse than HingeGAN in most setups. RSGAN and RaSGAN performed better than SGAN in two out of four setups, although differences were small. WGAN-GP generally performed poorly which we suspect is due to the single discriminator update per generator update. Overall, this provide good support for the improved stability of using the relative discriminator with LSGAN, but not with HingeGAN and SGAN. Although results are worse for the relativistic discriminator in some settings, differences are minimal and probably reflect natural variations.
It is surprising to observe low FID for SGAN without batch normalization considering its well-known difficulty with this setting (Arjovsky et al., 2017). Given these results, we suspected that CIFAR-10 may be too easy to fully observe the stabilizing effects of using the relative discriminator. Therefore, in the manuscript, we focused on the more difficult CAT dataset with high resolution pictures.

heading: G LOSS FUNCTIONS USED IN EXPERIMENTS

heading: G.1 SGAN (NON-SATURATING)

text:
LSGAND = −Exr∼P [log (sigmoid(C(xr)))]− Exf∼Q [log (1− sigmoid(C(xf )))] (20)
LSGANG = −Exf∼Q [log (sigmoid(C(xf )))] (21)

heading: G.2 RSGAN

text:
LRSGAND = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xr)− C(xf )))] (22)
Table 4: Fréchet Inception Distance (FID) at exactly 100k generator iterations on the CIFAR-10 dataset using unstable setups with different GAN loss functions. Unless otherwise specified, we used lr = .0002, β = (.50, .999), nD = 1, and batch norm (BN) in D and G. All models were trained using the same a priori selected seed (seed=1).
Loss lr = .001 β = (.9, .9) No BN Tanh
SGAN 154.20 35.29 35.54 59.17 RSGAN 50.95 45.12 37.11 77.21 RaSGAN 55.55 43.46 41.96 54.42
LSGAN 52.27 225.94 38.54 147.87 RaLSGAN 33.33 48.92 34.66 53.07 HingeGAN 43.28 33.47 34.21 58.51
RaHingeGAN 51.05 42.78 43.75 50.69 WGAN-GP 61.97 104.95 85.27 59.94
LRSGANG = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xf )− C(xr)))] (23)

heading: G.3 RASGAN

text:
LRaSGAND = −Exr∼P [ log ( D̃(xr) )] − Exf∼Q [ log ( 1− D̃(xf ) )]
(24)
LRaSGANG = −Exf∼Q [ log ( D̃(xf ) )] − Exr∼P [ log ( 1− D̃(xr) )]
(25) D̃(xr) = sigmoid ( C(xr)− Exf∼QC(xf ) ) D̃(xf ) = sigmoid (C(xf )− Exr∼PC(xr))

heading: G.4 LSGAN

text:
LLSGAND = Exr∼P [ (C(xr)− 0)2 ] + Exf∼Q [ (C(xf )− 1)2 ] (26)
LLSGANG = Exf∼Q [ (C(xf )− 0)2 ] (27)

heading: G.5 RALSGAN

text:
LRaLSGAND = Exr∼P [ (C(xr)− Exf∼QC(xf )− 1)2 ] + Exf∼Q [ (C(xf )− Exr∼PC(xr) + 1)2 ] (28)
LRaLSGANG = Exf∼P [ (C(xf )− Exr∼PC(xr)− 1)2 ] + Exr∼P [ (C(xr)− Exf∼QC(xf ) + 1)2 ] (29)

heading: G.6 HINGEGAN

text:
LHingeGAND = Exr∼P [max(0, 1− C(xr))] + Exf∼Q [max(0, 1 + C(xf ))] (30)
LHingeGANG = −Exf∼Q [C(xf )] (31)

heading: G.7 RAHINGEGAN

text:
LHingeGAND = Exr∼P [ max(0, 1− D̃(xr)) ] + Exf∼Q [ max(0, 1 + D̃(xf )) ] (32)
LHingeGANG = Exf∼P [ max(0, 1− D̃(xf )) ] + Exr∼Q [ max(0, 1 + D̃(xr)) ] (33)
D̃(xr) = C(xr)− Exf∼QC(xf ) D̃(xf ) = C(xf )− Exr∼PC(xr)

heading: G.8 WGAN-GP

text:
LWGAN−GPD = −Exr∼P [C(xr)] + Exf∼Q [C(xf )] + λEx̂∼Px̂ [ (||∇x̂C(x̂) ||2 − 1)2 ] (34)
LWGAN−GPG = −Exf∼Q [C(xf )] (35)
Px̂ is the distribution of x̂ = xr + (1− )xf , where xr ∼ P, xf ∼ Q, ∼ U [0, 1].

heading: G.9 RSGAN-GP

text:
LRSGAND = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xr)− C(xf )))] + λEx̂∼Px̂ [ (||∇x̂C(x̂) ||2 − 1)2 ] (36)
LRSGANG = −E(xr,xf )∼(P,Q) [log(sigmoid(C(xf )− C(xr)))] (37)
Px̂ is the distribution of x̂ = xr + (1− )xf , where xr ∼ P, xf ∼ Q, ∼ U [0, 1].

heading: G.10 RASGAN-GP

text:
LRaSGAND = −Exr∼P [ log ( D̃(xr) )] −Exf∼Q [ log ( 1− D̃(xf ) )] +λEx̂∼Px̂ [ (||∇x̂C(x̂) ||2 − 1)2
] (38)
LRaSGANG = −Exf∼Q [ log ( D̃(xf ) )] − Exr∼P [ log ( 1− D̃(xr) )]
(39)
D̃(xr) = sigmoid ( C(xr)− Exf∼QC(xf ) ) D̃(xf ) = sigmoid (C(xf )− Exr∼PC(xr)) Px̂ is the distribution of x̂ = xr + (1− )xf , where xr ∼ P, xf ∼ Q, ∼ U [0, 1].

heading: H ALGORITHMS

text:
Algorithm 1 Training algorithm for non-saturating RGANs with symmetric loss functions Require: The number of D iterations nD (nD = 1 unless one seeks to train D to optimality), batch
size m, and functions f which determine the objective function of the discriminator (f is f1 from equation 10 assuming that f2(−y) = f1(y), which is true for many GANs). while θ has not converged do
for t = 1, . . . , nD do Sample {x(i)}mi=1 ∼ P Sample {z(i)}mi=1 ∼ Pz Update w using SGD by ascending with ∇w 1m ∑m i=1 [ f(Cw(x (i))− Cw(Gθ(z(i)))) ] end for Sample {x(i)}mi=1 ∼ P Sample {z(i)}mi=1 ∼ Pz Update θ using SGD by ascending with∇θ 1m ∑m i=1 [ f(Cw(Gθ(z (i)))− Cw(x(i))) ]
end while
Algorithm 2 Training algorithm for non-saturating RaGANs Require: The number of D iterations nD (nD = 1 unless one seek to train D to optimality), batch
size m, and functions f1 and f2 which determine the objective function of the discriminator (see equation 10). while θ has not converged do
for t = 1, . . . , nD do Sample {x(i)}mi=1 ∼ P Sample {z(i)}mi=1 ∼ Pz Let Cw(xr) = 1m ∑m i=1 Cw(x (i))
Let Cw(xf ) = 1m ∑m i=1 Cw(Gθ(z
(i))) Update w using SGD by ascending with ∇w 1m ∑m i=1 [ f1(Cw(x (i))− Cw(xf )) + f2(Cw(Gθ(z(i)))− Cw(xr)) ]
end for Sample {x(i)}mi=1 ∼ P Sample {z(i)}mi=1 ∼ Pz Let Cw(xr) = 1m ∑m i=1 Cw(x (i))
Let Cw(xf ) = 1m ∑m i=1 Cw(Gθ(z
(i))) Update θ using SGD by ascending with ∇θ 1m ∑m i=1 [ f1(Cw(Gθ(z (i)))− Cw(xr)) + f2(Cw(x(i))− Cw(xf )) ]
end while

heading: I ARCHITECTURES

heading: I.1 STANDARD CNN

text:
Generator
z ∈ R128 ∼ N(0, I) linear, 128 -> 512*4*4
Reshape, 512*4*4 -> 512 x 4 x 4
ConvTranspose2d 4x4, stride 2, pad 1, 512->256
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, 256->128
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, 128->64
BN and ReLU
ConvTranspose2d 3x3, stride 1, pad 1, 64->3
Tanh
Discriminator
x ∈ R3x32x32
Conv2d 3x3, stride 1, pad 1, 3->64
LeakyReLU 0.1
Conv2d 4x4, stride 2, pad 1, 64->64
LeakyReLU 0.1
Conv2d 3x3, stride 1, pad 1, 64->128
LeakyReLU 0.1
Conv2d 4x4, stride 2, pad 1, 128->128
LeakyReLU 0.1
Conv2d 3x3, stride 1, pad 1, 128->256
LeakyReLU 0.1
Conv2d 4x4, stride 2, pad 1, 256->256
LeakyReLU 0.1
Conv2d 3x3, stride 1, pad 1, 256->512
Reshape, 512 x 4 x 4 -> 512*4*4
linear, 512*4*4 -> 1
I.2 DCGAN 64X64
Generator
z ∈ R128 ∼ N(0, I) ConvTranspose2d 4x4, stride 1, pad 0, no bias, 128->512
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 512->256
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 256->128
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 128->64
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 64->3
Tanh
Discriminator
x ∈ R3x64x64
Conv2d 4x4, stride 2, pad 1, no bias, 3->64
LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 64->128
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 128->256
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 256->512
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 512->1
I.3 DCGAN 128X128
Generator
z ∈ R128 ∼ N(0, I) ConvTranspose2d 4x4, stride 1, pad 0, no bias, 128->1024
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 1024->512
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 512->256
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 256->128
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 128->64
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 64->3
Tanh
Discriminator
x ∈ R3x128x128
Conv2d 4x4, stride 2, pad 1, no bias, 3->64
LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 64->128
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 128->256
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 256->512
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 512->1024
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 1024->1
I.4 DCGAN 256X256
Generator
z ∈ R128 ∼ N(0, I) ConvTranspose2d 4x4, stride 1, pad 0, no bias, 128->1024
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 1024->512
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 512->256
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 256->128
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 128->64
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 64->32
BN and ReLU
ConvTranspose2d 4x4, stride 2, pad 1, no bias, 64->3
Tanh
Discriminator (PACGAN2 (Lin et al., 2017))
x1 ∈ R3x256x256, x2 ∈ R3x256x256
Concatenate [x1, x2] ∈ R6x256x256
Conv2d 4x4, stride 2, pad 1, no bias, 6->32
LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 32->64
LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 64->128
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 128->256
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 256->512
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 512->1024
BN and LeakyReLU 0.2
Conv2d 4x4, stride 2, pad 1, no bias, 1024->1

heading: J SAMPLES

text: This shows a selection of cats from certain models. Images shown are from the lowest FID registered at every 10k generator iterations. Given space constraint, with cats in high resolution, we show some of the nicer looking cats for each approach, there are evidently some worse looking cats (See https://github.com/AlexiaJM/RelativisticGAN/tree/ master/images/full_minibatch for all cats of the mini-batch).

title: THE RELATIVISTIC DISCRIMINATOR: A KEY ELEMENT MISSING FROM STANDARD GAN