Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of exposure bias. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation. Neural networks have revolutionized the field of text generation, in machine translation (Sutskever et al., 2014; Neubig, 2017; Luong et al., 2015; Chen et al., 2018), summarization (See et al., 2017), image captioning (You et al., 2016) and many other applications (Goldberg, 2017). Traditionally, text generation models are trained by going over a gold sequence of symbols (char-acters or words) from left-to-right, and maximizing the probability of the next symbol given the history, namely, a language modeling (LM) objective. A commonly discussed drawback of such LM-based text generation is exposure bias (Ran-zato et al., 2015): during training, the model predicts the next token conditioned on the ground truth history, while at test time prediction is based on predicted tokens, causing a train-test mismatch. Models trained in this manner often struggle to overcome previous prediction errors. Originally introduced for images, GANs leverage a discriminator, which is trained to discriminate between real images and generated images via an adversarial loss. In such a framework, the generator is not directly exposed to the ground truth data, but instead learns to imitate it using global feedback from the discriminator. This has led to several attempts to use GANs for text generation, with a generator using either a recurrent neural network (RNN) (Yu et al., 2017; Guo et al., 2017; Press et al., 2017; Rajeswar et al., 2017), or a Convolutional Neural Network (CNN) (Gulrajani et al., 2017; Rajeswar et al., 2017). However, evaluating GANs is more difficult than evaluating LMs. While in language modeling, evaluation is based on the log-probability of a model on held-out text, this cannot be straightforwardly extended to GAN-based text generation, because the generator outputs discrete tokens, rather than a probability distribution. Currently, there is no single evaluation metric for GAN-based text generation, and existing metrics that are based on n-gram overlap are known to lack robustness and have low correlation with semantic coherence (Semeniuta et al., 2018). In this paper, we propose a method for evaluating GANs with standard probability-based evaluation metrics. We show that the expected prediction of a GAN generator can be viewed as a LM, and suggest a simple Monte-Carlo method for approximating it. The approximated probability distribution can then be evaluated with standard LM metrics such as perplexity or Bits Per Character (BPC). To empirically establish our claim, we implement our evaluation on several RNN-based GANs: (Press et al., 2017; Yu et al., 2017; Guo et al., 2017). We find that all models have substantially lower BPC compared to state-of-the-art LMs. By directly comparing to LMs, we put in perspective the current performance of RNN-based GANs for text generation. Our results are also in line with recent concurrent work by Caccia et al. (2018), who reached a similar conclusion by comparing the performance of textual GANs to that of LMs using metrics suggested for GAN evaluation. Our code is available at: http: //github.com/GuyTevet/SeqGAN-eval and http://github.com/GuyTevet/ rnn-gan-eval . Following the success of GANs in image generation, several works applied the same idea to texts using convolutional neural networks (Gul-rajani et al., 2017; Rajeswar et al., 2017), and later using RNNs (Press et al., 2017; Yu et al., 2017). RNNs enable generating variable-length sequences, conditioning each token on the tokens generated in previous time steps. We leverage this characteristic in our approximation model (  4.1). A main challenge in applying GANs for text is that generating discrete symbols is a nondifferentiable operation. One solution is to perform a continuous relaxation of the GAN output, which leads to generators that emit a nearly discrete continuous distribution (Press et al., 2017). This keeps the model differentiable and enables end-to-end training through the discriminator. Alternatively, SeqGAN (Yu et al., 2017) and LeakGAN (Guo et al., 2017) used policy gradient methods to overcome the differentiablity requirement. We apply our approximation to both model types. LM Evaluation. Text generation from LMs is commonly evaluated using probabilistic metrics. Specifically, given a test sequence of symbols ( t 1 , . . . , t n ) , and a LM q , the average cross-entropy over the entire test set is computed: ACE =  1 n (cid:80) ni =1 log 2 q ( t i | t 1 ...t i  1 ) . For word-based models, the standard metric is perplexity: P P = 2 ACE , while for character-based models it is BP C = ACE directly. Intrinsic improvement in perplexity does not guarantee an improvement in an extrinsic downstream task that uses a language model. However, perplexity often correlates with extrinsic measures (Jurafsky and Martin, 2018), and is the de-facto metric for evaluating the quality of language models today. GAN-based Text Generation Evaluation. By definition, a text GAN outputs a discrete sequence of symbols rather than a probability distribution. As a result, LM metrics cannot be applied to evaluate the generated text. Consequently, other metrics have been proposed:  N-gram overlap: (Yu et al., 2017; Press et al., 2017): Inspired by BLEU (Papineni et al., 2002), this measures whether n-grams generated by the model appear in a held-out corpus. A major drawback is that this metric favors conservative models that always generate very common text (e.g., it is ). To mitigate this, self-BLEU has been proposed (Lu et al., 2018) as an additional metric, where overlap is measured between two independently sampled texts from the model. LM score: The probability of generated text according to a pre-trained LM. This has the same problem of favoring conservative models. Zhao et al. (2017) suggested an indirect score by training a LM on GAN-generated text, and evaluating it using perplexity. The drawback in this setting is the coupling of the performance of the GAN with that of the proxy LM. Heusel et al. (2017) used Frechet InferSent Distance (FID) to compute the distance between distributions of features extracted from real and generated samples. However, this approach relies on a problematic assumption that features are normally distributed. Rajeswar et al. (2017) used a context-free grammar (CFG) to generate a reference corpus, and evaluated the model by the likelihood the CFG assigns to generated samples. However, simple CFGs do not fully capture the complexity of natural language. To overcome the drawbacks of each individual method, Semeniuta et al. (2018) proposed a uni-fied measure based on multiple evaluation metrics (N-grams, BLEU variations, FID, LM score variations and human evaluation). Specifically, they argue that the different measures capture different desired properties of LMs, e.g., quality vs. diversity. Following Semeniuta et al. (2018), and in parallel to this work, Caccia et al. (2018) proposed a temperature sweep method that trades-off quality for diversity using a single parameter. Similar to our findings, they concluded that GANs perform worse than LMs on this metric. Overall, current evaluation methods cannot fully capture the performance of GAN-based text generation models. While reporting various scores as proposed by Semeniuta et al. (2018) is possible, it is preferable to have a single measure of progress when comparing different text generation models. We propose a method for approximating a distribution over tokens from a GAN, and then evaluate the model with standard LM metrics. We will describe our approach given an RNN-based LM, which is the most commonly-used architecture, but the approximation can be applied to other auto-regressive models (Vaswani et al., 2017). The inputs to an RNN at time step t , are the state vector h t and the current input token x t . The output token (one-hot) is denoted by o t . In RNN-based GANs, the previous output token is used at inference time as the input x t (Yu et al., 2017; Guo et al., 2017; Press et al., 2017; Rajeswar et al., 2017). In contrast, when evaluating with BPC or perplexity, the gold token x t is given as input. Hence, LM-based evaluation neutralizes the problem of exposure bias addressed by GANs. Nevertheless, this allows us to compare the quality of text produced by GANs and LMs on an equal footing. Figure 1 illustrates the difference between inference time and during LM approximation. We can therefore define the generator function at time step t as a function of the initial state h 0 and the past generated tokens ( x 0 . . . x t ) , which we denote as o t = G t ( h 0 , x 0 ...x t ) ( x 0 is a start token). Given a past sequence ( x 0 . . . x t ) , G t is a stochastic function: the stochasticity of G t can Algorithm 1 LM Evaluation of RNN-based GANs Input: G t (  ) : the generator function at time step t ( x 0 , ..., x t ) : previous gold tokens x t +1 : the gold next token (as ground truth) f (  ,  ) : a LM evaluation metric N : number of samples 1: for n  1 to N do 2: g t,n  sample from G t ( x 0 ...x t ) 3:  G t,N = 1 N  Nn =1 g t,n 4: return f (  G t,N , x t +1 ) be gained either by using a noise vector as the initial state h 0 (Press et al., 2017), or by sampling from the GAN's internal distribution over possible output tokens (Yu et al., 2017; Guo et al., 2017). Since h 0 is constant or a noise vector that makes G t stochastic, we can omit it to get G t ( x 0 . . . x t ) . In such a setup, the expected value E [ G t ( x 0 . . . x t )] is a distribution q over the next vocabulary token a t : q ( a t | a 0 . . . a t  1 ) = { E [ G t ( x 0 . . . x t )] } a t To empirically approximate q , we can sample from it N i.i.d samples, and compute an approximation  G t,N = 1 N  Nn =1 g t,n , where g t,n is one sample from G t ( x 0 ...x t ) . Then, according to the strong law of large numbers: E [ G t ( x 0 . . . x t )] = lim N   G t,N (1) Given this approximate LM distribution, we can evaluate a GAN using perplexity or BPC. We summarize the evaluation procedure in Algorithm 1. 1 4.2 Approximation Bound We provide a theoretical bound for choosing a number of samples N that results in a good approximation of  G t,N to E [ G t ] . Perplexity and BPC rely on the log-probability of the ground truth token. Since the ground truth token is unknown, we conservatively define the bad event B in which there exists v  V such that |{ E [ G t ] } v  {  G t,N } v | >  , where V is the vocabulary. We can then bound the probability of B by some (cid:15) . We define the following notations: 1. The probability of a token a t to be v is p v  = q ( a t = v | a 0 . . . a t  1 ) = { E [ G t ( x 0 . . . x t )] } v . 2.  v,n  = { g t,n } v is a random variable representing the binary value of the v 'th index of 1 Our evaluation algorithm is linear in the length of the test set and in the number of samples N . g t,n which is a single sample of G t . Note that the average of  v,n over N samples is X v  = 1 N (cid:80) Nn =1  v,n = (cid:110) 1 N (cid:80) Nn =1 g t,n (cid:111) v = {  G t,N } v . Using the above notation, we can re-define the probability of the bad event B with respect to the individual coordinates in the vectors: Pr ( B ) = Pr (cid:16) (cid:107) E [ G t ]   G t,N (cid:107)  >  (cid:17) = Pr (cid:32) (cid:91) v  V | p v  X v | >  (cid:33) ! < (cid:15) (2) We note that  v,n  Bernoulli ( p v ) , and given that {  v,n } Nn =1 are i.i.d., we can apply the Chernoff-Hoeffding theorem (Chernoff et al., 1952; Hoeffding, 1963). According to the theorem, for every v  V , P r ( | X v  p v | >  ) < 2 e  2 N 2 . Taking the union bound over V implies: Pr ( B ) = Pr (cid:0)(cid:83) v  V | X v  p v | >  (cid:1) < 2 | V | e  2 N 2 < (cid:15) (3) Hence, we get a lower bound on N : N > 1 2  2 ln (cid:18) 2 | V | (cid:15) (cid:19) (4) As a numerical example, choosing  = 10  3 and (cid:15) = 10  2 , for a character-based LM over the text8 dataset, with | V | = 27 , we get the bound: N > 4 . 3  10 6 . With the same  and (cid:15) , a typical word-based LM with vocabulary size | V | = 50 , 000 would require N > 8 . 1  10 6 . In practice, probability vectors of LMs tend to be sparse (Kim et al., 2016). Thus, we argue that we can use a much smaller N for a good approximation  G t,N . Since the sparsity of LMs is difficult to bound, as it differs between models, we suggest an empirical method for choosing N . The approximation  G t,N is a converging sequence, particularly over (cid:107)  (cid:107)  (see Equation 1). Hence, we can empirically choose an N which satisfies (cid:107)  G t,N     G t,N (cid:107)  <  (cid:48) ,   N . In Section 5 we empirically measure (cid:107)  G t,N     G t,N (cid:107)  as a function of N to choose N . We choose a global N for a model, rather than for every t , by averaging over a subset of the evaluation set. We focus on character-based GANs as a test-case for our method. We evaluate two RNN-based GANs with different characteristics. As opposed to the original GAN model (Goodfellow et al., 2014), in which the generator is initialized with random noise, the GANs we evaluated both leverage gold standard text to initialize the generator, as detailed below. Recurrent GAN (Press et al., 2017) is a continuous RNN-based generator which minimizes the improved WGAN loss (Gulrajani et al., 2017). To guide the generator, during training it is initialized with the first i  1 characters from the ground truth, starting the prediction in the i th character. Stochasticity is obtained by feeding the generator with a noise vector z as a hidden state. At each time step, the input to the RNN generator is the output distribution of the previous step. SeqGAN (Yu et al., 2017) is a discrete RNN-based generator. To guide the generator, it is pre-trained as a LM on ground truth text. Stochasticity is obtained by sampling tokens from an internal distribution function over the vocabulary. To overcome differentiation problem, it is trained using a policy gradient objective (Sutton et al., 2000). We also evaluated LeakGAN (Guo et al., 2017), another discrete RNN-based generator, but since it is similar to SeqGAN and performed worse, we omit it for brevity. To compare to prior work in LM, we follow the common setup and train on the text8 dataset. 2 The dataset is derived from Wikipedia, and includes 26 English characters plus spaces. We use the standard 90/5/5 split to train/validation/test. Finally, we measure performance with BPC. We tuned hyper-parameters on the validation set, including sequence length to generate at test time (7 for Press et al. (2017), 1000 for Yu et al. (2017)). We chose the number of samples N empirically for each model, as described in Section 4.2. We set  to 10, and the boundary to  (cid:48) = 10  3 as a good trade-off between accuracy and run-time. Figure 2 plots the approximate error (cid:107)  G t,N     G t,N (cid:107)  as a function of N . For both models, N > 1600 satisfies this condition (red line in Figure 2). To be safe, we used N = 2000 . Table 1 shows model performance on the test 2 http://mattmahoney.net/dc/textdata Approach Model BPC Approx. Because SeqGAN models output a distribution over tokens at every time step, we can measure the true BPC and assess the quality of our approximation. Indeed, we observe that approximate BPC is only slightly higher than the true BPC. GAN-based models perform worse than state-of-the-art LMs by a large margin. Moreover, in SeqGAN, the pre-trained LM performs better than the fully trained model with approximate BPC scores of 1.95 and 2.06, respectively, and the BPC deteriorates as adversarial training continues. Finally, we note that generating sequences larger than 7 characters hurts the BPC of Press et al. (2017). It is difficult to assess the quality of generation with such short sequences. In Table 2 we present a few randomly generated samples from each model. We indeed observe that adversarial training slightly reduces the quality of generated text for SeqGAN, and find that the quality of 100-character long sequences generated from Press et al. (2017) is low. We propose an evaluation procedure for text GANs that is based on approximating the GAN output distribution and using standard LM metrics. We provide a bound for the number of samples required for the approximation and empirically show in practice as few as 2000 samples per time-step suffice. We evaluate character-based GAN models using our procedure, and show their performance is substantially lower than state-of-the-art LM. We hope our simple evaluation method leads to progress in GAN-based text generation by shedding light on the quality of such models. We would like to thank Shimi Salant for his comments and suggestions. This research was partially supported by The Israel Science Foundation grant 942/16, the Blavatnik Computer Science Research Fund, and The Yandex Initiative for Machine Learning.
abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain method result method result other other other abstain other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain method other method method method method method abstain method method method abstain method abstain abstain method method abstain method abstain abstain abstain abstain method method abstain method method method method method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method method abstain abstain abstain method abstain other abstain result abstain abstain abstain abstain abstain result objective result result abstain other other