The variational autoencoder (VAE) imposes a probabilistic distribution (typically Gaussian) on the latent space and penalizes the KullbackLeibler (KL) divergence between the posterior and prior. In NLP, VAEs are extremely difficult to train due to the problem of KL collapsing to zero. One has to implement various heuristics such as KL weight annealing and word dropout in a carefully engineered manner to successfully train a VAE for text. In this paper, we propose to use the Wasserstein autoencoder (WAE) for probabilistic sentence generation, where the encoder could be either stochastic or deterministic. We show theoretically and empirically that, in the original WAE, the stochastically encoded Gaussian distribution tends to become a Dirac-delta function, and we propose a variant of WAE that encourages the stochasticity of the encoder. Experimental results show that the latent space learned by WAE exhibits properties of continuity and smoothness as in VAEs, while simultaneously achieving much higher BLEU scores for sentence reconstruction. 1 1 Introduction Natural language sentence generation in the deep learning regime typically uses a recurrent neural network (RNN) to predict the most probable next word given previous words (Mikolov et al., 2010). Such RNN architecture can be further conditioned on some source information, for example, an input sentence, resulting in a sequence-to-sequence (Seq2Seq) model. Traditionally, sentence generation is accomplished in a deterministic fashion, i.e., the model uses a deterministic neural network to encode an 1 Our code is availabe at https://github.com/ HareeshBahuleyan/probabilistic_nlg A preliminary version of this paper was preprinted at https://arxiv.org/abs/1806.08462 input sentence to some hidden representations, from which it then decodes an output sentence using another deterministic neural network. Bowman et al. (2016) propose to use the variational autoencoder (VAE, Kingma and Welling, 2014) to map an input sentence to a probabilistic continuous latent space. VAE makes it possible to generate sentences from a distribution, which is desired in various applications. For example, in an open-domain dialog system, the information of an utterance and its response is not necessarily a one-to-one mapping, and multiple plausible responses could be suitable for a given input. Probabilistic sentence generation makes the dialog system more diversified and more meaningful (Serban et al., 2017; Bahuleyan et al., 2018). Besides, probabilistic modeling of the hidden representations serves as a way of posterior regularization (Zhang et al., 2016), facilitating interpolation (Bowman et al., 2016) and manipulation of the latent representation (Hu et al., 2017). However, training VAEs in NLP is more difficult than the image domain (Kingma and Welling, 2014). The VAE training involves a reconstruction loss and a KullbackLeibler (KL) divergence between the posterior and prior of the latent space. In NLP, the KL term tends to vanish to zero during training, leading to an ineffective latent space. Previous work has proposed various engineering tricks to alleviate this problem, including KL annealing and word dropout (Bowman et al., 2016). In this paper, we address the difficulty of training VAE sentence generators by using a Wasserstein autoencoder (WAE, Tolstikhin et al., 2018). WAE modifies VAE in that it requires the integration of the posterior to be close to its prior, where the closeness is measured with empirical samples drawn from the distributions. In this way, the encoder could be either stochastic or deterministic, but the model still retains probabilistic properties. Moreover, we show both theoretically and empirically that the stochastic Gaussian encoder in the original form tends to be a Dirac-delta function. We thus propose a WAE variant that encourages the encoder's stochasticity by penalizing an auxiliary KL term. Experiments show that the sentences generated by WAE exhibit properties of continuity and smoothness as in VAE, while achieving a much higher reconstruction performance. Our proposed variant further encourages the stochasticity of the encoder. More importantly, WAE is robust to hyperparameters and much easier to train, without the need for KL annealing or word dropout as in VAE. In a dialog system, we demonstrate that WAEs are capable of generating better quality and more diverse sentences than VAE. Base Model: Deterministic Autoencoder (DAE). DAE encodes an input sentence with a recurrent neural network (RNN) and then decodes the same sentence through another RNN. For the encoder, the hidden state of the last word is represented as the latent space of the input sentence x . The latent representation is denoted as z . We feed z to the decoder RNN, which predicts one word at a time using a softmax layer, given by p (x t | z , x <t ) .The training objective for DAE is the sequence-aggregated cross-entropy loss, given by J =  N (cid:88) n =1 | x ( n ) | (cid:88) t =1 log p (x ( n ) t | z ( n ) , x ( n ) <t ) (1) where superscript ( n ) indicates the n th data point among 1 ,    , N . In DAE, the latent space is encoded and then decoded in a deterministic way, i.e., there is no probabilistic modeling of the hidden space. The hidden representations of data may be located on an arbitrary manifold (Figure 1a), which is not suitable for probabilistic generation. Variational Autoencoder (VAE). VAE extends DAE by imposing a prior distribution p ( z ) on the latent variable z , which is typically set to the standard normal N ( 0 , I ) (Kingma and Welling, 2014). Given an input sentence x , we would like to model the posterior of z by another normal distribution, q ( z | x ) = N (  post , diag  2 post ) , where  post and  2 post are the outputs of the encoder. In the training of VAE, z is sampled from q ( z | x ) , and the training objective is to maximize a variational lower bound of the likelihood of data. This is equivalent to minimizing the (expected) reconstruction loss similar to (1), while being regularized by the KL divergence between q ( z | x ) and p ( z ) , given by J = N (cid:88) n =1 (cid:104)  E z ( n )  q | x ( n ) | (cid:88) t =1 log p (x ( n ) t | z ( n ) , x ( n ) <t ) +  VAE  KL( q ( z ( n ) | x ( n ) ) (cid:107) p ( z )) (cid:105) (2) where in the expectation z ( n ) is sampled from q ( z | x ( n ) ) and  VAE is a hyperparameter balancing the two terms. Since VAE penalizes the divergence of z 's posterior from its prior, it serves as a way of posterior regularization, making it possible to generate sentences from the continuous latent space. However, the two objectives in (2) are contradictory to each other, as argued by Tolstikhin et al. (2018). VAE pushes the posterior of z , given any input x ( n ) , to be close to its prior, i.e., every blue ellipse in Figure 1b should be close to the red one. This makes perfect reconstruction impossible. Further, VAE is difficult to train in NLP due to the problem of KL collapse, where the KL term tends to be zero, meaning that the encoder captures no information and the decoder learns an unconditioned language model. This phenomenon is observed in variational auto-regressive decoders using RNN. To alleviate this problem, existing tricks include KL annealing and word dropout (Bowman et al., 2016), but both require extensive engineering. alternative way of posterior regularization is to impose a constraint that the aggregated posterior of z should be the same as its prior (Tolstikhin et al., 2018), i.e., q ( z ) def = (cid:80) x q ( z | x ) p D ( x ) set = p ( z ) , where p D is the data distribution. This is also demonstrated in Figure 1c. By contrast, VAE requires that q ( z | x ) should be close to p ( z ) for every input sentence x . For computational purposes, Tolstikhin et al. (2018) relax the above constraint by penalizing the Wasserstein distance between q ( z ) and p ( z ) . In particular, it is computed by the Maximum Mean Discrepancy (MMD), defined as MMD = (cid:13)(cid:13)(cid:13)(cid:13)(cid:90) k ( z ,  ) d P ( z )  (cid:90) k ( z ,  ) d Q ( z ) (cid:13)(cid:13)(cid:13)(cid:13) H k where P ( z ) and Q ( z ) are cumulative density functions. H k refers to the reproducing kernel Hilbert space defined by the kernel k , which is often chosen as the inverse multiquadratic kernel k ( x, y ) = C C + (cid:107) x  y (cid:107) 22 for high-dimensional Gaus-sians. One advantage of the Wasserstein distance is that it can be estimated by empirical samples as (cid:92) MMD = 1 N ( N  1) (cid:88) n (cid:54) = m k ( z ( n ) , z ( m ) ) (3) + 1 N ( N  1) (cid:88) n (cid:54) = m k ( (cid:101) z ( n ) , (cid:101) z ( m ) )  1 N 2 (cid:88) n,m k ( z ( n ) , (cid:101) z ( m ) ) where (cid:101) z ( n ) is a sample from the prior p ( z ) , and z ( n ) is a sample from the aggregated posterior q ( z ) , which is obtained by sampling x ( n ) from the data distribution and then sampling z ( n ) from q ( z | x ( n ) ) . In summary, the training objective of WAE is JWAE =  N (cid:88) n =1 | x ( n ) | (cid:88) t =1 log p (x ( n ) t | z ( n ) , x ( n ) <t ) +  WAE  (cid:92) MMD (4) where  WAE balances the MMD penalty and the reconstruction loss. Alternatively, the dual form (adversarial loss) can also be used for WAE (Zhao et al., 2018). In our preliminary experiments, we found MMD similar to but slightly better than the adversarial loss. The difference between our work and Zhao et al. (2018)who extend the original WAE to sequence generationis that we address the KL annealing problem of VAE and further analyze the stochasticity of WAE from a theoretical perspective, as follows. WAE with Auxiliary Loss. In WAE, the aggregated posterior q ( z ) involves an integration of data distribution, which allows using a deterministic function to encode z as z = f encode ( x ) as suggested by Tolstikhin et al. (2018). This would largely alleviate the training difficulties as in VAE, because backpropagating gradient into the encoder no longer involves a stochastic layer. The stochasticity of the encoder, however, is still a desired property in some applications, for example, generating diverse responses in a dialog system. We show both theoretically and empirically that a dangling Gaussian stochastic encoder could possibly degrade to a deterministic one. Theorem 1. Suppose we have a Gaussian family N (  , diag  2 ) , where  and  are parameters. The covariance is diagonal, meaning that the variables are independent. If the gradient of  completely comes from sample gradient and  is small at the beginning of training, then the Gaussian converges to a Dirac delta function with stochastic gradient descent, i.e.,   0 . (See Appendix A for the proof.) To alleviate this problem, we propose a simple heuristic that encourages the stochasticity of the encoder. In particular, we penalize, for every data point, a KL term between the predicted posterior q ( z | x ) = N (  post , diag  2 post ) and a Gaussian with covariance I centered at the predicted mean, i.e., N (  post , I ) . This is shown in Figure 1d, where each posterior is encouraged to stretch with covariance I . Formally, the loss is J = J rec +  WAE  (cid:92) MMD +  KL (cid:88) n KL (cid:16) N (  ( n ) post , diag(  ( n ) post ) 2 ) (cid:13)(cid:13) N (  ( n ) post , I ) (cid:17) (5) While our approach appears heuristic, the next theorem shows its theoretical justification. Theorem 2. Objective (5) is a relaxed optimization of the WAE loss (4) with a constraint on  post . (See Appendix B for the proof.) We will show empirically that such auxiliary loss enables us to generate smoother and more diverse sentences in WAE. It, however, does not suffer from KL collapse as in VAEs. The auxiliary KL loss that we define for stochastic WAE is computed against a target distribution N (  ( n ) post , I ) for each data sample x ( n ) . Here, the predicted posterior mean itself is used in the target distribution. As a result, this KL term does not force the model to learn the same posterior for all data samples (as in VAE), and thus, the decoder does not degrade to an unconditioned language model. We evaluate WAE in sentence generation on the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) as well as dialog response generation. All models use single-layer RNN with long short term memory (LSTM) units for both the encoder and decoder. Appendix C details our experimental settings. VAE training. VAE is notoriously difficult to train in the RNN setting. While different researchers have their own practice of training VAE, we follow our previous experience (Bahuleyan et al., 2018) and adopt the following tricks to stabilize the training: (1)  VAE was annealed in a sigmoid manner. We monitored the value of   KL and stop annealing once it reached its peak value, known as peaking annealing . (2) For word dropout, we started with no dropout, and gradually increased the dropout rate by 0 . 05 every epoch until it reached a value of 0 . 5 . The effect of KL annealing is further analyzed in Appendix D. 3.1 SNLI Generation The SNLI sentences are written by crowd-sourcing human workers in an image captioning task. It is a massive corpus but with comparatively simple sentences (examples shown in Table 4). This task could be thought of as domain-specific sentence generation, analogous to hand written digit generation in computer vision. In Table 1, we compare all methods in two aspects. (1) We evaluate by BLEU (Papineni et al., 2002) how an autoencoder preserves input information in a reconstruction task. (2) We also evaluate the quality of probabilistic sentence generation from the latent space. Although there is no probabilistic modeling of the latent space in DAE, we nevertheless draw samples from N ( 0 , I ) , which could serve as a non-informative prior. Perplexity ( PPL ) evaluates how fluent the generated sentences are. This is given by a third-party n -gram language model trained on the Wikipedia dataset. The unigram-KL ( UniKL ) evaluates if the word distribution of the generated sentences is close to that of the training corpus. Other surface metrics (entropy of the word distribution and average sentence length) also measure the similarity of the latent space generated sentence set to that of the corpus. We see that DAE achieves the best BLEU score, which is not surprising because DAE directly optimizes the maximum likelihood of data as a surrogate of word prediction accuracy. Consequently, DAE performs poorly for probabilistic sentence generation as indicated by the other metrics. VAE and WAE have additional penalties that depart from the goal of reconstruction. However, we see that WAEs, when trained with appropriate hyperparameters (  WAE ,  KL ), achieve close performance to DAE, outperforming VAE by 40 BLEU points. This is because VAE encodes each input's posterior to be close to the prior, from which it is impossible to perfectly reconstruct the data. Comparing the deterministic and stochastic encoders in WAE, we observe the same trade-off between reconstruction and sampling. However, our proposed stochastic encoder, with  KL = 0 . 1 for WAE, consistently outperforms VAE in the contradictory metrics BLEU and PPL. The hyperparameters  WAE = 10 . 0 and  KL = 0 . 01 appear to have the best balance between reconstruction, sentence fluency, as well as similarity to the original corpus. Moreover, all our WAEs are trained without annealing or word dropout. It is significantly simpler than training a VAE, whose KL annealing typically involves a number of engineering tricks, such as the time step when KL is included, the slope of annealing, and the stopping criterion for annealing. We extend WAE to an encoder-decoder framework (denoted by WED) and evaluate it on the DailyDialog corpus (Li et al., 2017). 2 We follow Bahuleyan et al. (2018), using the encoder to capture an utterance and the decoder to generate a reply. Table 2 shows that WED with a deterministic encoder (WED-D) is better than the variational encoder-decoder (VED) in BLEU scores, but the generated sentences lack variety, which is measured by output entropy and the percentage of distinct unigrams and bigrams (Dist-1/Dist-2, Li et al., 2016), evaluated on the generated test set responses. We then applied our stochastic encoder for WED and see that, equipped with our KL-penalized stochastic encoder, WED-S outperforms DED, VED, and WED-D in all diversity measures. WED-S also outperforms VED in generation quality, consistent with the results in Table 1. 4 Conclusion In this paper, we address the difficulty of training VAE by using a Wasserstein autoencoder (WAE) for probabilistic sentence generation. WAE implementation can be carried out with either a deterministic encoder or a stochastic one. The deterministic version achieves high reconstruction performance, but lacks diversity for generation. The stochastic encoder in the original form may collapse to a Dirac delta function, shown by both a theorem and empirical results. We thus propose to encourage stochasticity by penalizing a heuristic 2 In our pilot experiment, we obtained a BLEU-4 score of 6 by training a pure Seq2Seq model with LSTM units for 200 epochs, whereas Li et al. (2017) report 0.009 BLEU-4 and Luo et al. (2018) report 2.84 BLEU-4. Due to our unreasonably high performance, we investigated this in depth and found that the training and test sets of the DailyDialog corpus have overlaps. For the results reported in our paper, we have removed duplicate data in the test set, which is also available on our website (Footnote 1). To the best of our knowledge, we are the first to figure out the problem, which, unfortunately, makes comparison with previous work impossible. KL loss for WAE, which turns out to be a relaxed optimization of the Wasserstein distance with a constraint on the posterior family. We evaluated our model on both SNLI sentence generation and dialog systems. We see that WAE achieves high reconstruction performance as DAE, while retaining the probabilistic property as VAE. Our KL-penalty further improves the stochasticity of WAE, as we achieve the highest performance in all diversity measures. We would like to acknowledge Yiping Song and Zhiliang Tian for their independent investigation on the DailyDialog corpus. We also thank Yanran Li, one of the authors who released DailyDialog, for discussion on this issue. This work was supported in part by the NSERC grant RGPIN-261439-2013 and an Amazon Research Award.
abstain abstain abstain objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result objective abstain objective abstain objective abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain result result other other other