Variational autoencoders (VAEs) are widely used for latent variable modeling of text. We focus on variations that learn expressive prior distributions over the latent variable. We find that existing training strategies are not effective for learning rich priors, so we add the importance-sampled log marginal likelihood as a second term to the standard VAE objective to help when learning the prior. Doing so improves results for all priors evaluated, including a novel choice for sentence VAEs based on normalizing flows (NF). Priors parameterized with NF are no longer constrained to a specific distribution family, allowing a more flexible way to encode the data distribution. Our model, which we call FlowPrior, shows a substantial improvement in language modeling tasks compared to strong baselines. We demonstrate that FlowPrior learns an expressive prior with analysis and several forms of evaluation involving generation. Variational autoencoders (VAEs; Kingma and Welling, 2014) have been widely applied to many natural language processing tasks (Bowman et al., 2016; Zhang et al., 2016; Shen et al., 2017; Kim et al., 2018; Fang et al., 2019; Chen et al., 2019). VAEs provide statistical transparency in describing observations in a latent space and flexibility when used in applications that require directly manipulating the learned representation (Hu et al., 2017). Recent work (Li et al., 2020) has combined VAEs with BERT/GPT in representation learning and guided generation. However, the representation capacity of VAEs is still limited for modeling sentences due to two main reasons. One is known as the posterior collapse problem, in which the posterior collapses to the prior and the generator learns to ignore the latent variable (Bowman et al., 2016). Many methods have been developed to address it: annealing (Fu et al., 2019), weakening the capacity of the generator (Semeniuta et al., 2017; Yang et al., 2017), manipulating training objectives (Burda et al., 2016; Higgins et al., 2017; Zhao et al., 2017), including the use of free bits (FB) (Kingma et al., 2016; Li et al., 2019), and changing training (He et al., 2019). The other reason is the restrictive assumption of the parametric forms for the prior and approximate posterior. While these forms are computationally efficient, they limit the expressivity of the model. The main existing solutions (Kingma et al., 2016; Tomczak and Welling, 2018; Razavi et al., 2019) focus on enriching the variational posterior, while other work focuses on learning an expressive prior (Tomczak and Welling, 2018; Serban et al., 2017; Chen et al., 2017). In this paper, we follow the latter line of research and draw upon methods in building and learning expressive priors. We first show empirically that the original VAE objective, the evidence lower bound (ELBO), is not effective when learning priors. The issue is not solely due to posterior collapse since it is not resolved by using modifications based on free bits. To address this issue, we propose using a combined objective, adding to the ELBO a second objective (denoted MIS ) which is a different lower bound on the log marginal likelihood obtained using importance sampling (Burda et al., 2016). Using the combination of the ELBO and MIS , we compare multiple choices for the prior, including a mixture of Gaussians, a prior based on a variational mixture of posteriors (VampPrior; Tomczak and Welling, 2018), and a prior based on normalizing flows (NF), specifically real NVP transformations (Dinh et al., 2016). Using a real NVP prior entails creating an invertible mapping from a simple base distribution to the prior distribution of the latent variable in a VAE. This choice allows a flexible prior distribution that is not constrained to a specific parametric family. The hope is that it would be better at modeling the data distribution. We perform an empirical evaluation of priors and objective functions for training VAE sentence models on four standard datasets. We find the best performance overall when using the flow-based prior and the combined objective in the training objective. We refer to this setting as FlowPrior . The generation of prior samples with FlowPrior comports to the training distribution while maintaining a higher diversity than competing models in our quantitative and qualitative evaluation. To summarize, this paper contributes: (1) a strategy for improved training of sentence VAEs based on combining multiple lower bounds on the log marginal likelihood; (2) the first results applying real NVP to model the prior in sentence VAEs; and (3) comprehensive evaluation and analysis with three expressive priors and training objective variations. Variational autoencoders (VAEs; Kingma and Welling, 2014) are a popular framework for learning latent variable models with continuous latent variables. Let x be the observed variable and z the latent variable. The model factorizes the joint distribution over x and z into a prior p  ( z ) and a generator p  ( x | z ) . Maximizing the log marginal likelihood log p ( x ) is intractable in general, so VAEs introduce an approximate posterior q  ( z | x ) parameterized using a neural network (i.e., an in-ference network), and replace the log marginal likelihood with the evidence lower bound (ELBO): log p ( x )  E q  ( z | x ) [log p  ( x | z )]  KL( q  ( z | x ) || p  ( z )) (1) Maximizing the right-hand side of the equation above can be viewed as a regularized autoencoder in which the first term is the negative reconstruction error and the second is the negative KL divergence between the approximate posterior q  ( z | x ) and the latent variable prior p  ( z ) . It is common in practice to fix the prior p  ( z ) to be a standard Gaussian distribution and only learn  and  (Bowman et al., 2016; Yang et al., 2017; Shen et al., 2017). While constraining the prior to be a fixed standard Gaussian is common, it is not necessary for tractability. Researchers have found benefit from using richer priors and posteriors (Rezende and Mohamed, 2015; Kingma et al., 2016; Chen et al., 2017; Ziegler and Rush, 2019; Ma et al., 2019). In this paper, we consider investigating alternative priors while still using the standard Gaussian form for the approximate posterior. We now describe the three kinds of priors we will compare in our experiments. The first two are based on Gaussian mixtures (Sec. 3.1) and the third is based on normalizing flows (Sec. 3.2). We take these three prior families into consideration because they represent the three main categories of work in learning priors: simple Gaussian mixtures (usually as baselines), defining the prior as a function of the approximate posterior (Tomczak and Welling, 2018; Chen et al., 2018), and flow-based priors (Chen et al., 2017; Ziegler and Rush, 2019; Ma et al., 2019; Lee et al., 2020). Note that we do not make any changes to the approximate posterior distribution. That is, the approximate posterior follows a Gaussian distribution with a diagonal covariance matrix as in standard VAEs. Our first choice is a uniform mixture of K Gaussians (MoG): where f ( z ; , ) is the density function of a d dimensional Gaussian with mean  and covariance matrix  . The  k and  k are learnable parameter vectors with dimensionality d (which is 32 in our experiments). This prior was used as a baseline by Tomczak and Welling (2018). We refer to a VAE that uses this prior as MoG-VAE . Tomczak and Welling (2018) extend MoG-VAE to a Variational Mixture of Posteriors prior (VampPrior). This approach parameterizes the prior using a mixture of Gaussians with components given by a variational posterior conditioned on learnable pseudo-inputs: p  ( z ) = 1 KK (cid:88) k =1 q  ( z | u k ) (3) where K is the number of pseudo-inputs, each of which is denoted u k . Pelsmaeker and Aziz (2020) applied this idea to text modeling and we follow their strategy for defining pseudo-inputs. That is, each u k consists of a sequence of embeddings that have the same dimensionality as word embeddings. For each component k , the lengths of pseudo-inputs can vary; they are sampled based on the statistics of the lengths in the training set. We refer to a VAE with this prior as Vamp-VAE . Our third choice for a prior distribution is to leverage normalizing flows (NF). A normalizing flow is a sequence of invertible, deterministic transformations. By repeatedly applying the rule for change of variables (see the Appendices for details), the base density is transformed into a more complex one. Networks parameterized using NF can be trained through exact maximum log-likelihood computation. Exact sampling is performed by drawing a sample from the base distribution and performing the chain of transformations. This allows a flexible prior and is expected to have more expressive latent components compared to those based on Gaussian mixtures. Computing the Jacobian of functions with high dimension and the determinants of large matrices (i.e., the two main computations in NF) are very expensive. Our flow-based prior uses real-valued non-volume preserving (real NVP) transformations (Dinh et al., 2016) which are efficient in both training and sampling. The transformations are based on scale and translation operations. 1 It is worth noting that these two operations are not used in computing the Jacobian determinant and inverse. So one can design arbitrarily complex operations that allow a flexible transformation without incurring large computational cost. More specifically, we apply real NVP as a prior by creating an invertible mapping between a base distribution p 0 ( z 0 ) (in our case, z 0  N (0 , I) ) and the prior distribution p  ( z L ) in the VAE: z L = f L  f L  1  ...  f 1 ( z 0 ) (4) where z L is the sentence latent variable and f 1 , f 2 , ..., f L are all bijective functions. Using the change-of-variables theorem, given a latent variable z L , we can compute the exact density under the prior with the image z 0 acquired by inverting the transformation: z 0 = f  1 1  ...  f  1 L  1  f  1 L ( z L ) (5) log p  ( z L ) = log p 0 ( z 0 )  L (cid:88) l =1 log | det( f l ( z l  1 ) z l  1 ) | (6) 1 More details about normalizing flows and real NVP are in the Appendices. We refer to a VAE with a real NVP prior as real NVP-VAE. We find our best setting to consist of a real NVP prior and the combined objective in Section 4.1 and we refer to this setting as FlowPrior . ELBO. Our preliminary experiments found that, when training with the standard ELBO, using more sophisticated priors does not improve perplexity compared to standard Gaussian priors (Table 3). Though these priors could potentially be highly multimodal, the learned prior parameters yield approximately unimodal forms (Figure 1, left). Several approaches have been proposed to mitigate or avoid collapse in the approximate posterior. One method that we include in our experiments is a variation of KL divergence known as free bits (FB) KL (Li et al., 2019; Kingma et al., 2016). Posterior collapse is mitigated, but the VAE models still do not benefit much from expressive priors (Tables 1-2). Pelsmaeker and Aziz (2020) made similar observations with an improved FB objective. We speculate that these undesirable results are due to the lack of learning signal for the prior parameters. Marginal Likelihood via Importance Sampling. In the ELBO, the prior distribution only appears in the KL term. As a consequence, the prior parameters receive a limited amount of learning signal. The posterior network, by contrast, receives gradient updates from both the reconstruction and KL terms. When minimizing the KL term the potentially expressive prior density can collapse to a unimodal form, as this may facilitate minimizing the KL divergence between the approximate posterior and prior. We consider optimizing another objective, a different lower bound on the log marginal likelihood obtained using importance sampling (Burda et al., 2016): log 1 NN (cid:88) i =1 p  ( x | z ( i ) ) p  ( z ( i ) ) q  ( z ( i ) | x ) , s . where x is an input in the training data and N is the number of samples in use. This objective was proposed as the training objective in the importance-weighted autoencoder (IWAE; Burda et al., 2016), and was shown to be a tighter lower bound on the log marginal likelihood than the ELBO. In this paper, we denote this objective by MIS . In addition to providing a tighter lower bound, MIS also increases the flexibility of the approximate posterior, as shown by Cremer et al. (2017). By increasing N , the approximate posterior has an implicit complex distribution that approaches the true posterior, which may also be beneficial in learning an expressive prior. Combination of the Two. However, MIS is not necessarily optimal by itself for training VAEs. Rainforth et al. (2018) prove that using MIS with a large value of N is detrimental in learning the posterior, which is also shown in our empirical evaluation in Table 3. If we only have MIS , the approximate posterior q only appears in the denominator so learning seeks to make samples from the posterior q less likely under q , which could cause q to become a poor proposal distribution. The ELBO, with its reconstruction loss, appears helpful in learning a better posterior. Therefore, we optimize the sum of the ELBO and MIS , which was proposed by Rainforth et al. (2018). Our combined training objective then contains three terms: MIS , reconstruction, and sample-based KL. We draw N samples from q  ( z | x ) , and compute the three terms using the same samples: L ( , ,  ; x ) = log 1 NN (cid:88) i =1 p  ( x | z ( i ) ) p  ( z ( i ) ) q  ( z ( i ) | x ) + 1 NN (cid:88) i =1 log p  ( x | z ( i ) )  KL , ( x, { z ( i ) } Ni =1 ) s . t . z ( i )  q  ( z | x ) (7) When training with the ELBO alone, one typically uses a single sample from q  ( z | x ) . However, since we draw multiple samples anyway in order to compute MIS , we use those same samples for the reconstruction term, which can lead to more robust gradients of that term than the standard approach of using one sample. The reason we use sample-based estimates for the KL divergence is because our choices for the prior preclude the possibility of a closed form for the KL. We consider two different approaches when computing sample-based KLs: standard KL and a modified one inspired by free bits (Li et al., 2019; Pelsmaeker and Aziz, 2020; Kingma et al., 2016), which we refer to as FB KL . in computing the KL divergence with N samples: KL , ( x, { z ( i ) } Ni =1 ) = 1 NN (cid:88) i =1 (log q  ( z ( i ) | x )  log p  ( z ( i ) )) (8) For the FB KL, we follow prior work (Kingma et al., 2016) that replaces the KL with a hinge loss term in each latent dimension: FB KL , ( x, { z ( i ) } Ni =1 ) = d (cid:88) j =1 max( , KL j, ( x, { z ( i ) } Ni =1 )) where KL j, denotes the KL computed only for dimension j of the latent variable, and  is the target rate hyperparameter. We describe our training procedure below for FlowPrior, which combines a real NVP prior with the objective in Eq. 7. For simplicity, our description only uses one input x . In practice, we use mini-batches with a stochastic gradient based optimizer. All the parameters ( , ,  ) are updated simultaneously during training. 1. Draw N samples z (1) L , z (2) L , ..., z ( N ) L from the inference network using the reparameterization trick. 2. Perform the inverse transformation to get the image of each point under the base distribution: z (1)0 , z (2)0 , ..., z ( N ) 0 . 3. Compute the exact log likelihood of the sample prior with change of variable theorem (Eq. 6). 4. Compute and backpropagate the loss (Eq. 7). When using the other priors (standard Gaussian, MoG, and VampPrior), we do not need steps 2 and 3 above because those priors can be computed directly without the inverse transformation or change of variable theorem. We consider four widely-used, publicly available English datasets: the Penn Treebank (PTB) (Mar-cus et al., 1993; Bowman et al., 2016), Yahoo (Yang et al., 2017; He et al., 2019), Yelp sentiment (Shen et al., 2017), and SNLI (Bowman et al., 2015). Our baselines include standard VAE with linear KL annealing (Bowman et al., 2016); Cyc-VAE (Fu et al., 2019) in which the KL term is reweighted with a cyclical annealing schedule; Lag-VAE (He et al., 2019) which updates the encoder multiple times before each decoder update; VAE+FB (Kingma et al., 2016; Chen et al., 2017) which replaces the standard KL with FB (i.e., Eq. 9 with N = 1 ); and Pre-VAE+FB (Li et al., 2019) which initializes the VAE with a pretrained autoencoder and replaces standard KL with FB. We evaluated these baselines using their open source implementations. 2 In addition, we include two prior-learning baselines: MoG-VAE (Eq. 2) and Vamp-VAE (Eq. 3). We follow Pelsmaeker and Aziz (2020) and set 100 components/pseudo-inputs. Unlike the earlier baselines, for which we used open source codebases, we implemented the MoG-VAE and Vamp-VAE models on top of our standard VAE implementation, which was also used for FlowPrior. Across all the experiments for our implemented baselines (i.e., standard VAE, MoG-VAE, Vamp-VAE) and our proposed model FlowPrior, we follow prior work (Kim et al., 2018; He et al., 2019; Li et al., 2019) and use a single-layer LSTM encoder and decoder with a 32-dimensional latent variable. We use a batch size of 32 and train using SGD. 3 5.4 Evaluation Metrics Our evaluation measures language modeling performance, the use of the latent variable, and the quality and diversity of generations from the prior and posterior. The metrics are listed below: PPL: We estimate log marginal likelihood using importance sampling (Burda et al., 2016) and calculate perplexity on the test set. 4 KL: We report the KL term in the ELBO on the test set. When training with FB KL, we still report standard KL. For standard VAE, we compute KL with its closed-form expression. Otherwise, we report the KL estimated with samples. 2 The links to their implementations are in the Appendix. 3 We use the open source implementations for other baselines. All models are trained with the simple linear annealing schedule, with same hyperparameter search space. We run each setting with 5 random seeds and report the medians. 4 We use 1000 samples which appears to be more than sufficient for estimation; Ziegler and Rush (2019) found that using more than 50 samples did not even show much difference. MI: We follow Hoffman and Johnson (2016) and report estimated mutual information between the observation and its latent variable. AU: A dimension z in the latent variable is considered active if Cov x ( E z (cid:118) q ( z | x ) [ z ]) > 10  2 . AU is then the number of active latent dimensions (Burda et al., 2016). F-PPL and R-PPL: These metrics measure the correspondence between generated sentences from the model and the training corpus. We evaluate both F-PPL and R-PPL by estimating 5-gram language models using the KenLM toolkit (Heafield, 2011) with its default smoothing method. For F-PPL, we estimate language models from the actual text and compute the perplexity of the generated samples. For R-PPL, we estimate language models from the generated samples and compute the perplexity of the actual text. 5 Self-BLEU: The self-BLEU metric is one measure of the diversity of a set of samples (Zhu et al., 2018). It is calculated by averaging the BLEU scores computed between all pairs of samples. We first perform language modeling tasks to characterize models' efficacy at modeling texts in terms of modeling the distribution of language data and making use of the latent variable. We refer to our model as FlowPrior , which uses the training objective in Eq. 7 which includes MIS and the standard KL (Eq. 8). We use FlowPrior + FB to refer to our model with the FB KL (Eq. 9). 5 Our R-PPL is slightly different from that in Fang et al. (2019). For R-PPL, we always concatenate the training set vocabulary (one word per line) to the set of samples from the models to ensure LMs have seen the entire vocabulary. Comparison to baselines. Table 1 shows results on the PTB dataset for several VAEs from prior work and our implemented models. Since our contributions lie in learning the prior instead of changing the training procedure or manipulating the KL term, we set the baselines as standard VAE, MoG, and VampPrior for the rest of the paper. We report the performance of FlowPrior and those baselines on Yahoo, Yelp, and SNLI in Table 2. From Tables 1 and 2, we observe that FlowPrior consistently outperforms the baselines in test set perplexity, sometimes by large margins. This is not surprising since the MIS term in our training objective directly targets the perplexity metric because the expressions are identical (differing only in the number of samples used). While FB typically improves models on PTB, and helps FlowPrior to reach a higher AU and KL on the other datasets, it does not lead to better test PPL and reconstruction. We report additional results on measuring the impact of FB in the Appendix. Another finding is that simply enriching the parametric family of the prior is not sufficient to improve our evaluation metrics. Tables 1 and 2 show mixed results when moving from the VAE with its standard Gaussian prior to the MoGor Vamp-VAE. Though these priors have the potential to be multimodal, they could still be unimodal after training. For example, the MoG-VAE might learn a mixture in which all Gaussians have the same location and Prior PPL(  ) KL AU(  ) PTB Standard 101.8 / 101.4 / 98.4 0.0 / 0.0 / 3.2 0 / 0 / 2 MoG 101.9 / 98.2 / 96.7 0.0 / 0.0 / 0.0 0 / 0 / 0 Vamp 101.7 / 98.3 / 96.1 0.0 / 0.0 / 3.1 0 / 0 / 4 Real NVP 102.5 / 98.4 / 94.7 0.0 / 0.0 / 3.3 0 / 0 / 2 Yahoo Standard 65.6 / 65.8 / 63.9 0.0 / 0.0 / 2.7 0 / 0 / 1 MoG 65.6 / 64.6 / 62.7 0.0 / 0.0 / 0.5 0 / 0 / 1 Vamp 78.5 / 74.8 / 62.9 0.0 / 0.0 / 1.5 0 / 0 / 2 Real NVP 65.6 / 65.8 / 62.5 0.0 / 0.0 / 1.4 0 / 0 / 4 Yelp Standard 35.4 / 35.1 / 33.2 0.0 / 0.0 / 2.9 0 / 0 / 2 MoG 36.0 / 35.2 / 34.9 0.0 / 0.0 / 0.0 0 / 0 / 0 Vamp 38.0 / 35.0 / 33.7 0.0 / 0.0 / 4.1 0 / 0 / 1 Real NVP 35.6 / 35.1 / 31.8 0.0 / 0.0 / 4.2 0 / 0 / 2 SNLI Standard 27.4 / 26.0 / 25.3 0.0 / 0.0 / 1.2 0 / 0 / 3 MoG 27.2 / 28.1 / 24.3 0.0 / 0.4 / 4.2 0 / 1 / 5 Vamp 27.6 / 26.0 / 23.7 0.0 / 0.0 / 2.8 0 / 0 / 2 Real NVP 27.7 / 26.1 / 22.4 0.0 / 0.0 / 3.8 0 / 0 / 3 Table 3: Comparing training objectives with several choices for priors. scale. Also, the complexity of the prior learned by the Vamp-VAE is dependent upon the inference network, so if the inference network does not learn anything useful, the learned prior may not be useful either. Impact of selection of objectives. The learned prior baselines (MoG-VAE and Vamp-VAE) fail to learn to use the latent variable, as shown by the small numbers (nearly zero) for the AU and MI metrics in Tables 1-2. Similar observations were made by Pelsmaeker and Aziz (2020). We argue that only improving the prior might not be sufficient, as the ELBO objective is difficult to optimize and little information may be learnable for the prior from the ELBO alone. To measure the utility of the MIS term, we introduce this term to standard-VAE, MoG-VAE, and Vamp-VAE and evaluate the improved models under the same language model metrics. Table 3 compares models trained with MIS , the ELBO, and the combined training objective (Eq. 7). The combined objective is beneficial to all metrics for all priors and datasets. Our results are consistent with the observations of Rainforth et al. (2018) that tighter bounds are preferable for training the gener-Vamp-VAE + MIS Three people are sitting on a bench . ative network, while looser bounds are preferable for training the inference network. Still, FlowPrior (real NVP + MIS ) performs the best in PPL and MI compared to other models, showing the flexibility and the power of the real NVP architecture. For the Standard setting in Table 3, the prior is fixed and not learned while in the other three settings the prior is learned. The combination of ELBO and MIS is helpful across all settings. 6 6.2 Interpolations Between Prior Samples One appealing aspect of VAEs for sentence modeling is the potential for learning a smooth, interpretable space for sentences. A qualitative way to explore the latent space is to interpolate between samples from the prior distribution. We randomly sample two latent vectors from the prior and linearly interpolate between them with evenly divided intervals (Bowman et al., 2016). 7 We use greedy 6 For the MoG setting, we also performed experiments with setting the number of Gaussian components K = 1 and observed comparable or slightly worse test PPL under all 3 choices of training loss than Standard setting. 7 FlowPrior is slightly different. Instead of directly sampling from the latent variable of VAE (in MoG-VAE and VampVAE), FlowPrior samples from the base distribution of real NVP, interpolates in the base distribution, and maps to the decoding in generation. 8 Table 4 shows linear interpolation between prior samples in FlowPrior and Vamp-VAE + MIS (i.e., Vamp-VAE with the combined training objective). We observe substantial improvement with FlowPrior, as it can generate sentences with smooth semantic evolution while maintaining plausible generations in terms of fluency. This semantic evolution may reflect the complex structure in the learned prior distribution. Interpolations with MoG-VAE + MIS and Vamp-VAE + MIS have more repetitions and do not transit smoothly from one to the other. (Results with MoG-VAE are in the appendix.) 6.3 Visualization of Learned Priors We randomly select 4 dimensions from the learned priors per model and plot their densities in Fig. 1. In MoG-VAE, each dimension is a Gaussian mixture with 100 components. When only using the ELBO for training (Fig. 1(a)), the four visualized components all have similar shapes. After adding MIS (Fig. 1(b)), different dimensions have similar locations but different scales. Vamp-VAE permits relatively complex components because the means and variances are acquired from the inference network applied to learned latent with Eq. 4. We also experiment with interpolating the two samples after mapping, namely interpolating in the VAE latent space, and find similar results. 8 We additionally tried various sampling methods for decoding. This leads to more noise and becomes harder to interpret. Generations can be found in the Appendices. pseudo-inputs. Fig. 1(c) shows that Vamp-VAE trained without MIS does not show much difference compared to MoG-VAE. However, when training with MIS (Fig. 1(d)), the distributions in several dimensions appear to be multimodal. The real NVP prior learns little information when training without MIS , as all dimensions are akin to standard normal distributions. When training with MIS , different dimensions show distinct placement and shape. The prior in FlowPrior is highly multimodal overall and smooth in each dimension. Sampling from Prior. To measure the expressiveness of the prior and the richness of the learned latent variable, we randomly sample 5000 times from the prior distribution and evaluate their greedy-decoded generations qualitatively and quantitatively. Table 5 shows greedy generations from prior samples. We observe substantial improvements in term of generation diversity when adding MIS in the training objective. Note that these diverse samples are achieved with a purely deterministic decoding. A diverse set of samples implies that (1) richer latent codes and a highly multimodal distribution is learned by the model; (2) and the generator is trained to attend to the latent codes. Sample Mundanity and Coverage. A strongly-performing generative model should be able to generate samples that comport to the training data distribution. We use the forward and reverse PPL to estimate the similarity between the training data and samples. We can consider F-PPL as a generation precision as it reflects the amount of information in the samples that is relevant to the actual text. Analogously, we can consider R-PPL Yelp SNLIF-PPL R-PPL SB F-PPL R-PPL SB VAE 4 30248 96 4 51127 100 VAE+M IS 5 10818 30 4 19047 73 Vamp-VAE 4 32504 100 4 56050 100 Vamp-VAE+M IS 7 5280 10 5 8420 29 FlowPrior 209 1677 3 42 5725 13 Table 6: Forward PPL (F-PPL), Reverse PPL (R-PPL), and Self-BLEU (SB) of greedy-decoded prior samples. as a generation recall' ' as it reflects how much the samples as a whole provide coverage of the actual text. Moreover, both F-PPL and R-PPL reflect whether the decoder is able to attend to the latent variable in generation. Table 6 shows the F-PPL and R-PPL with greedy generation from prior samples. While Fang et al. (2019) treats a lower F-PPL as an indicator of better samples, we argue that it is not necessarily true. A model could achieve a low F-PPL by simply generating identical (or nearly-identical) high-probability sequences, like those observed from the VAE, MoG-VAE, and Vamp-VAE in Table 5. This reflects how an overly-simplified or restrictive assumption in the prior can lead to less diversity in samples. Indeed, we find that models with very low F-PPL values often have very high R-PPL values. A lower R-PPL indicates the distribution of generated samples matches the distribution of the training data. From Table 6 we observe that adding MIS is beneficial as it leads to a lower R-PPL. FlowPrior has the best R-PPL, and shows the capability of capturing characteristics of the target distribution that are not captured by simpler priors. Generation Diversity. To identify which model has richer usage of latent variables, we use self-BLEU to measure the diversity of a set of samples. We observe significant improvements in FlowPrior in Table 6, which implies a diverse latent representation and a better utilization of the latent variable. When considering the parameterized family of VAE models, expressive latent components (i.e., posterior and prior) have been widely studied in computer vision (Dinh et al., 2015, 2016; Kingma and Dhariwal, 2018). However, multimodal priors have been seldom applied to language, with some exceptions (Serban et al., 2017; He et al., 2018; Ziegler and Rush, 2019; Ma et al., 2019; Lee et al., 2020). Chen et al. (2017) use autoregressive flow for the prior and posterior and experiment with images. Ziegler and Rush (2019) propose several autoregressive NF architectures and characterize performance on character-level language modeling. Ma et al. (2019) design priors using the Glow architecture to improve the performance of non-autoregressive neural machine translation. Lee et al. (2020) empirically characterize the performance of NF and simple Gaussian priors in token-level latent variable models, and observe that flexible priors yield higher log-likelihoods but not better BLEU scores on machine translation tasks. Our work differs from that of Ziegler and Rush (2019) and Chen et al. (2017) as we are using a non-autoregressive flow-based architecture for the prior, while they are using autoregressive NF. Also, we focus on models with a single latent variable for an entire sentence, while similar prior work has focused on token-level latent variables (Ziegler and Rush, 2019; Ma et al., 2019; Lee et al., 2020). Several others have employed NF for flexible modeling in NLP. Setiawan et al. (2020) present a variational translation model that uses NF in the approximate posterior while keeping the prior as Gaussian. Wang and Wang (2019) apply NF to a variational Wasserstein autoencoder in order to make the posterior more flexible. Jin et al. (2019) use transformed distributions via NF to model the emission density, which improves parsing performance as compared to Gaussian baselines. We proposed a method, FlowPrior, that uses normalizing flow to define the prior in a sentence VAE and adds the importance-sampled marginal likelihood (MIS ) as a second term to the standard VAE objective. Our empirical results show FlowPrior yields a substantial improvement in language modeling and generation tasks as compared to prior work. Adding MIS improves performance for other models as well, especially in settings when the prior parameters are being learned. We would like to thank Sam Wiseman, Qingming Tang, and Mingda Chen for helpful discussions, and the anonymous reviewers for their comments that improved this paper.
abstain method objective abstain abstain result objective abstain abstain abstain abstain abstain other abstain abstain abstain method objective abstain abstain method abstain abstain abstain objective objective abstain abstain objective other other other other other other other objective method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result other other other other other other abstain method other other other other objective abstain result abstain other