Variational autoencoders (VAEs) combine latent variables with amortized variational inference, whose optimization usually converges into a trivial local optimum termed posterior collapse , especially in text modeling. By tracking the optimization dynamics, we observe the encoder-decoder incompatibility that leads to poor parameterizations of the data manifold. We argue that the trivial local optimum may be avoided by improving the encoder and decoder parameterizations since the posterior network is part of a transition map between them. To this end, we propose Coupled-VAE, which couples a VAE model with a deterministic autoencoder with the same structure and improves the encoder and decoder parameterizations via encoder weight sharing and decoder signal matching. We apply the proposed Coupled-VAE approach to various VAE models with different regularization, posterior family, decoder structure, and optimization strategy. Experiments on benchmark datasets (i.e., PTB, Yelp, and Yahoo) show consistently improved results in terms of probability estimation and richness of the latent space. We also generalize our method to conditional language modeling and propose CoupledC VAE, which largely improves the diversity of dialogue generation on the Switchboard dataset. 1 1 Introduction The variational autoencoder (VAE) (Kingma and Welling, 2014) is a generative model that combines neural latent variables and amortized variational inference, which is efficient in estimating and sampling from the data distribution. It infers a posterior distribution for each instance with a shared inference network and optimizes the evidence lower bound (ELBO) instead of the intractable marginal 1 Our code is publicly available at https://github. log-likelihood. Given its potential to learn representations from massive text data, there has been much interest in using VAE for text modeling (Zhao et al., 2017; Xu and Durrett, 2018; He et al., 2019). Prior work has observed that the optimization of VAE suffers from the posterior collapse problem, i.e., the posterior becomes nearly identical to the prior and the decoder degenerate into a standard language model (Bowman et al., 2016; Zhao et al., 2017). A widely mentioned explanation is that a strong decoder makes the collapsed posterior a good local optimum of ELBO, and existing solutions include weakened decoders (Yang et al., 2017; Semeniuta et al., 2017), modified regularization terms (Higgins et al., 2017; Wang and Wang, 2019), alternative posterior families (Rezende and Mohamed, 2015; Davidson et al., 2018), richer prior distributions (Tomczak and Welling, 2018), improved optimization strategies (He et al., 2019), and narrowed amortization gaps (Kim et al., 2018). In this paper, we provide a novel perspective for the posterior collapse problem. By comparing the optimization dynamics of VAE with deterministic autoencoders (DAE), we observe the incompatibility between a poorly optimized encoder and a decoder with too strong expressiveness. From the perspective of differential geometry, we show that this issue indicates poor chart maps from the data manifold to the parameterizations , which makes it difficult to learn a transition map between them. Since the posterior network is a part of the transition map, we argue that the posterior collapse would be mitigated with better parameterizations. To this end, we propose the Coupled-VAE approach, which couples the VAE model with a deterministic network with the same structure. For better encoder parameterization, we share the encoder weights between the coupled networks. For better decoder parameterization, we propose a signal matching loss that pushes the stochastic decoding signals to the deterministic ones. Notably, our approach is model-agnostic since it does not make any assumption on the regularization term, the posterior family, the decoder architecture, or the optimization strategy. Experiments on PTB, Yelp, and Yahoo show that our method consistently improves the performance of various VAE models in terms of probability estimation and the richness of the latent space. The generalization to conditional modeling, i.e., CoupledC VAE, largely improves the diversity of dialogue generation on the Switchboard dataset. Our contributions are as follows:  We observe the encoder-decoder incompatibility in VAE and connect it to the posterior collapse problem. We propose the Coupled-VAE, which helps the encoder and the decoder to learn better parameterizations of the data manifold with a coupled deterministic network, via encoder weight sharing and decoder signal matching. Experiments on PTB, Yelp, and Yahoo show that our approach improves the performance of various VAE models in terms of probability estimation and richness of the latent space. We also generalize Coupled-VAE to conditional modeling and propose CoupledC VAE, which largely improves the diversity of dialogue generation on the Switchboard dataset. The generative process of VAE is first to sample a latent code z from the prior distribution P ( z ) and then to sample the data x from P ( x | z ;  ) (Kingma and Ba, 2015). Since the exact marginalization of the log-likelihood is intractable, a variational family of posterior distributions Q ( z | x ;  ) is adopted to derive the evidence lower bound (ELBO), i.e., log P ( x ;  )  E z Q ( z | x ;  ) [log P ( x | z ;  )]  KL[ Q ( z | x ;  ) (cid:107) P ( z )] (1) For training, as shown in Figure 1(a), the encoded text e is transformed into its posterior via a posterior network. A latent code is sampled and mapped to the decoding signal h . Finally, the decoder infers the input with the decoding signal. The objective can be viewed as a reconstruction loss L rec plus a regularization loss L reg (whose form varies), i.e., L = L rec + L reg (2)  Encoder(  ) f    Posterior(  |  )   Decoder(  |  )   MLP(  ) g    reg  rec However, the optimization of the VAE objective is challenging. We usually observe a very small L reg and a L rec similar to a standard language model, i.e., the well-known posterior collapse problem. An older family of autoencoders is the deterministic autoencoder (DAE) (Rumelhart et al., 1986; Ballard, 1987). Figure 1(b) shows an overview of DAE for text modeling, which is composed of a text encoder, an optional MLP, and a text decoder. The reconstruction loss of DAE is usually much lower than that of VAE after convergence. To understand the posterior collapse problem, we take a deeper look into the training dynamics of VAE. We investigate the following questions. How much backpropagated gradient does the encoder receive from reconstruction? How much does it receive from regularization? How much information does the decoder receive from the encoded text? To answer the first question, we study the gradient norm of the reconstruction loss w.r.t. the encoded text, i.e., (cid:107)  L rec / e (cid:107) 2 , which shows the magni-0k tude of gradients received by the encoder parameters. From Figure 2(a), we observe that it constantly increases in DAE, while in VAE it increases marginally in the early stage and then decreases continuously. It shows that the reconstruction loss actively optimizes the DAE encoder, while the VAE encoder lacks backpropagated gradients after the early stage of training. We seek the answer to the second question by studying the gradient norm of the regularization loss w.r.t. the encoded text, i.e., (cid:107)  L reg / e (cid:107) 2 . In a totally collapsed posterior, i.e., Q ( z | x ;  ) = P ( z ) for each x , (cid:107)  L reg / e (cid:107) 2 would be zero. Thus, (cid:107)  L reg / e (cid:107) 2 can show how far the posterior of each instance is from the aggregate posterior or the prior. Figure 2(b) shows a constant decrease of the gradient norm in VAE from the 2.5K step until convergence, which shows that the posterior collapse is aggravated as the KL weight increases. For the third question, we compute the normalized gradient norm of the decoding signal w.r.t. the encoded text, i.e., (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 . As this term shows how relatively the decoding signal changes with the perturbation of the encoded text, it reflects the amount of information passed from the encoder to the decoder. Figure 2(c) shows that for DAE, it constantly increases. For VAE, it at first increases even faster than DAE, slows down, and finally decreases until convergence, indicating that the VAE decoder, to some extent, ignores the encoder in the late stage of training. Based on the training dynamics in Section 3.1 and the observations in previous work (Bowman et al., 2016; Zhao et al., 2017), text VAE has three features, listed as follows. First, the encoder is poorly optimized, as shown by the low (cid:107)  L rec / e (cid:107) 2 . Second, the decoder degenerates into a powerful language model. Third, h contains less information from e in VAE than in DAE, which is indicated by the lower (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 . We call these features as encoder-decoder incompatibility . To bridge the incompatibility and posterior collapse, we start with the manifold hypothesis which states that real-world data concentrates near a manifold with a lower dimensionality than the ambient space (Narayanan and Mitter, 2010; Bengio et al., 2013). In our case, we denote the manifold of text data as X  (cid:83) l  NV l where V is the vocabulary. In the language of differential geometry, the encoded text e  E  R d and the decoding signal h  H  R d can be viewed as the parameterizations (or coordinates ) of x  X under two different charts (or coordinate systems ). Formally, we denote the chart maps as  e : X  E and  h : X  H , which satisfy e =  e ( x ) and h =  h ( x ) for any x  X . Given the two charts, the map from E to H is called the transition map  h    1 e : E  H between the two charts. In DAE, the two chart maps and the transition map between them are learned simultaneously via the single reconstruction loss, which we rewrite as L rec = E x X [ L ( x,   1 h (  h    1 e (  e ( x ))))] (3) where  e ,  h    1 e , and   1 h are modeled as the encoder, the MLP, and the decoder (strictly speaking, in text modeling, the range of   1 h is not X but distributions on X ), as illustrated in Figure 3. In VAE, as discussed before, both  e and  h inadequately parameterize the data manifold. We argue that the inadequate parameterizations make it harder to find a smooth transition map in VAE than in DAE, as shown by the lower (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 . Since the posterior network is a part of the transition map, it consequently seeks to map each instance to the prior (discussed in Section 3.1) rather than learning the transition map. Based on the above analysis, we argue that posterior collapse could be alleviated by learning chart maps (i.e.,  e and  h ) that better parameterize the data manifold. Inspired by the chart maps in DAE, we propose to couple the VAE model with a deterministic network, outlined in Figure 3. Modules with a subscript c are deterministic networks that share the structure with those in the stochastic network. Sampling is disabled in the deterministic network, e.g., in the case of Gaussian posterior, we use the predicted mean vector for later computation. Please find details for other posterior families in Appendix B. Similar to DAE, the coupled deterministic network is optimized solely by the coupled reconstruction loss L c rec , which is the same autoregressive cross-entropy loss as L rec . To learn a well-optimized  e , we share the encoder between the stochastic and the deterministic networks, which leverages the rich gradients backpropagated from L c rec . To learn better  h , we propose to guide  h with a well-learned chart map, i.e., the one characterized by Decoder c . Thus, we introduce a signal matching loss L match that pushes the h to h c . The objective of our approach is L = L rec + L reg +  r L c rec +  m L match (4) where  r and  m are hyperparameters 2 , L c rec is the coupled reconstruction loss, and the signal matching loss L match is essentially a distance function 2 To avoid heavy hyperparameter tuning, we set  r = 1 . 0 unless otherwise specified. between h and h c . We evaluate both the Euclidean distance and the Rational Quadratic kernel 3 , i.e., L match = (cid:40) (cid:107) h  Detach( h c ) (cid:107) 2 Eucl (cid:80) s  s  C s  C + (cid:107) h  Detach( h c ) (cid:107) 2 RQ (5) where s  { 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 } , C is a hyperparameter, and Detach prevents gradients to be propagated into h c since we would like h c to guide h but not the opposite. One would question the necessity of sharing the structure of the posterior network by resorting to universal approximation (Hornik et al., 1989). Specifically, a common question is: why not using an MLP as Posterior c ? We argue that each structure has a favored distribution of H in R d , so structure sharing facilitates the optimization when we are learning by gradient descent. For example, the latent space learned by planar flows (Rezende and Mohamed, 2015) has compression and expansion, and vMF-VAE (Xu and Durrett, 2018), which is supported on a sphere, may significantly influence the distribution of H in its ambient space R d . We conduct the experiments on three commonly used datasets for text modeling, i.e., the Penn Tree-bank (PTB) (Marcus et al., 1993), Yelp (Xu et al., 2016), and Yahoo. The training/validation/test splits are 42K/3370/3761 for PTB, 63K/7773/8671 for Yelp, and 100K/10K/10K for Yahoo. The vocabulary size for PTB/Yelp/Yahoo is 10K/15K/20K. We discard the sentiment labels in Yelp. We evaluate the proposed Coupled-VAE approach by applying it to various VAE models, which in-3 clude VAE (Kingma and Welling, 2014),  -VAE (Higgins et al., 2017), vMF-VAE (Xu and Durrett, 2018; Davidson et al., 2018) with learnable  , CNN-VAE (Yang et al., 2017), WAE (Tolstikhin et al., 2018), VAE with normalizing flows (VAE-NF) (Rezende and Mohamed, 2015), WAE with normalizing flows (WAE-NF), VAE with cyclic annealing schedule (CycAnn-VAE) (Fu et al., 2019), VAE with encoder pretraining and the free bits objective (PreFB-VAE) (Li et al., 2019), and Lagging-VAE (He et al., 2019). We also show the result of GRU-LM (Cho et al., 2014) and SA-VAE (Kim et al., 2018). We do not apply our method to SA-VAE since it does not follow amortized variational inference. Please find more details in Appendix C and previous footnotes. We report negative log-likelihood (NLL), KL divergence, and perplexity as the metrics for language modeling. NLL is estimated with importance sampling, KL is approximated by its Monte Carlo estimate, and perplexity is computed based on NLL. Please find the metric details in Appendix D. Table 1 displays the language modeling results. For all models, our proposed approach achieves smaller negative log-likelihood and lower perplexity, which shows the effectiveness of our method to improve the probability estimation capability of various VAE models. Larger KL divergence is also observed, showing that our approach helps address the posterior collapse problem. Language modeling results only evaluate the probability estimation ability of VAE. We are also interested in how rich the latent space is. We report the mutual information (MI) between the text x and the latent code z under Q ( z | x ) , which is approximated with Monte Carlo estimation. Better PTB Yelp Yahoo MI BLEU-1/2 MI BLEU-1/2 MI BLEU-1/2 VAE 10.48 23.2 / 4.4 8.28 28.7 / 5.3 15.43 21.2 / 3.6 Coupled-VAE 11.99 23.4 / 4.5 9.65 30.4 / 5.8 16.44 23.1 / 4.1  (0 . 8) -VAE 15.43 24.5 / 4.9 13.52 30.6 / 6.0 24.16 24.0 / 4.3 Coupled (0 . 8) -VAE 18.13 24.3 / 4.8 17.69 32.6 / 6.6 28.03 26.4 / 4.9  (1 . 2) -VAE 9.16 22.8 / 4.3 6.60 28.0 / 5.0 11.83 18.2 / 2.9 Coupled (1 . 2) -VAE 10.28 22.9 / 4.2 7.90 29.8 / 5.6 13.51 22.4 / 3.8 vMF-VAE 1.74 15.2 / 2.0 0.03 22.4 / 2.8 2.06 8.5 / 1.1 Coupled-vMF-VAE 2.37 16.1 / 2.3 2.60 25.1 / 4.0 3.37 10.3 / 1.4 CNN-VAE 78.49 32.0 / 7.8 17.26 32.9 / 7.1 30.18 24.9 / 5.3 Coupled-CNN-VAE 80.54 31.8 / 7.7 19.15 33.4 / 7.3 37.62 26.9 / 5.9 WAE 15.09 24.8 / 5.1 15.08 30.7 / 6.1 24.73 24.2 / 4.5 Coupled-WAE 18.51 24.7 / 5.1 18.56 32.5 / 6.6 30.08 27.7 / 5.3 VAE-NF 5.63 19.2 / 3.3 5.64 25.6 / 4.5 8.02 13.7 / 2.1 Coupled-VAE-NF 5.86 19.4 / 3.3 6.06 26.3 / 4.6 9.14 15.3 / 2.5 WAE-NF 7.18 19.7 / 3.5 7.95 26.0 / 4.6 11.43 13.8 / 2.2 Coupled-WAE-NF 8.10 20.7 / 3.7 8.53 27.2 / 5.0 12.56 14.9 / 2.5 CycAnn-VAE 1.55 16.3 / 2.3 1.18 22.6 / 3.2 3.09 8.3 / 1.1 Coupled-CycAnn-VAE 2.27 16.7 / 2.6 2.01 23.1 / 3.4 3.89 10.9 / 1.5 PreFB-VAE 20.6 25.5 / 5.7 20.3 33.1 / 6.8 26.2 27.2 / 5.2 Coupled-PreFB-VAE 23.2 25.8 / 5.8 21.0 33.3 / 6.8 27.0 27.2 / 5.3 Lagging-VAE  2.90 -0.96 -3.04 -Coupled-Lagging-VAE  3.29 -2.36 -3.06 Table 2: Mutual information (MI) and reconstruction. reconstruction from the encoded text is another way to show the richness of the latent space. For each text x , we sample ten latent codes from Q ( z | x ) and decode them with greedy search. We report the BLEU-1 and BLEU-2 scores between the reconstruction and the input. Please find the metric details in Appendix E. In Table 2, we observe that our approach improves MI on all datasets, showing that our approach helps learn a richer latent space. BLEU-1 and BLEU-2 are consistently improved on Yelp and Yahoo, but not on PTB. Given that text samples in PTB are significantly shorter than those in Yelp and Yahoo, we conjecture that it is easier for the decoder to reconstruct on PTB by exploiting its autoregressive expressiveness, even without a rich latent space. We investigate the effect of key hyperparameters. Results are shown in Table 3. Note that the lowest NLL does not guarantee the best other metrics, which shows the necessity to use multiple metrics for a more comprehensive evaluation. For the distance function, we observe that the Euclidean distance (denoted as Eucl in Table 3) is more sensitive to  m than the Rational Quadratic kernel (denoted as RQ in Table 3). The first and the third block in Table 3 show that, with larger  m , the model achieves higher KL divergence, MI, and reconstruction metrics. Our interpretation is that by pushing the stochastic decoding signals closer to the deterministic ones, we get latent codes with richer text information. We leave the analysis of  m = 0 . 0 in Section 5.6. The second block in Table 3 shows the role of  r , which we interpret as follows. When  r is too small (e.g., 0 . 5 ), the learned parameterizations are still inadequate for a smooth transition map; when  r is too large (e.g., 5 . 0 ), it distracts the optimization too far away from the original objective (i.e., L rec + L reg ). Note that  r = 0 . 0 is equivalent to removing the coupled reconstruction loss L c rec in Eq. (4)). In Section 5.5 we observe richer latent space (i.e., larger MI and BLEU scores) with larger  m . However, a richer latent space does not guarantee a better probability estimation result. Thus, in this PTB Yelp Dist  m  r NLL (KL) PPL MI BLEU-1/2 NLL (KL) PPL MI BLEU-1/2 RQ 0 . We study three models of different posterior families (i.e., Coupled-VAE, Coupled-VAE-NF, and Coupled-vMF-VAE). Results are shown in Table 4, where we do not report the KL, MI, and BLEU scores because they have been shown to be improved with larger  m in Table 3. We observe that the effects of signal matching on probability estimation vary in different posterior families. We study the three gradient norms defined in Section 3 on the test sets, displayed in Table 5 (for Coupled-VAE,  m = 0 . 1 ). Notably, (cid:107)  L c rec / e (cid:107) 2 in Coupled-VAE is even larger than (cid:107)  L rec / e (cid:107) 2 in DAE. It has two indications. First, the encoder indeed encodes rich information of the text. Second, compared with DAE, Coupled-VAE better generalizes to the test sets, which we conjecture is due to the regularization on the posterior. Coupled-VAE also has a larger (cid:107)  L reg / e (cid:107) 2 compared with VAE, which based on the argument in Section 3.1 indicates that, in Coupled-VAE, the posterior of each instance is not similar to the prior. We also observe larger (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 in Coupled-VAE, which indicates a better transition map between the two parameterizations in Coupled-VAE than in VAE. We also track the gradient norms of Coupled-VAE (  m = 10 . 0 for a clearer comparison), plotted along with VAE and DAE in Figure 2. The curve for Coupled-VAE in Figure 2(a) stands for (cid:107)  ( L rec + L c rec ) / e (cid:107) 2 . We observe that Coupled-VAE receives constantly increasing backpropagated gradients from the reconstruction. In contrast to VAE, the (cid:107)  L reg / e (cid:107) 2 in Coupled-VAE does not decrease significantly as the KL weight increases. The decrease of (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 , which VAE suffers from, is not observed in Coupled-VAE. Plots on more datasets are in Appendix F. 5.8 Sample Diversity We evaluate the diversity of the samples from the prior distribution. We sample 3200 texts from the prior distribution and report the Dist-1 and Dist-2 metrics (Li et al., 2016), which are the ratios of distinct unigrams and bigrams over all generated unigrams and bigrams. Distinct-1 and Distinct-2 in Table 6 show that texts sampled from Coupled-VAE (  m = 10 . 0 ) are more diverse than those from VAE. Given limited space, we put several samples in Appendix G for qualitative analysis. A property of VAE is to match the interpolation in the latent space with the smooth transition in the data space (Bowman et al., 2016). In Table 7, we show the interpolation of VAE and Coupled-VAE on PTB. It shows that compared with VAE, Coupled-VAE has smoother transitions of subjects (cid:107)  L rec / e (cid:107) 2 (cid:107)  L c rec / e (cid:107) 2 (cid:107) (  L rec + L c rec ) / e (cid:107) 2 (cid:107)  L reg / e (cid:107) 2 (cid:107)  h / e (cid:107) F / (cid:107) h (cid:107) 2 PTB DAE 1719.8 --3.14 VAE 112.5 -19.4 2.05 Coupled-VAE 148.5 2109.6 2320.2 27.7 2.12 Yelp DAE 2443.6 --2.55 VAE 59.7 -18.8 1.62 Coupled-VAE 84.8 3640.8 3764.7 25.0 2.25 Yahoo DAE 4104.6 --3.39 VAE 257.9 -52.8 2.92 Coupled-VAE 335.3 5105.0 5615.0 65.0 3.91 Table 5: Gradient norms defined in Section 3.1 on each test set. ( both sides  it ) and verbs ( are expected  have been  has been  has ), indicating that the linguistic information is more smoothly encoded in the latent space of Coupled-VAE. To generalize our approach to conditional language modeling, we propose CoupledC VAE. A graphical overview is displayed in Figure 4. Specifically, the (coupled) posterior network and the (coupled) decoder are additionally conditioned. The objective of Coupled-CVAE is identical to Eq. (4). We compare Couple-CVAE with GRU encoder-decoder (Cho et al., 2014) and CVAE (Zhao et al., 2017) for dialogue generation. We use the Switchboard dataset (John and Holliman, 1993), whose training/validation/test splits are 203K/5K/5K, and the vocabulary size is 13K. For probability estimation, we report the NLL, KL, and PPL based on the gold responses. Since the key motivation of using CVAE in Zhao et al. (2017) is the diversity of responses, we sample one response for each post and report the Distinct-1 and Distinct-2 metrics over all samples. Please find more details in Appendix I. Table 8 shows that Coupled-CVAE greatly increases the diversity of dialogue modeling, while it only slightly harms the probability estimation capability. It indicates that Coupled-CVAE better captures the one-to-many nature of conversations than CVAE and GRU encoder-decoder. We also observe that the diversity is improved with increasing  m , which shows that  m can control diversity via specifying the richness of the latent space. Bowman et al. (2016) identify the posterior collapse problem of text VAE and propose KL annealing and word drop to handle the problem. Zhao et al. (2017) propose the bag-of-words loss to mitigate this issue. Later work on this problem focuses on less powerful decoders (Yang et al., 2017; Semeniuta et al., 2017), modified regularization objective (Higgins et al., 2017; Bahuleyan et al., 2019; Wang and Wang, 2019), alternative posterior families (Rezende and Mohamed, 2015; Xu and Durrett, 2018; Davidson et al., 2018; Xiao et al., 2018), richer prior distributions (Tomczak and Welling, 2018), improved optimization (He et al., 2019) or KL annealing strategy (Fu et al., 2019), the use of skip connections (Dieng et al., 2019), hierarchical or autoregressive posterior distributions (Park et al., 2018; Du et al., 2018), and narrowing the amortization gap (Hjelm et al., 2016; Kim et al., 2018; Marino et al., 2018). We provide the encoder-decoder incompatibility as a new perspective on the posterior collapse problem. Empirically, our approach can be combined with the above ones to alleviate the problem further. A model to be noted is  -VAE (Higgins et al., 2017), in which the reconstruction and regularization are modeled as a hyperparameterized trade-off, i.e., the improvement of one term compromises the other. Different from  -VAE, we adopt the idea of multi-task learning, i.e., the coupled reconstruction task helps improve the encoder chart map and the signal matching task helps improve the decoder chart map. Both our analysis in Section 3.2 and the empirical results show that the modeling of posterior distribution can be improved (but not necessarily compromised) with the additional tasks. Ghosh et al. (2020) propose to substitute stochas-ticity with explicit and implicit regularizations, which is easier to train and empirically improves the quality of generated outputs. Different from their work, we still strictly follow the generative nature (i.e., data density estimation) of VAE, and the deterministic network in our approach serves as an auxiliary to aid the optimization. Encoder pretraining (Li et al., 2019) initializes the text encoder and the posterior network with an autoencoding objective. Li et al. (2019) shows that encoder pretraining itself does not improve the performance of VAE, which indicates that initialization is not strong enough as an inductive bias to learn a meaningful latent space. Given the discrete nature of text data, we highlight the two-level representation learning for text modeling: 1) the encoder and decoder parameterizations via autoencoding and 2) a transition map between the parameterizations. Notably, the transition map has large freedom. In our case, the transition map decides the amount and type of information encoded in the variational posterior, and there are other possible instances of the transition map, e.g., flow-based models (Dinh et al., 2015). In this paper, we observe the encode-decoder incompatibility of VAE for text modeling. We bridge the incompatibility and the posterior collapse problem by viewing the encoder and the decoder as two inadequately learned chart maps from the data manifold to the parameterizations, and the posterior network as a part of the transition map between them. We couple the VAE model with a deterministic network and improve the parameterizations via encoder weight sharing and decoder signal matching. Our approach is model-agnostic and can be applied to a wide range of models in the VAE family. Experiments on benchmark datasets, i.e., PTB, Yelp, and Yahoo, show that our approach improves various VAE models in terms of probability estimation and the richness of the latent space. We also generalize Coupled-VAE to conditional language modeling and propose Coupled-CVAE. Results on Switchboard show that Coupled-CVAE largely improves diversity in dialogue generation. We would like to thank the anonymous reviewers for their thorough and helpful comments.
abstain result result objective objective abstain objective abstain other abstain abstain abstain other objective result result method objective abstain objective abstain result abstain objective objective result objective other other other other other other other abstain other other other other method objective other other other abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other other other objective method other abstain abstain other method other other abstain other method result abstain result method result objective abstain other