Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. We investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincar Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincar ball via wrapped normal distributions. By adopting the primal-dual formulation of Kullback-Leibler divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling, unaligned style transfer, and dialog-response generation demonstrate the effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space. The Variational Autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) is a generative model widely applied to language-generation tasks, which propagates latent codes drawn from a simple prior to manifest data samples through a decoder. The generative model is augmented by an inference network, which feeds observed data samples through an encoder to yield a distribution on the corresponding latent codes. Since natural language often manifests a latent hierarchical structure, it is desirable for the latent code in a VAE to reflect such inherent language structure, so that the generated text can be more natural and expressive. An example of language structure is illustrated in Figure 1, where sentences are organized into a tree structure. The root node corresponds to simple sentences ( e.g. ,  Yes ), while nodes on outer leaves represent sentences with more complex syntactic structure and richer, more specific semantic meaning ( e.g. ,  The food in the restaurant is awesome ) 1 . In existing VAE-based generative models, such structures are not explicitly considered. The latent code often employs a simple Gaussian prior, and the posterior is approximated as a Gaussian with diagonal covariance matrix. Such embeddings assume Euclidean structure, which is inadequate in capturing geometric structure illustrated in Figure 1. While some variants have been proposed to enrich the prior distributions (Xu and Durrett, 2018; Wang et al., 2019a,b; Shi et al., 2019), there is no evidence that structural information in language can be recovered effectively by the model. 2019; Nickel and Kiela, 2017). Informally, hyperbolic space can be considered as a continuous map of trees. For example, a Poincar disk (a hyperbolic space with two dimensions) can represent any tree with arbitrary low distortion (De Sa et al., 2018; Sarkar, 2011). In Euclidean space, however, it is difficult to learn such structural representation even with infinite dimensions (Linial et al., 1995). Motivated by these observations, we propose Adversarial Poincar Variational Autoencoder (APo-VAE), a text embedding and generation model based on hyperbolic representations, where the latent code is encouraged to capture the underlying tree-like structure in language. Such latent structure provides more control of the generated sentences, i.e. , an increase of sentence complexity and diversity can be achieved along some trajectory from a root to its children. In practice, we define both the prior and the variational posterior of the latent code over a Poincar ball, via the use of a wrapped normal distribution (Nagano et al., 2019). To obtain more stable model training and learn more flexible representation of the latent code, we exploit the primal-dual formulation of Kullback-Leibler (KL) divergence (Dai et al., 2018) based on the Fenchel duality (Rockafellar et al., 1966), to adversarially optimize the variational bound. Unlike the primal form that relies on Monte Carlo approximation (Mathieu et al., 2019), our dual formulation bypasses the need for tractable posterior likelihoods via the introduction of an auxiliary dual function. We apply the proposed approach to language modeling, unaligned style transfer and dialog-response generation. For language modeling, in order to enhance the distribution complexity of the prior, we use an additional variational mixture of posteriors prior (VampPrior) design (Tomczak and Welling, 2018) for the wrapped normal distribution. Specifically, VampPrior uses a mixture distribution with components from variational posteriors, coupling the parameters of the prior and variational posterior. For unaligned style transfer, we add a sentiment classifier to our model, and disentangle content and sentiment information by using adversarial training (Zhao et al., 2017a). For dialog-response generation, a conditional model variant of APo-VAE is designed to take into account the dialog context. a major obstacle preventing efficient learning of a VAE on text data. In posterior collapse, the encoder learns an approximate posterior similar to the prior, and the decoder tends to ignore the latent code for generation. Experiments show that our proposed model can effectively avoid posterior collapse. We hypothesize that this is due to the use of a more informative prior in hyperbolic space that enhances the complexity of the latent representation, which aligns well with previous work (Tomczak and Welling, 2018; Wang et al., 2019a) that advocates a better prior design. Our main contributions are summarized as follows. ( i ) We present Adversarial Poincar Variational Autoencoder (APo-VAE), a novel approach to text embedding and generation based on hyperbolic latent representations. ( ii ) In addition to the use of a wrapped normal distribution, an adversarial learning procedure and a VampPrior design are incorporated for robust model training. ( iii ) Experiments on language modeling, unaligned style transfer, and dialog-response generation benchmarks demonstrate the superiority of the proposed approach compared to Euclidean VAEs, as it benefits from capturing informative latent hierarchies in natural language. Let X = { x i } Ni =1 be a dataset of sentences, where each x i = [ x i, 1 , ..., x i,T i ] is a sequence of tokens of length T i . Our goal is to learn p  ( x ) that best models the observed sentences so that the expected log-likelihood is maximized, i.e. , L (  ) = 1 N (cid:80) i log p  ( x i ) . The variational autoencoder (VAE) (Kingma and Welling, 2013; Chen et al., 2018b) considers a latent-variable model p  ( x , z ) to represent sentences, with an auxilary encoder that draws samples of latent code z from the conditional density q  ( z | x ) , known as the approximate posterior. Given a latent code z , the decoder samples a sentence from the conditional density p  ( x | z ) = (cid:81) t p ( x t | x <t , z ) , where the decoding pass takes an auto-regressive form. Together with prior p ( z ) , the model is given by the joint p  ( x , z ) = p  ( x | z ) p ( z ) . The VAE leverages the approximate posterior to derive an evidence lower bound (ELBO) to the (intractable) marginal log-likelihood log p  ( x ) = log (cid:82) p  ( x , z ) d z : L ( x ;  ,  ) = E z  q  ( z | x ) (cid:20) log p  ( x , z ) q  ( z | x ) (cid:21) , (1) where (  ,  ) are jointly optimized during training, and the gap is given by the decomposition log p  ( x ) = L ( x ;  ,  )+ DKL ( p  ( z | x ) (cid:107) q  ( z | x )) , (2) where DKL denotes Kullback-Leibler divergence. Alternatively, the ELBO can be written as: L ( x ;  ,  ) = E z  q  ( z | x ) [log p  ( x | z )]  DKL ( q  ( z | x ) (cid:107) p ( z )) , (3) where the first conditional likelihood and second KL terms respectively characterize reconstruction and generalization capabilities. Intuitively, a good model is expected to strike a balance between good reconstruction and generalization. In most cases, both the prior and variational posterior are assumed to be Gaussian for computational convenience. However, such over-simplified assumptions may not be ideal for capturing the intrinsic characteristics of data that have unique geometrical structure, such as natural language. Riemannian manifolds can provide a more powerful and meaningful embedding space for complex data with highly non-Euclidean structure, that cannot be effectively captured in a vectorial form ( e.g. , social networks, biology and computer graph-ics). Of particular interest is the hyperbolic space (Ganea et al., 2018), where ( i ) the relatively simple geometry allows tractable computations, and ( ii ) the exponential growth of distance in finite dimensions naturally embeds rich hierarchical structure in a compact form. Riemannian Geometry. An n -dimensional Riemannian manifold M n is a set of points locally similar to a linear space R n . At each point x of the manifold M n , we can define a real vector space T x M n that is tangent to x , along with an associated metric tensor g x (  ,  ) : T x M n  T x M n  R which is an inner product on T x M n . Intuitively, a Riemannian manifold behaves like a vector space only in its infinitesimal neighborhood, allowing the generalization of common notation like angle, straight line and distance to a smooth manifold. For each tangent space T x M n , there exists a specific one-to-one map exp x ( v ) : T x M n  M n from an (cid:15) -ball at the origin of T x M n to a neighborhood of x on M n , called the exponential map . We refer to the inverse of an exponential map as the logarithm map , denoted log x ( y ) : M n  T x M n . In addition, a parallel transport P x  x (cid:48) : T x M n  T x (cid:48) M n intuitively transports tangent vectors along a straight line between x and x (cid:48) , so that they remain parallel. This is the basic machinery that allows us to generalize distributions and computations in the hyperbolic space, as detailed in later sections. Poincar Ball Model. Hyperbolic geometry is one type of non-Euclidean geometry with a constant negative curvature. As a classical example of hyperbolic space, an n -dimensional Poincar ball, with curvature parameter c  0 ( i.e. , radius 1  c ), can be denoted as B n c := (cid:8) z  R n | c (cid:107) z (cid:107) 2 < 1 (cid:9) with its metric tensor given by g c z =  2 z g E , where  z = 2 1  c (cid:107) z (cid:107) 2 and g E denotes the regular Euclidean metric tensor. Intuitively, as z moves closer to the boundary 1  c , the hyperbolic distance between z and a nearby z (cid:48) diverges at a rate of 1 1  c (cid:107) z (cid:107) 2   . This implies significant representation capacity, as very dissimilar objects can be encoded on a compact domain. Note that as c  0 , the model recovers the Euclidean space R n , i.e. , the lack of hierarchy. In comparison, a larger c implies a stronger hierarchical organization. 2 Mathematical Operations. We review the closed-form mathematical operations that enable differentiable training for hyperbolic space models, namely the hyperbolic algebra (vector addition) and tangent space computations (ex-ponential/logarithm map and parallel transport). The hyperbolic algebra is formulated under the framework of gyrovector spaces (Ungar, 2008), with the addition of two points z , z (cid:48)  B nc given by the Mbius addition : z  c z (cid:48) := (4) (1 + 2 c (cid:104) z , z (cid:48) (cid:105) + c (cid:107) z (cid:48) (cid:107) 2 ) z + (1  c (cid:107) z (cid:107) 2 ) z (cid:48) 1 + 2 c (cid:104) z , z (cid:48) (cid:105) + c 2 (cid:107) z (cid:107) 2 (cid:107) z (cid:48) (cid:107) 2 . For any point   B nc , the exponential map and the logarithmic map are given for u (cid:54) = 0 and y (cid:54) =  2 The fact that APo-VAE outperforms standard VAE evidences the existence of the hierarchical organization in NLP data. exp c  ( u ) :=   c (tanh(  c c  (cid:107) u (cid:107) 2 ) u  c (cid:107) u (cid:107) ) , log c  ( y ) := 2  c c  tanh  1 (  c (cid:107)   , y (cid:107) )   , y (cid:107)   , y (cid:107) , (5) where   , y := (   )  c y . Note that the Poincar ball model is geodesically complete in the sense that exp c  is well-defined on the full tangent space T  B nc . The parallel transport map from a vector v  T 0 B nc to another tangent space T  B nc is given by P c 0   ( v ) = log c  (   c exp c 0 ( v )) =  c 0  c  v . We first introduce our hyperbolic encoder and decoder, and how to apply reparametrization. We then provide detailed descriptions on model implementation, explaining how the primal-dual form of KL divergence can help stabilize training. Finally, we describe how to adopt VampPrior (Tomczak and Welling, 2018) to enhance performance. A summary of our model scheme is provided in Figure 2. 3.1 Flexible Wrapped Distribution Encoder We begin by generalizing the standard normal distribution to a Poincar ball (Ganea et al., 2018). While there are a few competing definitions of the hyperbolic normal, we choose the wrapped normal as our prior and variational posterior, largely due to its flexibility for more expressive generalization. A wrapped normal distribution NB nc (  ,  ) is defined as follows: ( i ) sample vector v from N ( 0 ,  ) , ( ii ) parallel transport v to u := P c 0   ( v ) , and ( iii ) using exponential map to project u back to z := exp c  ( u ) . For approximate posteriors, (  ,  ) depends on x . We further generalize the (restrictive) hyperbolic wrapped normal by acknowledging that under the implicit VAE (Fang et al., 2019) framework, one does not need the approximate posterior q  ( z | x ) to be analytically tractable. This allows us to replace the tangent space sampling step v  N ( 0 ,  ) in (7) with a more flexible implicit distribution from which we draw samples as v := G ( x ,  ;  1 ) for   N ( 0 , I ) . Note that now  := F ( x ;  2 ) can be regarded as a deterministic displacement vector that anchors embeddings to the correct semantic neighborhood, allowing the stochastic v to only focus on modeling the local uncertainty of the semantic embedding. The synergy between the deterministic and stochastic parts enables efficient representation learning relative to existing alternatives. For simplicity, we denote the encoder neural network as EncNet  , which contains G and F , with parameters  = {  1 ,  2 } . To build a geometry-aware decoder for a hyperbolic latent code, we follow Ganea et al. (2018), and use a generalized linear function analogously defined in the hyperbolic space. A Euclidean linear function takes the form f ( z ) = (cid:104) a , z  b (cid:105) = sign ( (cid:104) a , z  b (cid:105) ) (cid:107) a (cid:107) d E ( z , H a , b ) , where a is the coefficient, b is the intercept, H a , b is a hyperplane passing through b with a as the normal direction, and d E ( z , H ) is the distance between z and hyperplane H . The counterpart in Poincar ball analogously writes f c a , b ( z ) = sign ( (cid:104) a , log c b ( z ) (cid:105) b ) (cid:107) a (cid:107) b d B c ( z , H c a , b ) , (8) where H c a , b = { z  B nc |(cid:104) a , log c b ( z ) (cid:105) b = 0 } , and d B c ( z , H c a , b ) = 1  c sinh  1 (cid:16) 2  c |(cid:104)  b , z , a (cid:105)| (1  c (cid:107)  b , z (cid:107) 2 ) (cid:107) a (cid:107) (cid:17) are the the gyroplane and the distance between z and the gyroplane, respectively. Specifically, we use the hyperbolic linear function in (8) to extract features from the Poincar embedding z . The feature f c a , b ( z ) will be the input to the RNN decoder. We denote the combined network of f c a , b and the RNN decoder as DecNet  , where parameters  contain a and b . While it is straightforward to compute the ELBO (3) via Monte Carlo estimates using the explicit wrapped normal density (Mathieu et al., 2019), we empirically observe that: ( i ) the normal assumption restricts the expressiveness of the model, and ( ii ) the wrapped normal likelihood makes the training unstable. Therefore, we appeal to a primal-dual view of VAE training to overcome such difficul-ties (Rockafellar et al., 1966; Dai et al., 2018; Tao et al., 2019; Fang et al., 2019). Specifically, the KL term in (3) can be reformulated as: DKL ( q  ( z | x ) (cid:107) p ( z )) = max  (9) (cid:110) E z  q  ( z | x )   ( x , z )  E z  p ( z ) exp   ( x , z ) (cid:111) , where   ( x , z ) is the (auxiliary) dual function ( i.e. , a neural network) with parameters  . The primal-dual view of the KL term enhances the approximation ability, while also being tractable computationally. Meanwhile, since the density function in the original KL term in (3) is replaced by the dual function   ( x , z ) , we can avoid direct computation with respect to the probability density function of the wrapped normal distribution. To train our proposed APo-VAE with the primal-dual form of the VAE objective, we follow the training schemes of coupled variational Bayes (CVB) (Dai et al., 2018) and implicit VAE (Fang et al., 2019), which optimize the objective adversarially. Specifically, we update  in the dual function   ( x , z ) to maximize: L 1 = E x  X [ E z  q  ( z | x )   ( x , z )  E z  p ( z ) exp   ( x , z ) ] , (10) Algorithm 1 Training procedure of APo-VAE. where E x  X [  ] denotes the expectation over empirical distribution on observations. Accordingly, parameters  and  are updated to maximize: L 2 = E x  XE z  q  ( z | x ) [ log p  ( x | z )    ( x , z ) ] . Note that the term E x  XE z  q  ( z | x )   ( x , z ) is maximized in (10) while it is minimized in (11), i.e. , adversarial learning. In other words, one can consider the dual function as a discriminative network that distinguishes between the prior z  p ( z ) and the variational posterior z  q  ( z | x ) , both of which are paired with the input data x  X . While the use of a standard normal prior is a simple choice in Euclidean space, we argue that it induces bias in the hyperbolic setup. This is because natural sentences have specific meaning, and it is unrealistic to have the bulk of mass concentrated in the center (this is for low dimension; for high dimensions, it will concentrate near the surface of a sphere, which may partly explain why cosine similarity works favorably compared with Euclidean distance for NLP applications). To reduce the induced bias from a pre-fixed prior, we adopt the VampPrior framework (Tomczak and Welling, 2018), which is a mixture of variational posteriors conditioned on learnable pseudo-data points. Specifically, we consider the prior as a learnable distribution given by p  ( z ) = 1 K (cid:80) Kk =1 q  ( z | s k ) , (12) where q  is the learned approximate posterior, and we call the parameter  := { s k } Kk =1 pseudo inputs. Intuitively, p  ( z ) seeks to match the aggregated posterior (Makhzani et al., 2015): q ( z ) = 1 N (cid:80) Ni =1 q  ( z | x i ) in a cost-efficient manner via parameterizing the pseudo inputs. By replacing the prior distribution p ( z ) in (10) with p  ( z ) , we complete the final objective of the proposed APo-VAE. The detailed training procedure is summarized in Algorithm 1. 4 Related Work VAE for Text Generation. Many VAE models have been proposed for text generation, most of which focus on solving the issue of posterior collapse. The most popular strategy is to alter the training dynamics, keeping the encoder away from bad local optima. For example, variants of KL annealing (Bowman et al., 2016; Zhao et al., 2018; Fu et al., 2019) dynamically adjust the weight on the KL penalty term as training progresses. Lagging VAE (He et al., 2019) aggressively optimizes the encoder before each decoder update, to overcome the imbalanced training issue between the encoder and decoder. Alternative strategies have also been proposed based on competing theories or heuristics. -VAE (Razavi et al., 2019) tackles this issue by enforcing a minimum KL divergence between the posterior and the prior. Yang et al. (2017) blames mode-collapse on the auto-regressive design of the decoder and advocates alternative architectures. A semi-amortized inference network is considered by Kim et al. (2018) to bridge the amortization gap between log-likelihood and the ELBO. Recent work has also shown that posterior collapse can be ameliorated by using more expressive priors and variational posteriors other than Gaussian. Flow-based VAE is considered in Ziegler and Rush (2019) to enhance the flexibility of prior distributions. A topic-guided prior is proposed in Wang et al. (2019a) to achieve more controllable text generation. Fang et al. (2019) explores implicit sample-based representations, without requiring an explicit density form for the approximate posterior. Xu and Durrett (2018) considers replacing the Gaussian with the spherical von Mises-Fisher (vMF) distribution. Compared to these prior arts, our model features structured representation in hyperbolic space, which not only captures latent hierarchies but also combats posterior collapse. Hyperbolic Space Representation Learning. There has been a recent surge of interest in representation learning in hyperbolic space, largely due to its exceptional effectiveness modeling data with underlying graphical structure (Chamberlain et al., 2017), such as relation nets (Nickel and Kiela, 2017). In the context of NLP, hyperbolic geometry has been considered for word embeddings (Tifrea et al., 2018). A popular vehicle for hyperbolic representation learning is the autoencoder (AE) framework (Grattarola et al., 2019; Ovinnikov, 2019), where the decoders are built to efficiently exploit the hyperbolic geometry (Ganea et al., 2018). Closest to our APo-VAE are the works of hyperbolic VAEs (Mathieu et al., 2019; Nagano et al., 2019), where wrapped normal distributions have been used. Drawing power from the dual form of the KL, the proposed APo-VAE highlights an implicit posterior and data-driven prior, showing improved training stability. We evaluate the proposed model on three tasks: ( i ) language modeling, ( ii ) unaligned style transfer, and ( iii ) dialog-response generation, with quantitative results, human evaluation and qualitative analysis. Datasets. We use three datasets for language modeling: Penn Treebank (PTB) (Marcus et al., 1993), Yahoo and Yelp corpora (Yang et al., 2017). PTB contains one million words of 1989 Wall Street Journal material annotated in Treebank II style, with 42k sentences of varying lengths. Yahoo and Yelp are much larger datasets, each containing 100k sentences with greater average length. For unaligned style transfer, we use the Yelp restaurant reviews dataset (Shen et al., 2017), which is obtained by pre-processing the Yelp dataset, i.e. , sentences are shortened for more feasible sentence level sentiment analysis. Overall, the dataset includes 350k positive and 250k negative reviews (based on user rating). Following Gu et al. (2019), we use the Switchboard (Godfrey and Holliman, 1997) dataset for dialogue-response generation. The former contains 2.4k two-sided telephone conversations, manually transcribed and aligned. We split the data into training, validation and test sets following the protocol described in Zhao et al. (2017b). Evaluation Metrics. To benchmark language modeling performance, we report the ELBO and Perplexity (PPL) of APo-VAE and baselines. In order to verify our proposed Apo-VAE is more resistant to posterior collapse, we also report the KL-divergence DKL ( q  ( z | x ) (cid:107) p ( z )) and mutual information (MI) between z and x (He et al., 2019). The number of active units (AU) of the latent code is also reported, where the activity of a latent dimension z is measured as A z = Cov x E z  q  ( z | x ) [ z ] , and defined as active if A z > 0 . 01 . To evaluate our model on unaligned style transfer, we consider the transfer accuracy from one sentiment to another, the BLEU scores between original and transferred sentences, the reconstruction perplexity of original sentences, and the reverse perplexity (RPPL) based on a language model from the transferred sentences. For dialogue-response generation, we adopt the evaluation metrics used in previous studies (Zhao et al., 2017b; Gu et al., 2019), including BLEU (Pa-pineni et al., 2002), BOW (Liu et al., 2016), and intra/inter-dist values (Gu et al., 2019). The first two metrics are used to assess the relevance of the generated response, and the third is for diversity evaluation. Model Implementation. For language modeling, we adopt the LSTM (Hochreiter and Schmid-huber, 1997) for both the encoder and decoder, with dimension of the latent code set to 32 . Following Mathieu et al. (2019), the hyper-parameter c is set to 0 . 7 . For unaligned style transfer, we extend our model in the same fashion as Fang et al. (2019). For dialogue-response generation, we modify APo-VAE following the conditional VAE framework (Zhao et al., 2017b). Specifically, an extra input of context embedding s is supplied to the model ( i.e. , p  ( x , z | s ) , q  ( z | x , s ) ). The prior p ( z | s ) is a wrapped normal conditioned on context embedding, learned together with the posterior. Language Modeling. Table 1 shows results on language modeling. We compare APo-VAE with other VAE-based solutions, including  -VAE (Hig-gins et al., 2017), SA-VAE (Kim et al., 2018), lagging VAE (LAG-VAE) (He et al., 2019), vMF-VAE (Xu and Durrett, 2018), Poincar VAE ( P Model -ELBO PPL KL MI AU PTB VAE 102.6 108.26 1.1 0.8 2  -VAE 104.5 117.92 7.5 3.1 5 SA-VAE 102.6 107.71 1.2 0.7 2 vMF-VAE 95.8 93.70 2.9 3.2 21 P-VAE 91.4 76.13 4.5 2.9 23 iVAE 87.2 53.44 12.5 12.2 32 APo-VAE 87.2 53.32 8.4 4.8 32 APo-VAE+VP 87.0 53.02 8.9 4.5 32 Yahoo VAE 328.6 61.21 0.0 0.0 0  -VAE 328.7 61.29 6.3 2.8 8 SA-VAE 327.2 60.15 5.2 2.9 10 LAG-VAE 326.7 59.77 5.7 2.9 15 vMF-VAE 318.5 53.92 6.3 3.7 23 P-VAE 313.4 50.57 7.2 3.3 27 iVAE 309.1 47.93 11.4 10.7 32 APo-VAE 286.2 47.00 6.9 4.1 32 APo-VAE+VP 285.6 46.61 8.1 4.9 32 Yelp VAE 357.9 40.56 0.0 0.0 0  -VAE 358.2 40.69 4.2 2.0 4 SA-VAE 357.8 40.51 2.8 1.7 8 LAG-VAE 355.9 39.73 3.8 2.4 11 vMF-VAE 356.2 51.03 4.1 3.9 13 P-VAE 355.4 50.64 4.3 4.8 19 iVAE 348.7 36.88 11.6 11.0 32 APo-VAE 319.7 34.10 12.1 7.5 32 APo-VAE+VP 316.4 32.91 12.7 6.2 32 Table 1: Results on PTB, Yahoo, and Yelp datasets. A better language model achieves lower negative ELBO and PPL. Higher KL and MI indicate a better utilization of the latent space. VAE) (Mathieu et al., 2019) and iVAE 3 (Fang et al., 2019). On all three datasets, the proposed model achieves lower negative ELBO and PPL than other models, demonstrating its strong ability to better model sequential text data. Meanwhile, the larger KL term and higher mutual information (between z and x ) of APo-VAE model indicate its robustness in handling posterior collapse. In addition, the introduction of a data-driven prior (denoted as APo-VAE+VP) further boosts the performance, especially on negative ELBO and PPL. Visualization. To verify our hypothesis that the proposed model is capable of learning latent tree structure in text data, we visualize the two-dimensional projection of the learned latent code in Figure 3. For visualization, we randomly draw 5k samples from PTB-test, and encode them to the latent space using the APo-VAE encoder. We color-code each sentence based on its length ( i.e. , blue for long sentences and red for short sentences). Note that only a small portion of data have a length longer than 32 ( < 10% ), and human inspection 3 We report iVAE MI results in all our experiments. verified that most of them contain multiple sub-sentences. We exclude these samples from our analysis. As shown in Figure 3, longer sentences (dark blue) tend to occupy the outer rim of the Poincar ball, while the shorter ones (dark red) are concen-vs iVAE vs DialogWAE win loss tie win loss tie Informativeness 52.8 27.9 19.3 63.7 27.1 19.2 Coherence 41.7 35.5 22.8 41.2 34.4 24.4 Diversity 51.2 26.4 22.4 62.1 25.1 12.8 Table 4: Human evaluation results. trated in the inner area. We also select some long sample sentences (dark blue), and manually shorten them to create several variants of different lengths (ranging from 6 to 27), which are related in a hierarchical manner based on human judgement. We visualize their latent codes projected by the trained APo-VAE. The resulting plot is consistent with a hierarchical structure for APo-VAE: as the sentence becomes more specific, the embedding moves outward. We also decode from the neighbours of these latent codes, the outputs (see the Appendix) of which demonstrate a similar hierarchical structure. Unaligned Style Transfer. Table 3 shows the results on the Yelp restaurant reviews dataset. APo-VAE achieves over 1 point increased BLEU scores than iVAE, capturing a more informative and structured feature space. Comparable performance is achieved for the other evaluation metrics. Dialogue Response Generation. Results on SwitchBoard are summarized in Table 2. Our proposed model generates comparable or better responses than the baseline models in terms of both relevance (BLEU and BOW) and diversity (intra/inter-dist). APo-VAE improves the average recall from 0 . 427 (by iVAE) to 0 . 438 , while significantly enhancing generation diversity ( e.g. , from 0 . 692 to 0 . 792 for intra-dist-2). Human Evaluation. We further perform human evaluation via Amazon Mechanical Turk. We asked the turkers to compare generated responses from two models, and assess each model's informativeness, relevance to the dialog context (coherence), and diversity. We use 500 randomly sampled contexts from the test set, each assessed by three judges. In order to evaluate diversity, 5 responses are generated for each dialog context. For quality control, only workers with a lifetime task approval rating greater than 98% were allowed to participate in our study. Table 4 summarizes the human evaluation results. The responses generated by our model are clearly preferred by the judges compared with other competing methods. We present APo-VAE, a novel model for text generation in hyperbolic space. Our model can learn latent hierarchies in natural language via the use of wrapped normals for the prior. A primal-dual view of KL divergence is adopted for robust model training. Extensive experiments on language modeling, text style transfer, and dialog response generation demonstrate the superiority of the model. For future work, we plan to combine APo-VAE with the currently prevailing large-scale pre-trained language models.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method result abstain objective method abstain method abstain abstain abstain objective abstain objective abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain method method method abstain method abstain abstain abstain method abstain abstain abstain method other other method abstain method abstain abstain other abstain method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain method abstain method method abstain abstain method method abstain abstain method abstain abstain method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method