Variational autoencoders (VAEs) have received much attention recently as an endto-end architecture for text generation. Existing methods primarily focus on synthesizing relatively short sentences (with less than twenty words). In this paper, we propose a novel framework, hierarchically-structured variational autoencoders (hier-VAE), for generating long and coherent units of text. To enhance the model's plan-ahead ability, intermediate sentence representations are introduced into the generative networks to guide the word-level predictions. To alleviate the typical optimization challenges associated with textual VAEs, we further employ a hierarchy of stochastic layers between the encoder and decoder networks. Extensive experiments are conducted to evaluate the proposed method, where hier-VAE is shown to make effective use of the latent codes and achieve lower perplexity relative to language models. Moreover, the generated samples from hier-VAE also exhibit superior quality according to both automatic and human evaluations.
This paper proposes a hierarchical variational autoencoder for modeling paragraphs. The model creates two levels of latent variables; one level of sentence-level latent variables and another single global latent. This avoids posterior collapse issues and the authors show convincing results on a few different applications to two datasets.
Overall, it is an impressive result to be able to convincingly model paragraphs with a useful global latent variable. Apart from some issues with confusing/incomplete notation (see below), my main criticism is that the authors fail to compare their approach to "A Hierarchical Latent Structure for Variational Conversation Modeling" by Park et al. As far as I can tell, the approaches are extremely similar, except that Park et al. may not learn the prior parameters and also use a hierarchical RNN encoder rather than a CNN (which may be irrelevant). They also are primarily interested in dialog generation, so the lower-level of their hierarchy models utterances in a conversation rather than sentences in general, but I don't see this as a major difference. I'd encourage the authors to compare to this and potentially use it as a baseline. More generally, it would have been nice to see more ablation experiments (e.g. convolutional vs. LSTM encoder). Finally, I know that space is tight, but other papers on global-latent-variable models tend to include more demonstrations that teh global variable is capturing meaningful information, e.g. with attribute vector arithmetic. The authors could include results of manipulating review sentiment via attribute vector arithmetic, for example.
Specific comments:
- "The Kullback-Leibler (KL) divergence term ... which can be written in closed-form (Kingma & Welling, 2013), encourages the approximate posterior distribution qφ(z|x) to be close to the multivariate Gaussian prior p(z)." The prior is not always taken to be a multivariate Gaussian. You should add a sentence stating that the VAE prior is often taken to be a diagonal-covariance Gaussian for convenience.
- 3.2 has a few things which are unclear. In the second paragraph, you define z as the sampled latent code which is fed through an MLP "to obtain the starting state of the sentence-level LSTM decoder". But then LSTM^{sent} appears to be fed z at every timestep. LSTM^{sent} is also not defined - am I to assume that its arguments are the previous state and current input, so that z is the input at every timestep? Also, you write "where h^s_0 is a vector of zeros" which makes it sound like the starting state of the sentence-level LSTM decoder is a vector of zeros, not the output of the MLP which takes z as input. In contrast, LSTM^{word} takes three arguments as input. Which are the "state" and which are the "input" to the LSTM?
- I don't see any description of your CNN encoder (only the LSTM decoder in section 3.2, 3.3 only covers the hierarchy of latent variables, not the CNN architecture). What is its structure? Figure 1 shows a CNN encoder generating lower-level sentence embeddings and a high-level global embedding. How are those computed? It is briefly mentioned in 4.1 under "Datasets" but this seems insufficient.
- p_\theta(x | z) is defined as the generating distribution, but also as a joint distribution of z_1 and z_2. Unless I am missing something I think you are overloading the notation for p_\theta.
- I don't think enough information is given about the AAE and ARAE baselines. Are they the same as the flat-VAE, except with the KL term replaced by the an adversarial divergence between the prior and approximate posterior?