The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way -a capability that has never been attempted by previous methods.
This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.
Some questions:-
- I had some trouble following section 4.2 where the derivation for L-reg is explained, this is the loss that constrains the posterior to a simplex. Is E = [e_1, ..., e_k] learnt or a hyperparameter? Most of the text seems to indicate that it is learnt, but you mention that alpha is a tunable hyperparameter, where alpha = e_i^2. Could you elaborate on how e_i^2 can be fixed to alpha is E is learnt?
- You state that mu^2 reaches a minimum at alpha/K. Could you please provide a proof? This could be placed in an Appendix.
- In section 6.3, is K still 3 or is it now >=4?
Some citations issues 
- You're missing a couple key citations. I think your work should cite Hu et al.'s (2018) Toward Controlled Generation of Text. 
- You should definitely cite Lample et al.'s (2019)  Multiple-Attribute Text Rewriting. The results in Lample et al. are definitely comparable, and often better than the results presented in this paper. While I think this paper makes an independent and useful contribution, some of the claims are overblown since the results from Lample et al. are not presented or discussed. For example, the Lample's model on Yelp achieves a higher accuracy and a lower perplexity than the CP-VAE model presented in this paper.
- Correction of citations: Kim et al. (2018), Adversarially regularized autoencoders, is in fact Zhao et al. (2018), you're just missing the first author.
Additionally, I'd like to see some more detail about the robustness of the method. Is the model robust to initialization and hyperparameter settings? Is posterior collapse always avoided? In Table 1 you show that without L-Reg the CP-VAE suffer with posterior collapse, but this would carry more weight if you showed some statistics on how often using L-Reg helps avoid posterior collapse. In general it looks like we're mostly presented with the best results, I'd like to know a little more about the mean and standard deviation of the model runs as well.
Overall, I think this paper makes a worthwhile contribution. I was unclear on a few parts of the derivation for the regularizing losses, but the intuition seems sensible. The presented results are impressive but have a few missing pieces: citing some key results (Lample et al. 2019) and including results about robustness. I think the questions I have, and any shortcomings this paper has, could be addressed in a camera-ready version.
Other bits,
- I do not understand figure 2. It's also never referred back to once we learn what v_p and v_n are. Some clarification here would be helpful.
- Section 5.1, line 3: achieve -> achieved