Towards Text Generation with Adversarially Learned Neural Outlines
Recent progress in deep generative models has been fueled by two paradigms -autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualitative results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.
Update after author response: It's good to see the efforts of conducting human evaluation, and I encourage the authors to include more details, e.g., how the annotators are selected, what questions are asked, into the next revision. I'm changing the score to 5 now.  ============== This manuscript proposes a generative adversarial approach for text generation. Specifically, sentences are encoded into vectors in a "latent space" by a RNN encoder; a decoder conditions on the vector and generates text auto-regressively. The latent representations are adversarially regularized towards a fixed prior. The method is evaluated on both unconditional and conditional generation settings, and the paper argues for better performance than baselines.  The method is not that surprising, given Hu et al., (2017) (missing in the references), and Kim et al., (2017), among others. Many claims are made, including both theoretical justifications and better empirical performance. However, I do not find either of them is well-supported. I can hardly find any insightful theoretical discussion of the proposed method, in comparison to previous works. It is probably still okay, should the submission put more thoughts into the empirical part by, e.g., showing better performance in carefully designed human evaluation. Unfortunately, this is not the case.  That being said, the third point of the claimed contribution --- the gradient-based interpolation between sentence embeddings appears intriguing to me. I would encourage the authors to discuss how it relates to existing works in more detail, and put it earlier in the narrative.  In sum, this submission would have been much stronger should it come out, e.g., one year earlier. I cannot recommend that it is accepted to NIPS in its current status, but would be looking forward to a revised version.  References  Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward Controlled Generation of Text. In Proc. of ICML.