Adversarial Text Generation via Feature-Mover's Distance
Generative adversarial networks (GANs) have achieved significant success in generating real-valued data. However, the discrete nature of text hinders the application of GAN to text-generation tasks. Instead of using the standard GAN objective, we propose to improve text-generation GAN via a novel approach inspired by optimal transport. Specifically, we consider matching the latent feature distributions of real and synthetic sentences using a novel metric, termed the featuremover's distance (FMD). This formulation leads to a highly discriminative critic and easy-to-optimize objective, overcoming the mode-collapsing and brittle-training problems in existing methods. Extensive experiments are conducted on a variety of tasks to evaluate the proposed model empirically, including unconditional text generation, style transfer from non-parallel text, and unsupervised cipher cracking. The proposed model yields superior performance, demonstrating wide applicability and effectiveness.
The paper proposes a "feature-mover's distance" as the optimization objective of GAN applied to text. Experiments are conducted for text generation, text style transfer and text decipher tasks.  The proposed idea is a variation on the choice of loss functions compared to previous GAN models for text. However, both the motivation and the experimental results are not significant enough to show an advantage of the proposed loss form.  First of all, the paper mentions that "mode collapsing" and "brittle training" are 2 main problems facing GAN for text. These problems are true, but the paper shows no justification for why the proposed loss form could be helpful for mitigating these 2 problems, neither theoretically nor empirically.  Secondly, since the main contribution is the loss form, the paper should have included fair comparisons with some other major loss forms using the same generator and discriminator network design. This is missing in the paper, and it is very hard to grasp why the particular choice of loss form is made. Besides, a recent paper [1] shows that loss choices of GAN do not matter too much for the generative results (albeit for image generation), which makes is even more necessary to include such comparisons in this paper.  Thirdly, the evaluation results are not statistically significant. The first problem is the choice of evaluation metric (BLEU and self-BLEU), which are not well established scores for text generative models. It is questionable whether any form of BLEU is useful in this case, since it measures whether a phrase exists in a target sentence. Text generative models could simply generate phrases that could make sense, but beyond the measurement of such statistical matching scores because they do not exist as a target sample. One would argue that a large enough dataset would have solved this, but the datasets used in the paper are quite small compared to other available text corpora. Also, the self-BLEU score requires multiple generated texts, which is a hyper-parameter not clearly detailed in the paper.  Furthermore, it is unclear whether the improvement obtained in the paper under such metric is significant enough to demonstrate an advantage of the proposed loss form.  Finally, it is hard for me to perceive a difference from the generated samples in the supplemental material. The authors should perhaps consider including an explanation on why the proposed form generate better results from these samples.  In a summary, due to the lack of proper motivation, comparison and significance in results, I recommend rejection for the paper in its current form. That said, the loss form is an interesting idea. If the authors can develop the idea further and show a clear advantage, it could perhaps be submitted again.  [1] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet. Are GANs Created Equal? A Large-Scale Study. arXiv 1711.10337  ------  After reading the author rebuttal, I realized I missed figure 2 in the first round of reviewing. As a result, the main reason for recommending rejection -- lack of comparison with other GAN losses -- has been significantly weakened. I changed the score to recommend for acceptance now.