Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that none of the considered models performs convincingly better than the LM.
This paper tackles the problem of evaluation of language generation models and particularly focuses on the comparison between GAN-based language models (GAN-LM) vs likelihood-based language models (MLE-LM). Studying the behaviour of current evaluation metrics for language generation as well as finding new ones is an important research topic. I believe that this paper makes a step in the right direction but the magnitude of that step may be insufficient for publication. I appreciate the efforts but I find most of the findings about BLEU not being a good metric and characteristic of reverse PPL rather unsurprising. The majority of the paper is dedicated to describing models / metrics which are well-known instead of performing more solid experimental evaluation (Results start at page 6). Instead, the authors could have focused more on the study of FD for language generation.
-- Details
-"Our main finding is that, when compared carefully, a conventional neural Language Model performs at least as well as any of the tested GAN models", however the authors don't compare with the recent MaskGAN model, which, according to (https://arxiv.org/abs/1801.07736) outperforms MLE variants.
- "We demonstrate that previously used n-gram matching, such as BLEU scores, is an insufficient metric": the fact that BLEU is not ideal for evaluation natural language generation has been pointed out in multiple related papers (e.g. https://arxiv.org/abs/1603.08023) and thus is not surprising.
- "We find that reporting results from the best single run or not performing sufficient tuning introduces significant bias in the reported results": as the authors point out in related works, the variance in GAN results which hinders meaningfulness of the reported results is a also a well-known problem (e.g. https://arxiv.org/pdf/1711.10337.pdf), therefore cannot be considered as a contribution.
- The observed behaviour (sensitivity to mode collapse, word swap, word removal) of the "reverse PPL" metric is pretty much expected, but I agree some experimental results are still interesting.
- On the contrary, I liked the study on the FD metric but I would have loved the paper to focus more on the study of the behaviour of that metric: for example, by examining the robustness under different base models, while the authors only test with the model by Conneau et. al, 2017.
- It would have been good to train a state-of-the-art language model architecture, e.g. AWD-LSTM, and to control regularization. I cannot see if the MLE-LM model is overfitting or not.
-- Style remarks:
- Moving the figures closer to the paragraph where they are described avoids the reader the burden of going back and forth through the paper.