Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs (Holtzman et al., 2019). While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.
This paper proposes training losses, unlikelihood objective, for mitigating the repetition problem of the text generated by recent neural language models. The problem is well-motivated by evidence from the existing literature. Specifically, the paper argues that the main cause of the degenerated output is the maximum likelihood objective commonly used to train language models. Their main contribution is to introduce additional objectives to penalize “unlikely” word probabilities. The proposed penalty is derived into 2 objectives: token level (previous words in context) and sentence level (future decoded words). The prior objective is used along with the MLE, while the later and more expensive is used for fine-tuning. They perform experiments on Wikitext-103 and evaluate models on the perplexity of the models, and n-gram statistics such as repetition, and uniqueness of the decoded texts. The proposed training scheme (UL-token+seq) is shown to have the closest statistics to the original corpus while the perplexity slightly suffers. The additional manual analysis shows that human annotators prefer the outputs (sentence completion) of the proposed method over the other baselines.
Overall, this paper tackles a relevant problem and could propose a novel method.
For the unlikelihood objectives, there are a few clarifications on the design decision. There are some “correct” repetitions in the ground-truth text as well. However, the proposed loss minimizes all repetitions regardless. In addition, it is unclear how the proposed method mitigates the token distribution mismatch. Finally, there is a similar work where they attempt to “match” repetition (or n-gram distribution) of a reference corpus (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16961). A discussion of how this paper distinguishes from previous work would be helpful.
For the experiments, there are some missing detail and concerns:
1. The fine-tuning using UL-seq (eq 7) procedure is not well explained. For example, how many sequences are used per update? How many times that you decode in the 1,500 updates?
2. the stochastic decoding results are related as the paper motivation is built on top of Holtzman et al., 2019, the results also support the claim. However, how do you explain the discrepancy between the experts and the crowd workers?
3. GPT-2 results show that UL-seq does not significantly improve several evaluation metrics from MLE. This is a conflict with the result in Table 2. This could be a sign of the generalization problem of the proposed method. Additional results on different model architectures would be helpful.
Minor question:
1. It is uncertain how these repetition and uniqueness statistics translate to a downstream task (e.g. summarization or NMT). Do you have results regarding this?
2. It appears that UL-token does not do much. What is the increase in training time? Do you recommend using the token loss?