The Curious Case of Neural Text Degeneration
Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.
This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties. Pros: The problem addressed in this paper is highly interesting, and the proposed method is simple and intuitive. The paper is well motivated and the method is clearly presented. Extensive quantitative and qualitative experiments are conducted to compare different sampling methods. Cons: 1) Although the raised problem in this paper is interesting, the proposed Nucleus Sampling seems to be a trivial variant of Top-k sampling. With a reasonably large k suitable for different practical problems in question, it is unclear that Nucleus Sampling produces significant advantages over commonly used Top-k sampling. 2) The argued difficulty in choosing k in Top-k sampling is not that different from that of choosing the threshold p in Nucleus Sampling. 3) In section 4.3, the argument that natural language rarely remains in a high-probability zone is questionable. This happens only because our current neural language models are not well-specified for generating long texts and modeling long-range contexts. 4) In section 6.2, the qualitative comparison between Nucleus Sampling and Top-k sampling might be caused by randomness. With a large k, there is no technical barrier that prevents Top-k sampling from generating the sentences produced by Nucleus Sampling. 5) A recent stochastic beam search method based on Gumbel-max-k (Kool, Hoof, and Welling, ICML 2019) should be discussed and compared. In summary, although the studied problem in this paper is highly interesting, the proposed Nucleus Sampling is not technically significant compared to Top-k sampling.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Curious Case of Neural Text Degeneration
Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties. Pros: The problem addressed in this paper is highly interesting, and the proposed method is simple and intuitive. The paper is well motivated and the method is clearly presented. Extensive quantitative and qualitative experiments are conducted to compare different sampling methods. Cons: 1) Although the raised problem in this paper is interesting, the proposed Nucleus Sampling seems to be a trivial variant of Top-k sampling. With a reasonably large k suitable for different practical problems in question, it is unclear that Nucleus Sampling produces significant advantages over commonly used Top-k sampling. 2) The argued difficulty in choosing k in Top-k sampling is not that different from that of choosing the threshold p in Nucleus Sampling. 3) In section 4.3, the argument that natural language rarely remains in a high-probability zone is questionable. This happens only because our current neural language models are not well-specified for generating long texts and modeling long-range contexts. 4) In section 6.2, the qualitative comparison between Nucleus Sampling and Top-k sampling might be caused by randomness. With a large k, there is no technical barrier that prevents Top-k sampling from generating the sentences produced by Nucleus Sampling. 5) A recent stochastic beam search method based on Gumbel-max-k (Kool, Hoof, and Welling, ICML 2019) should be discussed and compared. In summary, although the studied problem in this paper is highly interesting, the proposed Nucleus Sampling is not technically significant compared to Top-k sampling.
assistant
This paper is motivated by an observation that maximization-based decoding approaches such as beam search can lead to incoherent and repetitive sentences when open-ended long-form text generation based on neural language model such as GPT-2 is performed. To solve the problem, this paper proposes a sampling method called Nucleus Sampling. Similar to Top-k sampling, Nucleus Sampling truncates the probability distribution of the words in the vocabulary. Instead of re-normalizing the probabilities for the top-k words, Nucleus Sampling re-normalizes the original probabilities for the words with values above a pre-chosen threshold p. Some quantitative and qualitative results show that the proposed sampling method can generate long-form texts with some nice properties. Pros: The problem addressed in this paper is highly interesting, and the proposed method is simple and intuitive. The paper is well motivated and the method is clearly presented. Extensive quantitative and qualitative experiments are conducted to compare different sampling methods. Cons: 1) Although the raised problem in this paper is interesting, the proposed Nucleus Sampling seems to be a trivial variant of Top-k sampling. With a reasonably large k suitable for different practical problems in question, it is unclear that Nucleus Sampling produces significant advantages over commonly used Top-k sampling. 2) The argued difficulty in choosing k in Top-k sampling is not that different from that of choosing the threshold p in Nucleus Sampling. 3) In section 4.3, the argument that natural language rarely remains in a high-probability zone is questionable. This happens only because our current neural language models are not well-specified for generating long texts and modeling long-range contexts. 4) In section 6.2, the qualitative comparison between Nucleus Sampling and Top-k sampling might be caused by randomness. With a large k, there is no technical barrier that prevents Top-k sampling from generating the sentences produced by Nucleus Sampling. 5) A recent stochastic beam search method based on Gumbel-max-k (Kool, Hoof, and Welling, ICML 2019) should be discussed and compared. In summary, although the studied problem in this paper is highly interesting, the proposed Nucleus Sampling is not technically significant compared to Top-k sampling.