Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story). The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops. To address this we propose Nucleus Sampling, a simple but effective method to draw considerably higher quality text out of neural language models than previous decoding strategies. Our approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass. To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition. Our results show that (1) maximization is an inappropriate decoding objective for openended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best available decoding strategy for generating longform text that is both high-quality -as measured by human evaluation -and as diverse as human-written text.
In the domain of language models, the paper introduces a new heuristic sampling method called top-p sampling, or nucleus sampling (NS). It is a variant of top-k sampling where the smallest k is selected to ensure the combined likelihood is no less than p. The paper centers on claiming and showing that the generated samples are of higher quality and more diverse than common alternatives such as beam search, pure sampling, top-k sampling, and low-temperature sampling.
While overall I think the proposed method is sound as an alternative to other heuristics such as beam search, I have reservations on the presentation and arguments made in the paper. 
Pros:
1. NS is sound as a heuristic sampling method.
2. The paper contains many interesting experimental observations and I speculate that some of them will find future uses. For example, the selection of parameter values (not just for NS, also for top-k) and the nontrivial perplexity of generated text.
Cons:
1. The ultimate performance measure (open-ended generation) of “high quality” and “diversity” is very vague. It seems that the authors end up doing is to evaluate by high self-BLEU, HUSE, few repetitions, and perplexity. Furthermore, it is unclear why one _should_ train with cross-entropy (trying to match the distributions) and then rely on the sampling procedure to fulfill these desiderata (See also Min1 and Min2).
2. The comparison with beam search (BS) is not well motivated. BS is devised to find the maximal sentence and it is not stochastic. It seems out of place in the context of generating a “diverse” set of samples.
3. The arguments in the comparison with pure sampling is vague and sometimes misplaced. The key argument seems to hinge on the idea that the low likelihood tail is of “low confidence.” But this claim is problematic. If the estimate is wrong on the low probability tail, then so is the estimate on p(head) = 1-p(tail) by virtue of p being a probability measure. 
4. The arguments in the comparison with top-k is vague and sometimes misplaced. The main argument against top-k is the “[d]ifficulty in choosing a suitable value of k” but the same can be said for choosing p. After all, top-k and top-p (NS) can be thought of as a variant of each other (by dynamically choosing k or p respectively). Moreover, in Figure 5, a selection for k value is suggested. I agree that this value might not _appear_ as intuitive as p, and maybe other works have chosen a smaller k than they should have, but similarly, people might intuitively choose too high a value for p (Figure 5). 
Possible mistakes/typos:
1. (2), “>=“ -> ≥.
2. Figure 7, the human self-BLEU4 < human self-BLEU5 and that seems wrong, especially when all other bars show the opposite ordering.
3. In References, “Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACL, 2018a” is duplicated.
4. In References, the citation of “Unifying human and statistical evaluation for natural language generation” is from NAACL 2019, not “2018.”
5. In References, the first names are shown as initials in “Sparse forward-backward using minimum divergence beams for fast training of conditional random fields.”
Questions:
1. In Table 1, how is Human perplexity estimated?
Minor issues:
1. Partly due to what the authors position NS to solve, i.e. open-ended generation, the core arguments is not as precise or rigorous as it could have been in my opinion. I feel that focusing on comparing NS to other heuristics as a heuristic might make the text appeal to a wider audience and the discussion more precise.  
2. The distinction drawn between open-ended generation and directed generation is unpersuasive to me. In the context of language modeling, the former is to approximate a distribution (over an extended alphabet) whereas the latter is to approximate a conditional distribution (given the input). However, the most common formulation to solve the former is to decompose the distribution into a product of conditional distributions (1).
3. The caption in Figure 1 draws a misleading comparison. The “admirable” generation (presumably referring to the OpenAI blog post) was from the full GPT-2 model, not the initially released GPT-2-117M.
Please point out my misunderstanding directly. I am open to acknowledging them and revising my assessment.