Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation
<|reference_start|>Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation: Sampling-based decoding strategies have been widely adopted for Large Language Models (LLMs) in numerous applications, which target a balance between diversity and quality via temperature tuning and tail truncation (e.g., top-k and top-p sampling). Considering the high dynamic range of the candidate next-token given different prefixes, recent studies propose to adaptively truncate the tail of LLM's predicted distribution. Although improved results haven been reported with these methods on open-ended text generation tasks, the results are highly dependent on the curated truncation parameters and exemplar text. In this paper, we propose a systematic way to estimate the intrinsic capacity of a truncation sampling method by considering the trade-off between diversity and risk at each decoding step, based on our collected prefix tree which preserves the context of a full sentence. Our work provides a comprehensive comparison between existing truncation sampling methods, as well as their recommended parameters as a guideline for users.<|reference_end|>
@article{zhou2024balancing,
    title={Balancing Diversity and Risk in LLM Sampling: How to Select Your Method
  and Parameter for Open-Ended Text Generation},
    author={Yuxuan Zhou, Margret Keuper, Mario Fritz},
    journal={arXiv preprint arXiv:2408.13586},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.13586},
    primaryClass={cs.CL cs.AI}
}