Top-nσ: Not All Logits Are You Need
Large language models (LLMs) typically employ greedy decoding or
low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
between diversity and accuracy. We challenge this convention by introducing
top-nsigma, a novel sampling method that operates directly on pre-softmax
logits by leveraging a statistical threshold. Our key insight is that logits
naturally separate into a Gaussian-distributed noisy region and a distinct
informative region, enabling efficient token filtering without complex
probability manipulations. Unlike existing methods (e.g., top-p, min-p)
that inadvertently include more noise tokens at higher temperatures,
top-nsigma maintains a stable sampling space regardless of temperature
scaling. We also provide a theoretical analysis of top-nsigma to better
understand its behavior. The extensive experimental results across four
reasoning-focused datasets demonstrate that our method not only outperforms
existing sampling approaches but also surpasses greedy decoding, while
maintaining consistent performance even at high temperatures.
67340d47a445f1d61926ce8e
Chenxia Tang
claimed_verified
2024-11-15T09:25:26.830Z
65d2fa0b6236ca85a4091763
https://cdn-avatars.huggingface.co/v1/production/uploads/65d2fa0b6236ca85a4091763/Cc6eS-Bj-dIb0mUv2QSSh.jpeg
Tang Chenxia
user
tomorrowdawn
67340d47a445f1d61926ce8f
Jianchun Liu
claimed_verified
2024-11-20T10:09:15.971Z
673d300db70db76ecadd8b37
/avatars/55cec467612e1b6daddb16f9414693f2.svg
Jianchun Liu
user
Jsen617
67340d47a445f1d61926ce90
Hongli Xu
67340d47a445f1d61926ce91
Liusheng Huang
Top-nσ: Not All Logits Are You Need
Large language models (LLMs) typically employ greedy decoding or
low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
between diversity and accuracy. We challenge this convention by introducing
top-nsigma, a novel sampling method that operates directly on pre-softmax
logits by leveraging a statistical threshold. Our key insight is that logits
naturally separate into a Gaussian-distributed noisy region and a distinct
informative region, enabling efficient token filtering without complex
probability manipulations. Unlike existing methods (e.g., top-p, min-p)
that inadvertently include more noise tokens at higher temperatures,
top-nsigma maintains a stable sampling space regardless of temperature
scaling. We also provide a theoretical analysis of top-nsigma to better
understand its behavior. The extensive experimental results across four
reasoning-focused datasets demonstrate that our method not only outperforms
existing sampling approaches but also surpasses greedy decoding, while
maintaining consistent performance even at high temperatures.
A novel sampling method called top-$n\sigma$ for large language models improves reasoning task performance by filtering pre-softmax logits and maintaining consistent results across different temperatures.
greedy decoding low-temperature sampling top-$n\sigma$ pre-softmax logits Gaussian-distributed noisy region informative region token filtering top-$p$ min-$p$ sampling approaches