title: Top-nσ: Not All Logits Are You Need
summary: Large language models (LLMs) typically employ greedy decoding or
  low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
  between diversity and accuracy. We challenge this convention by introducing
  top-nsigma, a novel sampling method that operates directly on pre-softmax
  logits by leveraging a statistical threshold. Our key insight is that logits
  naturally separate into a Gaussian-distributed noisy region and a distinct
  informative region, enabling efficient token filtering without complex
  probability manipulations. Unlike existing methods (e.g., top-p, min-p)
  that inadvertently include more noise tokens at higher temperatures,
  top-nsigma maintains a stable sampling space regardless of temperature
  scaling. We also provide a theoretical analysis of top-nsigma to better
  understand its behavior. The extensive experimental results across four
  reasoning-focused datasets demonstrate that our method not only outperforms
  existing sampling approaches but also surpasses greedy decoding, while
  maintaining consistent performance even at high temperatures.
paper_authors._id: 67340d47a445f1d61926ce8e
paper_authors.name: Chenxia Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:25:26.830Z
paper_authors._id: 65d2fa0b6236ca85a4091763
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d2fa0b6236ca85a4091763/Cc6eS-Bj-dIb0mUv2QSSh.jpeg
paper_authors.fullname: Tang Chenxia
paper_authors.type: user
paper_authors.user: tomorrowdawn
paper_authors._id: 67340d47a445f1d61926ce8f
paper_authors.name: Jianchun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:09:15.971Z
paper_authors._id: 673d300db70db76ecadd8b37
paper_authors.avatarUrl: /avatars/55cec467612e1b6daddb16f9414693f2.svg
paper_authors.fullname: Jianchun Liu
paper_authors.type: user
paper_authors.user: Jsen617
paper_authors._id: 67340d47a445f1d61926ce90
paper_authors.name: Hongli Xu
paper_authors._id: 67340d47a445f1d61926ce91
paper_authors.name: Liusheng Huang
paper_title: Top-nσ: Not All Logits Are You Need
paper_summary: Large language models (LLMs) typically employ greedy decoding or
  low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
  between diversity and accuracy. We challenge this convention by introducing
  top-nsigma, a novel sampling method that operates directly on pre-softmax
  logits by leveraging a statistical threshold. Our key insight is that logits
  naturally separate into a Gaussian-distributed noisy region and a distinct
  informative region, enabling efficient token filtering without complex
  probability manipulations. Unlike existing methods (e.g., top-p, min-p)
  that inadvertently include more noise tokens at higher temperatures,
  top-nsigma maintains a stable sampling space regardless of temperature
  scaling. We also provide a theoretical analysis of top-nsigma to better
  understand its behavior. The extensive experimental results across four
  reasoning-focused datasets demonstrate that our method not only outperforms
  existing sampling approaches but also surpasses greedy decoding, while
  maintaining consistent performance even at high temperatures.
paper_ai_summary: A novel sampling method called top-$n\sigma$ for large language models improves reasoning task performance by filtering pre-softmax logits and maintaining consistent results across different temperatures.
paper_ai_keywords: greedy decoding low-temperature sampling top-$n\sigma$ pre-softmax logits Gaussian-distributed noisy region informative region token filtering top-$p$ min-$p$ sampling approaches