title: The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore
    Non-Determinism
summary: Current evaluations of large language models (LLMs) often overlook
  non-determinism, typically focusing on a single output per example. This limits
  our understanding of LLM performance variability in real-world applications.
  Our study addresses this issue by exploring key questions about the performance
  differences between greedy decoding and sampling, identifying benchmarks'
  consistency regarding non-determinism, and examining unique model behaviors.
  Through extensive experiments, we observe that greedy decoding generally
  outperforms sampling methods for most evaluated tasks. We also observe
  consistent performance across different LLM sizes and alignment methods, noting
  that alignment can reduce sampling variance. Moreover, our best-of-N sampling
  approach demonstrates that smaller LLMs can match or surpass larger models such
  as GPT-4-Turbo, highlighting the untapped potential of smaller LLMs. This
  research shows the importance of considering non-determinism in LLM evaluations
  and provides insights for future LLM development and evaluation.
paper_authors._id: 6695cf864c450902c8cf3e5a
paper_authors.name: Yifan Song
paper_authors._id: 6695cf864c450902c8cf3e5b
paper_authors.name: Guoyin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:17:06.403Z
paper_authors._id: 6490d4ba1afdee3acd1147f6
paper_authors.avatarUrl: /avatars/ae13c7b21fe9ced7541dcd664d1b94ed.svg
paper_authors.fullname: Guoyin Wang
paper_authors.type: user
paper_authors.user: guoyinwang
paper_authors._id: 6695cf864c450902c8cf3e5c
paper_authors.name: Sujian Li
paper_authors._id: 6695cf864c450902c8cf3e5d
paper_authors.name: Bill Yuchen Lin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-16T01:40:23.582Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_title: The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore
    Non-Determinism
paper_summary: Current evaluations of large language models (LLMs) often overlook
  non-determinism, typically focusing on a single output per example. This limits
  our understanding of LLM performance variability in real-world applications.
  Our study addresses this issue by exploring key questions about the performance
  differences between greedy decoding and sampling, identifying benchmarks'
  consistency regarding non-determinism, and examining unique model behaviors.
  Through extensive experiments, we observe that greedy decoding generally
  outperforms sampling methods for most evaluated tasks. We also observe
  consistent performance across different LLM sizes and alignment methods, noting
  that alignment can reduce sampling variance. Moreover, our best-of-N sampling
  approach demonstrates that smaller LLMs can match or surpass larger models such
  as GPT-4-Turbo, highlighting the untapped potential of smaller LLMs. This
  research shows the importance of considering non-determinism in LLM evaluations
  and provides insights for future LLM development and evaluation.
paper_ai_keywords: greedy decoding sampling methods benchmarks non-determinism model behaviors best-of-N sampling