Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible
  Reasoning
Large Language Models (LLMs) are now integral across various domains and have
demonstrated impressive performance. Progress, however, rests on the premise
that benchmark scores are both accurate and reproducible. We demonstrate that
the reproducibility of LLM performance is fragile: changing system
configuration such as evaluation batch size, GPU count, and GPU version can
introduce significant difference in the generated responses. This issue is
especially pronounced in reasoning models, where minor rounding differences in
early tokens can cascade into divergent chains of thought, ultimately affecting
accuracy. For instance, under bfloat16 precision with greedy decoding, a
reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation
in accuracy and 9,000 tokens difference in response length due to differences
in GPU count, type, and evaluation batch size. We trace the root cause of this
variability to the non-associative nature of floating-point arithmetic under
limited numerical precision. This work presents the first systematic
investigation into how numerical precision affects reproducibility in LLM
inference. Through carefully controlled experiments across various hardware,
software, and precision settings, we quantify when and how model outputs
diverge. Our analysis reveals that floating-point precision -- while critical
for reproducibility -- is often neglected in evaluation practices. Inspired by
this, we develop a lightweight inference pipeline, dubbed LayerCast, that
stores weights in 16-bit precision but performs all computations in FP32,
balancing memory efficiency with numerical stability. Code is available at
https://github.com/nanomaoli/llm_reproducibility.
684a6e2f9b38e1e5a33a68ae
Jiayi Yuan
claimed_verified
2025-06-13T07:41:47.215Z
64b7879a6ab5d14ca7f9bdf4
/avatars/48990fe5b18ab17a3dedacdcc6ee3f3a.svg
Jiayi Yuan
user
jy-yuan
684a6e2f9b38e1e5a33a68af
Hao Li
claimed_verified
2025-06-15T07:04:00.218Z
66a052e610de1ccd3d21f0e4
/avatars/92989d04e488caf61dd5cab6d034faf7.svg
Hao Li
user
cane-sugar-soda
684a6e2f9b38e1e5a33a68b0
Xinheng Ding
684a6e2f9b38e1e5a33a68b1
Wenya Xie
684a6e2f9b38e1e5a33a68b2
Yu-Jhe Li
684a6e2f9b38e1e5a33a68b3
Wentian Zhao
684a6e2f9b38e1e5a33a68b4
Kun Wan
684a6e2f9b38e1e5a33a68b5
Jing Shi
684a6e2f9b38e1e5a33a68b6
Xia Hu
684a6e2f9b38e1e5a33a68b7
Zirui Liu
Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible
  Reasoning
Large Language Models (LLMs) are now integral across various domains and have
demonstrated impressive performance. Progress, however, rests on the premise
that benchmark scores are both accurate and reproducible. We demonstrate that
the reproducibility of LLM performance is fragile: changing system
configuration such as evaluation batch size, GPU count, and GPU version can
introduce significant difference in the generated responses. This issue is
especially pronounced in reasoning models, where minor rounding differences in
early tokens can cascade into divergent chains of thought, ultimately affecting
accuracy. For instance, under bfloat16 precision with greedy decoding, a
reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation
in accuracy and 9,000 tokens difference in response length due to differences
in GPU count, type, and evaluation batch size. We trace the root cause of this
variability to the non-associative nature of floating-point arithmetic under
limited numerical precision. This work presents the first systematic
investigation into how numerical precision affects reproducibility in LLM
inference. Through carefully controlled experiments across various hardware,
software, and precision settings, we quantify when and how model outputs
diverge. Our analysis reveals that floating-point precision -- while critical
for reproducibility -- is often neglected in evaluation practices. Inspired by
this, we develop a lightweight inference pipeline, dubbed LayerCast, that
stores weights in 16-bit precision but performs all computations in FP32,
balancing memory efficiency with numerical stability. Code is available at
https://github.com/nanomaoli/llm_reproducibility.
The study investigates reproducibility issues in Large Language Models (LLMs) arising from hardware and precision variations, proposing a lightweight inference pipeline to enhance numerical stability while maintaining memory efficiency.
Large Language Models LLMs reasoning models DeepSeek-R1-Distill-Qwen-7B floating-point arithmetic floating-point precision greedy decoding LayerCast FP32