Scaling Reasoning can Improve Factuality in Large Language Models
Recent studies on large language model (LLM) reasoning capabilities have
demonstrated promising improvements in model performance by leveraging a
lengthy thinking process and additional computational resources during
inference, primarily in tasks involving mathematical reasoning (Muennighoff et
al., 2025). However, it remains uncertain if longer reasoning chains inherently
enhance factual accuracy, particularly beyond mathematical contexts. In this
work, we thoroughly examine LLM reasoning within complex open-domain
question-answering (QA) scenarios. We initially distill reasoning traces from
advanced, large-scale reasoning models (QwQ-32B and DeepSeek-R1-671B), then
fine-tune a variety of models ranging from smaller, instruction-tuned variants
to larger architectures based on Qwen2.5. To enrich reasoning traces, we
introduce factual information from knowledge graphs in the form of paths into
our reasoning traces. Our experimental setup includes four baseline approaches
and six different instruction-tuned models evaluated across a benchmark of six
datasets, encompassing over 22.6K questions. Overall, we carry out 168
experimental runs and analyze approximately 1.7 million reasoning traces. Our
findings indicate that, within a single run, smaller reasoning models achieve
noticeable improvements in factual accuracy compared to their original
instruction-tuned counterparts. Moreover, our analysis demonstrates that adding
test-time compute and token budgets factual accuracy consistently improves by
2-8%, further confirming the effectiveness of test-time scaling for enhancing
performance and consequently improving reasoning accuracy in open-domain QA
tasks. We release all the experimental artifacts for further research.
682ad417500638b80a43471d
Mike Zhang
claimed_verified
2025-05-19T07:23:04.053Z
60d33fbbd7b174177faabd4f
https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
Mike Zhang
user
jjzha
682ad417500638b80a43471e
Johannes Bjerva
admin_assigned
2025-05-19T07:25:58.827Z
678fa79005ae7fe48d03ba47
/avatars/a78ab2b37fa3e18ace783f6f71f5a361.svg
Johannes Bjerva
user
bjerva
682ad417500638b80a43471f
Russa Biswas
admin_assigned
2025-05-19T07:26:04.749Z
60ed4c56abab3c2620df8ac8
/avatars/ad5508c1c94a96f6d1290e4735e81b73.svg
Russa Biswas
user
rubis
Scaling Reasoning can Improve Factuality in Large Language Models
Recent studies on large language model (LLM) reasoning capabilities have
demonstrated promising improvements in model performance by leveraging a
lengthy thinking process and additional computational resources during
inference, primarily in tasks involving mathematical reasoning (Muennighoff et
al., 2025). However, it remains uncertain if longer reasoning chains inherently
enhance factual accuracy, particularly beyond mathematical contexts. In this
work, we thoroughly examine LLM reasoning within complex open-domain
question-answering (QA) scenarios. We initially distill reasoning traces from
advanced, large-scale reasoning models (QwQ-32B and DeepSeek-R1-671B), then
fine-tune a variety of models ranging from smaller, instruction-tuned variants
to larger architectures based on Qwen2.5. To enrich reasoning traces, we
introduce factual information from knowledge graphs in the form of paths into
our reasoning traces. Our experimental setup includes four baseline approaches
and six different instruction-tuned models evaluated across a benchmark of six
datasets, encompassing over 22.6K questions. Overall, we carry out 168
experimental runs and analyze approximately 1.7 million reasoning traces. Our
findings indicate that, within a single run, smaller reasoning models achieve
noticeable improvements in factual accuracy compared to their original
instruction-tuned counterparts. Moreover, our analysis demonstrates that adding
test-time compute and token budgets factual accuracy consistently improves by
2-8%, further confirming the effectiveness of test-time scaling for enhancing
performance and consequently improving reasoning accuracy in open-domain QA
tasks. We release all the experimental artifacts for further research.
Enhancing factual accuracy in open-domain QA through reasoning trace distillation, knowledge graph integration, and test-time scaling improves smaller models' performance across various datasets.
large language model (LLM) reasoning capabilities length thinking process computational resources mathematical reasoning complex open-domain question-answering (QA) scenarios reasoning traces QwQ-32B DeepSeek-R1-671B instruction-tuned Qwen2.5 knowledge graphs factual accuracy test-time compute token budgets reasoning accuracy