title: The Hallucination Tax of Reinforcement Finetuning
summary: Reinforcement finetuning (RFT) has become a standard approach for enhancing
  the reasoning capabilities of large language models (LLMs). However, its impact
  on model trustworthiness remains underexplored. In this work, we identify and
  systematically study a critical side effect of RFT, which we term the
  hallucination tax: a degradation in refusal behavior causing models to produce
  hallucinated answers to unanswerable questions confidently. To investigate
  this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of
  unanswerable math problems designed to probe models' ability to recognize an
  unanswerable question by reasoning from the insufficient or ambiguous
  information. Our results show that standard RFT training could reduce model
  refusal rates by more than 80%, which significantly increases model's tendency
  to hallucinate. We further demonstrate that incorporating just 10% SUM during
  RFT substantially restores appropriate refusal behavior, with minimal accuracy
  trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage
  inference-time compute to reason about their own uncertainty and knowledge
  boundaries, improving generalization not only to out-of-domain math problems
  but also to factual question answering tasks.
paper_authors._id: 682d355b89b5f79a1d6546f9
paper_authors.name: Linxin Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T13:34:02.506Z
paper_authors._id: 64d660308ebc40443813f014
paper_authors.avatarUrl: /avatars/516bb2d2383be99794e366dfb41636b6.svg
paper_authors.fullname: Linxin Song
paper_authors.type: user
paper_authors.user: linxinso
paper_authors._id: 682d355b89b5f79a1d6546fa
paper_authors.name: Taiwei Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:17:53.968Z
paper_authors._id: 62e1b3cb3eb0730f621a83f6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
paper_authors.fullname: Taiwei Shi
paper_authors.type: user
paper_authors.user: MaksimSTW
paper_authors._id: 682d355b89b5f79a1d6546fb
paper_authors.name: Jieyu Zhao
paper_title: The Hallucination Tax of Reinforcement Finetuning
paper_summary: Reinforcement finetuning (RFT) has become a standard approach for enhancing
  the reasoning capabilities of large language models (LLMs). However, its impact
  on model trustworthiness remains underexplored. In this work, we identify and
  systematically study a critical side effect of RFT, which we term the
  hallucination tax: a degradation in refusal behavior causing models to produce
  hallucinated answers to unanswerable questions confidently. To investigate
  this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of
  unanswerable math problems designed to probe models' ability to recognize an
  unanswerable question by reasoning from the insufficient or ambiguous
  information. Our results show that standard RFT training could reduce model
  refusal rates by more than 80%, which significantly increases model's tendency
  to hallucinate. We further demonstrate that incorporating just 10% SUM during
  RFT substantially restores appropriate refusal behavior, with minimal accuracy
  trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage
  inference-time compute to reason about their own uncertainty and knowledge
  boundaries, improving generalization not only to out-of-domain math problems
  but also to factual question answering tasks.
paper_ai_summary: Reinforcement fine-tuning can degrade model refusal behavior, leading to increased hallucination; incorporating a synthetic dataset of unanswerable math problems during fine-tuning can restore appropriate refusal behavior with minimal accuracy loss and improve generalization.
paper_ai_keywords: Reinforcement finetuning (RFT) large language models (LLMs) hallucination tax refusal behavior SUM (Synthetic Unanswerable Math) unanswerable math problems solvable tasks inference-time compute knowledge boundaries generalization factual question answering tasks