title: Interleaved Reasoning for Large Language Models via Reinforcement
    Learning
summary: Long chain-of-thought (CoT) significantly enhances large language models'
  (LLM) reasoning capabilities. However, the extensive reasoning traces lead to
  inefficiencies and an increased time-to-first-token (TTFT). We propose a novel
  training paradigm that uses reinforcement learning (RL) to guide reasoning LLMs
  to interleave thinking and answering for multi-hop questions. We observe that
  models inherently possess the ability to perform interleaved reasoning, which
  can be further enhanced through RL. We introduce a simple yet effective
  rule-based reward to incentivize correct intermediate steps, which guides the
  policy model toward correct reasoning paths by leveraging intermediate signals
  generated during interleaved reasoning. Extensive experiments conducted across
  five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++)
  demonstrate consistent improvements over traditional think-answer reasoning,
  without requiring external tools. Specifically, our approach reduces TTFT by
  over 80% on average and improves up to 19.3% in Pass@1 accuracy. Furthermore,
  our method, trained solely on question answering and logical reasoning
  datasets, exhibits strong generalization ability to complex reasoning datasets
  such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to
  reveal several valuable insights into conditional reward modeling.
paper_authors._id: 68353e3f9f4e0a0f0496d0c6
paper_authors.name: Roy Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:02:55.735Z
paper_authors._id: 6555a124a6554059711b58a2
paper_authors.avatarUrl: /avatars/222bb6b8f252d6c2bbd4cf35a54fc1c9.svg
paper_authors.fullname: Roy
paper_authors.type: user
paper_authors.user: RRoy233
paper_authors._id: 68353e3f9f4e0a0f0496d0c7
paper_authors.name: David Qiu
paper_authors._id: 68353e3f9f4e0a0f0496d0c8
paper_authors.name: Deepak Gopinath
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:03:07.418Z
paper_authors._id: 66f6df6329dad7fec7cf26f2
paper_authors.avatarUrl: /avatars/e576fbeca6c5770905408316bd9bc0f2.svg
paper_authors.fullname: Deepak Gopinath
paper_authors.type: user
paper_authors.user: dpacgopinath
paper_authors._id: 68353e3f9f4e0a0f0496d0c9
paper_authors.name: Dong Lin
paper_authors._id: 68353e3f9f4e0a0f0496d0ca
paper_authors.name: Yanchao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:03:16.312Z
paper_authors._id: 6488a8e65815252b02b948c8
paper_authors.avatarUrl: /avatars/bb1ff64841c75b8d331986709b0a0a93.svg
paper_authors.fullname: Yanchao Sun
paper_authors.type: user
paper_authors.user: yanchao-sun
paper_authors._id: 68353e3f9f4e0a0f0496d0cb
paper_authors.name: Chong Wang
paper_authors._id: 68353e3f9f4e0a0f0496d0cc
paper_authors.name: Saloni Potdar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:03:28.257Z
paper_authors._id: 66711f329c609c2484c76d80
paper_authors.avatarUrl: /avatars/717e0ce3e8693e99da6d61199ea712f3.svg
paper_authors.fullname: Saloni Potdar
paper_authors.type: user
paper_authors.user: spotdar
paper_authors._id: 68353e3f9f4e0a0f0496d0cd
paper_authors.name: Bhuwan Dhingra
paper_title: Interleaved Reasoning for Large Language Models via Reinforcement
    Learning
paper_summary: Long chain-of-thought (CoT) significantly enhances large language models'
  (LLM) reasoning capabilities. However, the extensive reasoning traces lead to
  inefficiencies and an increased time-to-first-token (TTFT). We propose a novel
  training paradigm that uses reinforcement learning (RL) to guide reasoning LLMs
  to interleave thinking and answering for multi-hop questions. We observe that
  models inherently possess the ability to perform interleaved reasoning, which
  can be further enhanced through RL. We introduce a simple yet effective
  rule-based reward to incentivize correct intermediate steps, which guides the
  policy model toward correct reasoning paths by leveraging intermediate signals
  generated during interleaved reasoning. Extensive experiments conducted across
  five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++)
  demonstrate consistent improvements over traditional think-answer reasoning,
  without requiring external tools. Specifically, our approach reduces TTFT by
  over 80% on average and improves up to 19.3% in Pass@1 accuracy. Furthermore,
  our method, trained solely on question answering and logical reasoning
  datasets, exhibits strong generalization ability to complex reasoning datasets
  such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to
  reveal several valuable insights into conditional reward modeling.
paper_ai_summary: A reinforcement learning-guided training paradigm enhances large language models' reasoning efficiency and performance for multi-hop questions by interleaving thinking and answering.
paper_ai_keywords: chain-of-thought large language models reasoning capabilities time-to-first-token reinforcement learning interleaved reasoning rule-based reward reward modeling multi-hop questions think-answer reasoning Pass@1 accuracy MATH GPQA MMLU PPO GRPO REINFORCE++