Interleaved Reasoning for Large Language Models via Reinforcement
  Learning
Long chain-of-thought (CoT) significantly enhances large language models'
(LLM) reasoning capabilities. However, the extensive reasoning traces lead to
inefficiencies and an increased time-to-first-token (TTFT). We propose a novel
training paradigm that uses reinforcement learning (RL) to guide reasoning LLMs
to interleave thinking and answering for multi-hop questions. We observe that
models inherently possess the ability to perform interleaved reasoning, which
can be further enhanced through RL. We introduce a simple yet effective
rule-based reward to incentivize correct intermediate steps, which guides the
policy model toward correct reasoning paths by leveraging intermediate signals
generated during interleaved reasoning. Extensive experiments conducted across
five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++)
demonstrate consistent improvements over traditional think-answer reasoning,
without requiring external tools. Specifically, our approach reduces TTFT by
over 80% on average and improves up to 19.3% in Pass@1 accuracy. Furthermore,
our method, trained solely on question answering and logical reasoning
datasets, exhibits strong generalization ability to complex reasoning datasets
such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to
reveal several valuable insights into conditional reward modeling.
68353e3f9f4e0a0f0496d0c6
Roy Xie
admin_assigned
2025-05-27T13:02:55.735Z
6555a124a6554059711b58a2
/avatars/222bb6b8f252d6c2bbd4cf35a54fc1c9.svg
Roy
user
RRoy233
68353e3f9f4e0a0f0496d0c7
David Qiu
68353e3f9f4e0a0f0496d0c8
Deepak Gopinath
admin_assigned
2025-05-27T13:03:07.418Z
66f6df6329dad7fec7cf26f2
/avatars/e576fbeca6c5770905408316bd9bc0f2.svg
Deepak Gopinath
user
dpacgopinath
68353e3f9f4e0a0f0496d0c9
Dong Lin
68353e3f9f4e0a0f0496d0ca
Yanchao Sun
admin_assigned
2025-05-27T13:03:16.312Z
6488a8e65815252b02b948c8
/avatars/bb1ff64841c75b8d331986709b0a0a93.svg
Yanchao Sun
user
yanchao-sun
68353e3f9f4e0a0f0496d0cb
Chong Wang
68353e3f9f4e0a0f0496d0cc
Saloni Potdar
admin_assigned
2025-05-27T13:03:28.257Z
66711f329c609c2484c76d80
/avatars/717e0ce3e8693e99da6d61199ea712f3.svg
Saloni Potdar
user
spotdar
68353e3f9f4e0a0f0496d0cd
Bhuwan Dhingra
Interleaved Reasoning for Large Language Models via Reinforcement
  Learning
Long chain-of-thought (CoT) significantly enhances large language models'
(LLM) reasoning capabilities. However, the extensive reasoning traces lead to
inefficiencies and an increased time-to-first-token (TTFT). We propose a novel
training paradigm that uses reinforcement learning (RL) to guide reasoning LLMs
to interleave thinking and answering for multi-hop questions. We observe that
models inherently possess the ability to perform interleaved reasoning, which
can be further enhanced through RL. We introduce a simple yet effective
rule-based reward to incentivize correct intermediate steps, which guides the
policy model toward correct reasoning paths by leveraging intermediate signals
generated during interleaved reasoning. Extensive experiments conducted across
five diverse datasets and three RL algorithms (PPO, GRPO, and REINFORCE++)
demonstrate consistent improvements over traditional think-answer reasoning,
without requiring external tools. Specifically, our approach reduces TTFT by
over 80% on average and improves up to 19.3% in Pass@1 accuracy. Furthermore,
our method, trained solely on question answering and logical reasoning
datasets, exhibits strong generalization ability to complex reasoning datasets
such as MATH, GPQA, and MMLU. Additionally, we conduct in-depth analysis to
reveal several valuable insights into conditional reward modeling.
A reinforcement learning-guided training paradigm enhances large language models' reasoning efficiency and performance for multi-hop questions by interleaving thinking and answering.
chain-of-thought large language models reasoning capabilities time-to-first-token reinforcement learning interleaved reasoning rule-based reward reward modeling multi-hop questions think-answer reasoning Pass@1 accuracy MATH GPQA MMLU PPO GRPO REINFORCE++