R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs
  via Reinforcement Learning
Large Language Models (LLMs) are powerful but prone to hallucinations due to
static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting
external information, but current methods often are costly, generalize poorly,
or ignore the internal knowledge of the model. In this paper, we introduce
R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage
both internal and external knowledge sources. R1-Searcher++ employs a two-stage
training strategy: an initial SFT Cold-start phase for preliminary format
learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses
outcome-supervision to encourage exploration, incorporates a reward mechanism
for internal knowledge utilization, and integrates a memorization mechanism to
continuously assimilate retrieved information, thereby enriching the model's
internal knowledge. By leveraging internal knowledge and external search
engine, the model continuously improves its capabilities, enabling efficient
retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++
outperforms previous RAG and reasoning methods and achieves efficient
retrieval. The code is available at
https://github.com/RUCAIBox/R1-Searcher-plus.
683469f0df7cbb5c08a0498a
Huatong Song
683469f0df7cbb5c08a0498b
Jinhao Jiang
683469f0df7cbb5c08a0498c
Wenqing Tian
683469f0df7cbb5c08a0498d
Zhipeng Chen
683469f0df7cbb5c08a0498e
Yuhuan Wu
683469f0df7cbb5c08a0498f
Jiahao Zhao
683469f0df7cbb5c08a04990
Yingqian Min
claimed_verified
2025-05-28T09:00:55.996Z
6703ac76ea890f0ca5b225eb
/avatars/5f56c49a1940143d47dd484782a4abbf.svg
Yingqian Min
user
EliverQ
683469f0df7cbb5c08a04991
Wayne Xin Zhao
683469f0df7cbb5c08a04992
Lei Fang
683469f0df7cbb5c08a04993
Ji-Rong Wen
R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs
  via Reinforcement Learning
Large Language Models (LLMs) are powerful but prone to hallucinations due to
static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting
external information, but current methods often are costly, generalize poorly,
or ignore the internal knowledge of the model. In this paper, we introduce
R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage
both internal and external knowledge sources. R1-Searcher++ employs a two-stage
training strategy: an initial SFT Cold-start phase for preliminary format
learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses
outcome-supervision to encourage exploration, incorporates a reward mechanism
for internal knowledge utilization, and integrates a memorization mechanism to
continuously assimilate retrieved information, thereby enriching the model's
internal knowledge. By leveraging internal knowledge and external search
engine, the model continuously improves its capabilities, enabling efficient
retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++
outperforms previous RAG and reasoning methods and achieves efficient
retrieval. The code is available at
https://github.com/RUCAIBox/R1-Searcher-plus.
R1-Searcher++, a novel framework, enhances LLMs by adaptively integrating internal and external knowledge through two-stage training, improving retrieval-augmented reasoning efficiency and performance.
Large Language Models (LLMs) hallucinations Retrieval-Augmented Generation (RAG) R1-Searcher++ SFT Cold-start Reinforcement Learning (RL) outcome-supervision reward mechanism memorization mechanism retrieval-augmented reasoning