title: R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs
    via Reinforcement Learning
summary: Large Language Models (LLMs) are powerful but prone to hallucinations due to
  static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting
  external information, but current methods often are costly, generalize poorly,
  or ignore the internal knowledge of the model. In this paper, we introduce
  R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage
  both internal and external knowledge sources. R1-Searcher++ employs a two-stage
  training strategy: an initial SFT Cold-start phase for preliminary format
  learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses
  outcome-supervision to encourage exploration, incorporates a reward mechanism
  for internal knowledge utilization, and integrates a memorization mechanism to
  continuously assimilate retrieved information, thereby enriching the model's
  internal knowledge. By leveraging internal knowledge and external search
  engine, the model continuously improves its capabilities, enabling efficient
  retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++
  outperforms previous RAG and reasoning methods and achieves efficient
  retrieval. The code is available at
  https://github.com/RUCAIBox/R1-Searcher-plus.
paper_authors._id: 683469f0df7cbb5c08a0498a
paper_authors.name: Huatong Song
paper_authors._id: 683469f0df7cbb5c08a0498b
paper_authors.name: Jinhao Jiang
paper_authors._id: 683469f0df7cbb5c08a0498c
paper_authors.name: Wenqing Tian
paper_authors._id: 683469f0df7cbb5c08a0498d
paper_authors.name: Zhipeng Chen
paper_authors._id: 683469f0df7cbb5c08a0498e
paper_authors.name: Yuhuan Wu
paper_authors._id: 683469f0df7cbb5c08a0498f
paper_authors.name: Jiahao Zhao
paper_authors._id: 683469f0df7cbb5c08a04990
paper_authors.name: Yingqian Min
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:55.996Z
paper_authors._id: 6703ac76ea890f0ca5b225eb
paper_authors.avatarUrl: /avatars/5f56c49a1940143d47dd484782a4abbf.svg
paper_authors.fullname: Yingqian Min
paper_authors.type: user
paper_authors.user: EliverQ
paper_authors._id: 683469f0df7cbb5c08a04991
paper_authors.name: Wayne Xin Zhao
paper_authors._id: 683469f0df7cbb5c08a04992
paper_authors.name: Lei Fang
paper_authors._id: 683469f0df7cbb5c08a04993
paper_authors.name: Ji-Rong Wen
paper_title: R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs
    via Reinforcement Learning
paper_summary: Large Language Models (LLMs) are powerful but prone to hallucinations due to
  static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting
  external information, but current methods often are costly, generalize poorly,
  or ignore the internal knowledge of the model. In this paper, we introduce
  R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage
  both internal and external knowledge sources. R1-Searcher++ employs a two-stage
  training strategy: an initial SFT Cold-start phase for preliminary format
  learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses
  outcome-supervision to encourage exploration, incorporates a reward mechanism
  for internal knowledge utilization, and integrates a memorization mechanism to
  continuously assimilate retrieved information, thereby enriching the model's
  internal knowledge. By leveraging internal knowledge and external search
  engine, the model continuously improves its capabilities, enabling efficient
  retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++
  outperforms previous RAG and reasoning methods and achieves efficient
  retrieval. The code is available at
  https://github.com/RUCAIBox/R1-Searcher-plus.
paper_ai_summary: R1-Searcher++, a novel framework, enhances LLMs by adaptively integrating internal and external knowledge through two-stage training, improving retrieval-augmented reasoning efficiency and performance.
paper_ai_keywords: Large Language Models (LLMs) hallucinations Retrieval-Augmented Generation (RAG) R1-Searcher++ SFT Cold-start Reinforcement Learning (RL) outcome-supervision reward mechanism memorization mechanism retrieval-augmented reasoning