title: Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning
    of LLMs
summary: Large language models have demonstrated impressive reasoning capabilities but
  are inherently limited by their knowledge reservoir. Retrieval-augmented
  reasoning mitigates this limitation by allowing LLMs to query external
  resources, but existing methods often retrieve irrelevant or noisy information,
  hindering accurate reasoning. In this paper, we propose AutoRefine, a
  reinforcement learning post-training framework that adopts a new
  ``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit
  knowledge refinement steps between successive search calls, enabling the model
  to iteratively filter, distill, and organize evidence before generating an
  answer. Furthermore, we incorporate tailored retrieval-specific rewards
  alongside answer correctness rewards using group relative policy optimization.
  Experiments on single-hop and multi-hop QA benchmarks demonstrate that
  AutoRefine significantly outperforms existing approaches, particularly in
  complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine
  issues frequent, higher-quality searches and synthesizes evidence effectively.
paper_authors._id: 6834f8d4bb7d1147551cc8f6
paper_authors.name: Yaorui Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:51.919Z
paper_authors._id: 63edd2d1f765928ceeb49057
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
paper_authors.fullname: Yaorui SHI
paper_authors.type: user
paper_authors.user: yrshi
paper_authors._id: 6834f8d4bb7d1147551cc8f7
paper_authors.name: Sihang Li
paper_authors._id: 6834f8d4bb7d1147551cc8f8
paper_authors.name: Chang Wu
paper_authors._id: 6834f8d4bb7d1147551cc8f9
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:49:10.763Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 6834f8d4bb7d1147551cc8fa
paper_authors.name: Junfeng Fang
paper_authors._id: 6834f8d4bb7d1147551cc8fb
paper_authors.name: Hengxing Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:35:49.890Z
paper_authors._id: 6340c214181f5648a58f5f82
paper_authors.avatarUrl: /avatars/2ee353d8ec2b0ebc5b58d0641beddc26.svg
paper_authors.fullname: Hengxing Cai
paper_authors.type: user
paper_authors.user: enjoy
paper_authors._id: 6834f8d4bb7d1147551cc8fc
paper_authors.name: An Zhang
paper_authors._id: 6834f8d4bb7d1147551cc8fd
paper_authors.name: Xiang Wang
paper_title: Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning
    of LLMs
paper_summary: Large language models have demonstrated impressive reasoning capabilities but
  are inherently limited by their knowledge reservoir. Retrieval-augmented
  reasoning mitigates this limitation by allowing LLMs to query external
  resources, but existing methods often retrieve irrelevant or noisy information,
  hindering accurate reasoning. In this paper, we propose AutoRefine, a
  reinforcement learning post-training framework that adopts a new
  ``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit
  knowledge refinement steps between successive search calls, enabling the model
  to iteratively filter, distill, and organize evidence before generating an
  answer. Furthermore, we incorporate tailored retrieval-specific rewards
  alongside answer correctness rewards using group relative policy optimization.
  Experiments on single-hop and multi-hop QA benchmarks demonstrate that
  AutoRefine significantly outperforms existing approaches, particularly in
  complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine
  issues frequent, higher-quality searches and synthesizes evidence effectively.
paper_ai_summary: AutoRefine, a reinforcement learning framework for large language models, enhances retrieval-augmented reasoning by iteratively refining knowledge and optimizing searches, leading to improved performance in complex question-answering tasks.
paper_ai_keywords: Large language models Retrieval-augmented reasoning AutoRefine Reinforcement learning Search-and-refine-during-think Reinforcement learning post-training Group relative policy optimization Single-hop QA benchmarks Multi-hop QA benchmarks