title: s3: You Don't Need That Much Data to Train a Search Agent via RL
summary: Retrieval-augmented generation (RAG) systems empower large language models
  (LLMs) to access external knowledge during inference. Recent advances have
  enabled LLMs to act as search agents via reinforcement learning (RL), improving
  information acquisition through multi-turn interactions with retrieval engines.
  However, existing approaches either optimize retrieval using search-only
  metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM
  to jointly reason and retrieve-entangling retrieval with generation and
  limiting the real search utility and compatibility with frozen or proprietary
  models. In this work, we propose s3, a lightweight, model-agnostic framework
  that decouples the searcher from the generator and trains the searcher using a
  Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG.
  s3 requires only 2.4k training samples to outperform baselines trained on over
  70x more data, consistently delivering stronger downstream performance across
  six general QA and five medical QA benchmarks.
paper_authors._id: 68343b73f0b7aba41a360e8f
paper_authors.name: Pengcheng Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T13:18:29.181Z
paper_authors._id: 63724cfada3183d9d53f2009
paper_authors.avatarUrl: /avatars/17838fcf244ecf8d139343bb6c6d8562.svg
paper_authors.fullname: Patrick Jiang
paper_authors.type: user
paper_authors.user: pat-jj
paper_authors._id: 68343b73f0b7aba41a360e90
paper_authors.name: Xueqiang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:25.830Z
paper_authors._id: 66a3f1c4c38ce500371fd8d4
paper_authors.avatarUrl: /avatars/381de938091f1a5c179eef72aa247bbf.svg
paper_authors.fullname: Xueqiang Xu
paper_authors.type: user
paper_authors.user: XueqiangXu
paper_authors._id: 68343b73f0b7aba41a360e91
paper_authors.name: Jiacheng Lin
paper_authors._id: 68343b73f0b7aba41a360e92
paper_authors.name: Jinfeng Xiao
paper_authors._id: 68343b73f0b7aba41a360e93
paper_authors.name: Zifeng Wang
paper_authors._id: 68343b73f0b7aba41a360e94
paper_authors.name: Jimeng Sun
paper_authors._id: 68343b73f0b7aba41a360e95
paper_authors.name: Jiawei Han
paper_title: s3: You Don't Need That Much Data to Train a Search Agent via RL
paper_summary: Retrieval-augmented generation (RAG) systems empower large language models
  (LLMs) to access external knowledge during inference. Recent advances have
  enabled LLMs to act as search agents via reinforcement learning (RL), improving
  information acquisition through multi-turn interactions with retrieval engines.
  However, existing approaches either optimize retrieval using search-only
  metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM
  to jointly reason and retrieve-entangling retrieval with generation and
  limiting the real search utility and compatibility with frozen or proprietary
  models. In this work, we propose s3, a lightweight, model-agnostic framework
  that decouples the searcher from the generator and trains the searcher using a
  Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG.
  s3 requires only 2.4k training samples to outperform baselines trained on over
  70x more data, consistently delivering stronger downstream performance across
  six general QA and five medical QA benchmarks.
paper_ai_summary: A lightweight, model-agnostic framework decouples the retrieval and generation processes in RAG systems, enhancing performance with minimal training data.
paper_ai_keywords: Retrieval-augmented generation (RAG) large language models (LLMs) reinforcement learning (RL) retrieval engines NDCG Gain Beyond RAG search utility model-agnostic framework QA benchmarks