s3: You Don't Need That Much Data to Train a Search Agent via RL
Retrieval-augmented generation (RAG) systems empower large language models
(LLMs) to access external knowledge during inference. Recent advances have
enabled LLMs to act as search agents via reinforcement learning (RL), improving
information acquisition through multi-turn interactions with retrieval engines.
However, existing approaches either optimize retrieval using search-only
metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM
to jointly reason and retrieve-entangling retrieval with generation and
limiting the real search utility and compatibility with frozen or proprietary
models. In this work, we propose s3, a lightweight, model-agnostic framework
that decouples the searcher from the generator and trains the searcher using a
Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG.
s3 requires only 2.4k training samples to outperform baselines trained on over
70x more data, consistently delivering stronger downstream performance across
six general QA and five medical QA benchmarks.
68343b73f0b7aba41a360e8f
Pengcheng Jiang
claimed_verified
2025-05-26T13:18:29.181Z
63724cfada3183d9d53f2009
/avatars/17838fcf244ecf8d139343bb6c6d8562.svg
Patrick Jiang
user
pat-jj
68343b73f0b7aba41a360e90
Xueqiang Xu
claimed_verified
2025-05-26T14:07:25.830Z
66a3f1c4c38ce500371fd8d4
/avatars/381de938091f1a5c179eef72aa247bbf.svg
Xueqiang Xu
user
XueqiangXu
68343b73f0b7aba41a360e91
Jiacheng Lin
68343b73f0b7aba41a360e92
Jinfeng Xiao
68343b73f0b7aba41a360e93
Zifeng Wang
68343b73f0b7aba41a360e94
Jimeng Sun
68343b73f0b7aba41a360e95
Jiawei Han
s3: You Don't Need That Much Data to Train a Search Agent via RL
Retrieval-augmented generation (RAG) systems empower large language models
(LLMs) to access external knowledge during inference. Recent advances have
enabled LLMs to act as search agents via reinforcement learning (RL), improving
information acquisition through multi-turn interactions with retrieval engines.
However, existing approaches either optimize retrieval using search-only
metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM
to jointly reason and retrieve-entangling retrieval with generation and
limiting the real search utility and compatibility with frozen or proprietary
models. In this work, we propose s3, a lightweight, model-agnostic framework
that decouples the searcher from the generator and trains the searcher using a
Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG.
s3 requires only 2.4k training samples to outperform baselines trained on over
70x more data, consistently delivering stronger downstream performance across
six general QA and five medical QA benchmarks.
A lightweight, model-agnostic framework decouples the retrieval and generation processes in RAG systems, enhancing performance with minimal training data.
Retrieval-augmented generation (RAG) large language models (LLMs) reinforcement learning (RL) retrieval engines NDCG Gain Beyond RAG search utility model-agnostic framework QA benchmarks