ConvSearch-R1: Enhancing Query Reformulation for Conversational Search
  with Reasoning via Reinforcement Learning
Conversational search systems require effective handling of context-dependent
queries that often contain ambiguity, omission, and coreference. Conversational
Query Reformulation (CQR) addresses this challenge by transforming these
queries into self-contained forms suitable for off-the-shelf retrievers.
However, existing CQR approaches suffer from two critical constraints: high
dependency on costly external supervision from human annotations or large
language models, and insufficient alignment between the rewriting model and
downstream retrievers. We present ConvSearch-R1, the first self-driven
framework that completely eliminates dependency on external rewrite supervision
by leveraging reinforcement learning to optimize reformulation directly through
retrieval signals. Our novel two-stage approach combines Self-Driven Policy
Warm-Up to address the cold-start problem through retrieval-guided
self-distillation, followed by Retrieval-Guided Reinforcement Learning with a
specially designed rank-incentive reward shaping mechanism that addresses the
sparsity issue in conventional retrieval metrics. Extensive experiments on
TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly
outperforms previous state-of-the-art methods, achieving over 10% improvement
on the challenging TopiOCQA dataset while using smaller 3B parameter models
without any external supervision.
682eb95671d01f3fc74f0f8c
Changtai Zhu
claimed_verified
2025-05-22T07:15:31.485Z
644a41fcd9a3ae8341055179
/avatars/21b35abdc60a34589443b5879901eb46.svg
Changtai Zhu
user
BeastyZ
682eb95671d01f3fc74f0f8d
Siyin Wang
682eb95671d01f3fc74f0f8e
Ruijun Feng
682eb95671d01f3fc74f0f8f
Kai Song
682eb95671d01f3fc74f0f90
Xipeng Qiu
ConvSearch-R1: Enhancing Query Reformulation for Conversational Search
  with Reasoning via Reinforcement Learning
Conversational search systems require effective handling of context-dependent
queries that often contain ambiguity, omission, and coreference. Conversational
Query Reformulation (CQR) addresses this challenge by transforming these
queries into self-contained forms suitable for off-the-shelf retrievers.
However, existing CQR approaches suffer from two critical constraints: high
dependency on costly external supervision from human annotations or large
language models, and insufficient alignment between the rewriting model and
downstream retrievers. We present ConvSearch-R1, the first self-driven
framework that completely eliminates dependency on external rewrite supervision
by leveraging reinforcement learning to optimize reformulation directly through
retrieval signals. Our novel two-stage approach combines Self-Driven Policy
Warm-Up to address the cold-start problem through retrieval-guided
self-distillation, followed by Retrieval-Guided Reinforcement Learning with a
specially designed rank-incentive reward shaping mechanism that addresses the
sparsity issue in conventional retrieval metrics. Extensive experiments on
TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly
outperforms previous state-of-the-art methods, achieving over 10% improvement
on the challenging TopiOCQA dataset while using smaller 3B parameter models
without any external supervision.
ConvSearch-R1 uses reinforcement learning and self-distillation to improve conversational query reformulation without relying on external supervision, outperforming state-of-the-art methods.
Conversational Query Reformulation CQR reinforcement learning self-distillation cold-start problem rank-incentive reward shaping retrieval signals TopiOCQA QReCC