title: ConvSearch-R1: Enhancing Query Reformulation for Conversational Search
    with Reasoning via Reinforcement Learning
summary: Conversational search systems require effective handling of context-dependent
  queries that often contain ambiguity, omission, and coreference. Conversational
  Query Reformulation (CQR) addresses this challenge by transforming these
  queries into self-contained forms suitable for off-the-shelf retrievers.
  However, existing CQR approaches suffer from two critical constraints: high
  dependency on costly external supervision from human annotations or large
  language models, and insufficient alignment between the rewriting model and
  downstream retrievers. We present ConvSearch-R1, the first self-driven
  framework that completely eliminates dependency on external rewrite supervision
  by leveraging reinforcement learning to optimize reformulation directly through
  retrieval signals. Our novel two-stage approach combines Self-Driven Policy
  Warm-Up to address the cold-start problem through retrieval-guided
  self-distillation, followed by Retrieval-Guided Reinforcement Learning with a
  specially designed rank-incentive reward shaping mechanism that addresses the
  sparsity issue in conventional retrieval metrics. Extensive experiments on
  TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly
  outperforms previous state-of-the-art methods, achieving over 10% improvement
  on the challenging TopiOCQA dataset while using smaller 3B parameter models
  without any external supervision.
paper_authors._id: 682eb95671d01f3fc74f0f8c
paper_authors.name: Changtai Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:31.485Z
paper_authors._id: 644a41fcd9a3ae8341055179
paper_authors.avatarUrl: /avatars/21b35abdc60a34589443b5879901eb46.svg
paper_authors.fullname: Changtai Zhu
paper_authors.type: user
paper_authors.user: BeastyZ
paper_authors._id: 682eb95671d01f3fc74f0f8d
paper_authors.name: Siyin Wang
paper_authors._id: 682eb95671d01f3fc74f0f8e
paper_authors.name: Ruijun Feng
paper_authors._id: 682eb95671d01f3fc74f0f8f
paper_authors.name: Kai Song
paper_authors._id: 682eb95671d01f3fc74f0f90
paper_authors.name: Xipeng Qiu
paper_title: ConvSearch-R1: Enhancing Query Reformulation for Conversational Search
    with Reasoning via Reinforcement Learning
paper_summary: Conversational search systems require effective handling of context-dependent
  queries that often contain ambiguity, omission, and coreference. Conversational
  Query Reformulation (CQR) addresses this challenge by transforming these
  queries into self-contained forms suitable for off-the-shelf retrievers.
  However, existing CQR approaches suffer from two critical constraints: high
  dependency on costly external supervision from human annotations or large
  language models, and insufficient alignment between the rewriting model and
  downstream retrievers. We present ConvSearch-R1, the first self-driven
  framework that completely eliminates dependency on external rewrite supervision
  by leveraging reinforcement learning to optimize reformulation directly through
  retrieval signals. Our novel two-stage approach combines Self-Driven Policy
  Warm-Up to address the cold-start problem through retrieval-guided
  self-distillation, followed by Retrieval-Guided Reinforcement Learning with a
  specially designed rank-incentive reward shaping mechanism that addresses the
  sparsity issue in conventional retrieval metrics. Extensive experiments on
  TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly
  outperforms previous state-of-the-art methods, achieving over 10% improvement
  on the challenging TopiOCQA dataset while using smaller 3B parameter models
  without any external supervision.
paper_ai_summary: ConvSearch-R1 uses reinforcement learning and self-distillation to improve conversational query reformulation without relying on external supervision, outperforming state-of-the-art methods.
paper_ai_keywords: Conversational Query Reformulation CQR reinforcement learning self-distillation cold-start problem rank-incentive reward shaping retrieval signals TopiOCQA QReCC