title: QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
summary: Language agents have become a promising solution to complex interactive
  tasks. One of the key ingredients to the success of language agents is the
  reward model on the trajectory of the agentic workflow, which provides valuable
  guidance during training or inference. However, due to the lack of annotations
  of intermediate interactions, most existing works use an outcome reward model
  to optimize policies across entire trajectories. This may lead to sub-optimal
  policies and hinder the overall performance. To address this, we propose QLASS
  (Q-guided Language Agent Stepwise Search), to automatically generate
  annotations by estimating Q-values in a stepwise manner for open language
  agents. By introducing a reasoning tree and performing process reward modeling,
  QLASS provides effective intermediate guidance for each step. With the stepwise
  guidance, we propose a Q-guided generation strategy to enable language agents
  to better adapt to long-term value, resulting in significant performance
  improvement during model inference on complex interactive agent tasks. Notably,
  even with almost half the annotated data, QLASS retains strong performance,
  demonstrating its efficiency in handling limited supervision. We also
  empirically demonstrate that QLASS can lead to more effective decision making
  through qualitative analysis. We will release our code and data.
paper_authors._id: 67a2d59fd5ad3369a66ff394
paper_authors.name: Zongyu Lin
paper_authors._id: 67a2d59fd5ad3369a66ff395
paper_authors.name: Yao Tang
paper_authors._id: 67a2d59fd5ad3369a66ff396
paper_authors.name: Xingcheng Yao
paper_authors._id: 67a2d59fd5ad3369a66ff397
paper_authors.name: Da Yin
paper_authors._id: 67a2d59fd5ad3369a66ff398
paper_authors.name: Ziniu Hu
paper_authors._id: 67a2d59fd5ad3369a66ff399
paper_authors.name: Yizhou Sun
paper_authors._id: 67a2d59fd5ad3369a66ff39a
paper_authors.name: Kai-Wei Chang
paper_title: QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
paper_summary: Language agents have become a promising solution to complex interactive
  tasks. One of the key ingredients to the success of language agents is the
  reward model on the trajectory of the agentic workflow, which provides valuable
  guidance during training or inference. However, due to the lack of annotations
  of intermediate interactions, most existing works use an outcome reward model
  to optimize policies across entire trajectories. This may lead to sub-optimal
  policies and hinder the overall performance. To address this, we propose QLASS
  (Q-guided Language Agent Stepwise Search), to automatically generate
  annotations by estimating Q-values in a stepwise manner for open language
  agents. By introducing a reasoning tree and performing process reward modeling,
  QLASS provides effective intermediate guidance for each step. With the stepwise
  guidance, we propose a Q-guided generation strategy to enable language agents
  to better adapt to long-term value, resulting in significant performance
  improvement during model inference on complex interactive agent tasks. Notably,
  even with almost half the annotated data, QLASS retains strong performance,
  demonstrating its efficiency in handling limited supervision. We also
  empirically demonstrate that QLASS can lead to more effective decision making
  through qualitative analysis. We will release our code and data.
paper_ai_keywords: reward model agentic workflow outcome reward model policies QLASS Q-guided Language Agent Stepwise Search Q-values reasoning tree process reward modeling Q-guided generation strategy long-term value performance improvement