title: Language Models can Self-Improve at State-Value Estimation for Better
    Search
summary: Collecting ground truth task completion rewards or human demonstrations for
  multi-step reasoning tasks is often cost-prohibitive and time-consuming,
  especially in interactive domains like web tasks. To address this bottleneck,
  we present self-taught lookahead, a self-supervised method that leverages
  state-transition dynamics to train a value model capable of effectively guiding
  language model-controlled search. We find that moderately sized (8 billion
  parameters) open-weight value models improved with self-taught lookahead can
  match the performance of using a frontier LLM such as gpt-4o as the value
  model. Furthermore, we find that self-taught lookahead improves performance by
  20% while reducing costs 37x compared to previous LLM-based tree search,
  without relying on ground truth rewards.
paper_authors._id: 67c7bf7c40de8b1b534d23fa
paper_authors.name: Ethan Mendes
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-05T03:06:00.588Z
paper_authors._id: 635d76ce94e5b275ca74b967
paper_authors.avatarUrl: /avatars/5d9a389e5fd558c0b8f0724bf0838a3e.svg
paper_authors.fullname: Ethan Mendes
paper_authors.type: user
paper_authors.user: emendes3
paper_authors._id: 67c7bf7c40de8b1b534d23fb
paper_authors.name: Alan Ritter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:19:56.686Z
paper_authors._id: 6377da9065ce6b04bd493cf8
paper_authors.avatarUrl: /avatars/6be2632d6da49857928f8bcce8278f49.svg
paper_authors.fullname: Alan Ritter
paper_authors.type: user
paper_authors.user: rittera
paper_title: Language Models can Self-Improve at State-Value Estimation for Better
    Search
paper_summary: Collecting ground truth task completion rewards or human demonstrations for
  multi-step reasoning tasks is often cost-prohibitive and time-consuming,
  especially in interactive domains like web tasks. To address this bottleneck,
  we present self-taught lookahead, a self-supervised method that leverages
  state-transition dynamics to train a value model capable of effectively guiding
  language model-controlled search. We find that moderately sized (8 billion
  parameters) open-weight value models improved with self-taught lookahead can
  match the performance of using a frontier LLM such as gpt-4o as the value
  model. Furthermore, we find that self-taught lookahead improves performance by
  20% while reducing costs 37x compared to previous LLM-based tree search,
  without relying on ground truth rewards.
paper_ai_keywords: self-taught lookahead self-supervised method state-transition dynamics value model language model-controlled search open-weight value models frontier LLM gpt-4o tree search