title: AutoLibra: Agent Metric Induction from Open-Ended Feedback
summary: Agents are predominantly evaluated and optimized via task success metrics,
  which are coarse, rely on manual design from experts, and fail to reward
  intermediate emergent behaviors. We propose AutoLibra, a framework for agent
  evaluation, that transforms open-ended human feedback, e.g., "If you find that
  the button is disabled, don't click it again", or "This agent has too much
  autonomy to decide what to do on its own", into metrics for evaluating
  fine-grained behaviors in agent trajectories. AutoLibra accomplishes this by
  grounding feedback to an agent's behavior, clustering similar positive and
  negative behaviors, and creating concrete metrics with clear definitions and
  concrete examples, which can be used for prompting LLM-as-a-Judge as
  evaluators. We further propose two meta-metrics to evaluate the alignment of a
  set of (induced) metrics with open feedback: "coverage" and "redundancy".
  Through optimizing these meta-metrics, we experimentally demonstrate
  AutoLibra's ability to induce more concrete agent evaluation metrics than the
  ones proposed in previous agent evaluation benchmarks and discover new metrics
  to analyze agents. We also present two applications of AutoLibra in agent
  improvement: First, we show that AutoLibra-induced metrics serve as better
  prompt-engineering targets than the task success rate on a wide range of text
  game tasks, improving agent performance over baseline by a mean of 20%. Second,
  we show that AutoLibra can iteratively select high-quality fine-tuning data for
  web navigation agents. Our results suggest that AutoLibra is a powerful
  task-agnostic tool for evaluating and improving language agents.
paper_authors._id: 681cd2499722c5341cabd43f
paper_authors.name: Hao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:22:10.299Z
paper_authors._id: 61aa376688c20eebf1e8deb3
paper_authors.avatarUrl: /avatars/7c11dcb232c73547d7d87834be287822.svg
paper_authors.fullname: Hao Zhu
paper_authors.type: user
paper_authors.user: ProKil
paper_authors._id: 681cd2499722c5341cabd440
paper_authors.name: Phil Cuvin
paper_authors._id: 681cd2499722c5341cabd441
paper_authors.name: Xinkai Yu
paper_authors._id: 681cd2499722c5341cabd442
paper_authors.name: Charlotte Ka Yee Yan
paper_authors._id: 681cd2499722c5341cabd443
paper_authors.name: Jason Zhang
paper_authors._id: 681cd2499722c5341cabd444
paper_authors.name: Diyi Yang
paper_title: AutoLibra: Agent Metric Induction from Open-Ended Feedback
paper_summary: Agents are predominantly evaluated and optimized via task success metrics,
  which are coarse, rely on manual design from experts, and fail to reward
  intermediate emergent behaviors. We propose AutoLibra, a framework for agent
  evaluation, that transforms open-ended human feedback, e.g., "If you find that
  the button is disabled, don't click it again", or "This agent has too much
  autonomy to decide what to do on its own", into metrics for evaluating
  fine-grained behaviors in agent trajectories. AutoLibra accomplishes this by
  grounding feedback to an agent's behavior, clustering similar positive and
  negative behaviors, and creating concrete metrics with clear definitions and
  concrete examples, which can be used for prompting LLM-as-a-Judge as
  evaluators. We further propose two meta-metrics to evaluate the alignment of a
  set of (induced) metrics with open feedback: "coverage" and "redundancy".
  Through optimizing these meta-metrics, we experimentally demonstrate
  AutoLibra's ability to induce more concrete agent evaluation metrics than the
  ones proposed in previous agent evaluation benchmarks and discover new metrics
  to analyze agents. We also present two applications of AutoLibra in agent
  improvement: First, we show that AutoLibra-induced metrics serve as better
  prompt-engineering targets than the task success rate on a wide range of text
  game tasks, improving agent performance over baseline by a mean of 20%. Second,
  we show that AutoLibra can iteratively select high-quality fine-tuning data for
  web navigation agents. Our results suggest that AutoLibra is a powerful
  task-agnostic tool for evaluating and improving language agents.
paper_ai_summary: AutoLibra transforms human feedback into fine-grained agent evaluation metrics using machine learning, demonstrating improved agent performance and data selection.
paper_ai_keywords: LLM-as-a-Judge agent evaluation open-ended human feedback clustering coverage redundancy prompt-engineering fine-tuning web navigation agents