title: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World
    Domains
summary: Existing benchmarks do not test language agents on their interaction with
  human users or ability to follow domain-specific rules, both of which are vital
  for deploying them in real world applications. We propose tau-bench, a
  benchmark emulating dynamic conversations between a user (simulated by language
  models) and a language agent provided with domain-specific API tools and policy
  guidelines. We employ an efficient and faithful evaluation process that
  compares the database state at the end of a conversation with the annotated
  goal state. We also propose a new metric (pass^k) to evaluate the reliability
  of agent behavior over multiple trials. Our experiments show that even
  state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the
  tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point
  to the need for methods that can improve the ability of agents to act
  consistently and follow rules reliably.
paper_authors._id: 6674ffba568251b0b77afccf
paper_authors.name: Shunyu Yao
paper_authors._id: 6674ffba568251b0b77afcd0
paper_authors.name: Noah Shinn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:54:46.421Z
paper_authors._id: 63ca0095b047b9b12a1face0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ca0095b047b9b12a1face0/Eq_BCfFPbBig4XxSHs3xW.png
paper_authors.fullname: Noah Shinn
paper_authors.type: user
paper_authors.user: noahshinn
paper_authors._id: 6674ffba568251b0b77afcd1
paper_authors.name: Pedram Razavi
paper_authors._id: 6674ffba568251b0b77afcd2
paper_authors.name: Karthik Narasimhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:54:32.248Z
paper_authors._id: 6643e737ba3c6c8d116e1585
paper_authors.avatarUrl: /avatars/7a780509f14d86d1e6ec37cb39ea1f5d.svg
paper_authors.fullname: Karthik Narasimhan
paper_authors.type: user
paper_authors.user: karthikn-sierra
paper_title: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World
    Domains
paper_summary: Existing benchmarks do not test language agents on their interaction with
  human users or ability to follow domain-specific rules, both of which are vital
  for deploying them in real world applications. We propose tau-bench, a
  benchmark emulating dynamic conversations between a user (simulated by language
  models) and a language agent provided with domain-specific API tools and policy
  guidelines. We employ an efficient and faithful evaluation process that
  compares the database state at the end of a conversation with the annotated
  goal state. We also propose a new metric (pass^k) to evaluate the reliability
  of agent behavior over multiple trials. Our experiments show that even
  state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the
  tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point
  to the need for methods that can improve the ability of agents to act
  consistently and follow rules reliably.
paper_ai_summary: Tau-bench tests agents' interaction with simulated users and adherence to domain-specific rules, revealing inconsistencies in function calling agents.
paper_ai_keywords: tau-bench dynamic conversations language models domain-specific API tools policy guidelines pass^k function calling agents retail