title: TRAIL: Trace Reasoning and Agentic Issue Localization
summary: The increasing adoption of agentic workflows across diverse domains brings a
  critical need to scalably and systematically evaluate the complex traces these
  systems generate. Current evaluation methods depend on manual, domain-specific
  human analysis of lengthy workflow traces - an approach that does not scale
  with the growing complexity and volume of agentic outputs. Error analysis in
  these settings is further complicated by the interplay of external tool outputs
  and language model reasoning, making it more challenging than traditional
  software debugging. In this work, we (1) articulate the need for robust and
  dynamic evaluation methods for agentic workflow traces, (2) introduce a formal
  taxonomy of error types encountered in agentic systems, and (3) present a set
  of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and
  grounded in established agentic benchmarks. To ensure ecological validity, we
  curate traces from both single and multi-agent systems, focusing on real-world
  applications such as software engineering and open-world information retrieval.
  Our evaluations reveal that modern long context LLMs perform poorly at trace
  debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our
  dataset and code are made publicly available to support and accelerate future
  research in scalable evaluation for agentic workflows.
paper_authors._id: 6824bddc7ecdd81cdac3b346
paper_authors.name: Darshan Deshpande
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:32:13.902Z
paper_authors._id: 60390e04770949ef34f12d9b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632225571380-60390e04770949ef34f12d9b.jpeg
paper_authors.fullname: Darshan Deshpande
paper_authors.type: user
paper_authors.user: DarshanDeshpande
paper_authors._id: 6824bddc7ecdd81cdac3b347
paper_authors.name: Varun Gangal
paper_authors._id: 6824bddc7ecdd81cdac3b348
paper_authors.name: Hersh Mehta
paper_authors._id: 6824bddc7ecdd81cdac3b349
paper_authors.name: Jitin Krishnan
paper_authors._id: 6824bddc7ecdd81cdac3b34a
paper_authors.name: Anand Kannappan
paper_authors._id: 6824bddc7ecdd81cdac3b34b
paper_authors.name: Rebecca Qian
paper_title: TRAIL: Trace Reasoning and Agentic Issue Localization
paper_summary: The increasing adoption of agentic workflows across diverse domains brings a
  critical need to scalably and systematically evaluate the complex traces these
  systems generate. Current evaluation methods depend on manual, domain-specific
  human analysis of lengthy workflow traces - an approach that does not scale
  with the growing complexity and volume of agentic outputs. Error analysis in
  these settings is further complicated by the interplay of external tool outputs
  and language model reasoning, making it more challenging than traditional
  software debugging. In this work, we (1) articulate the need for robust and
  dynamic evaluation methods for agentic workflow traces, (2) introduce a formal
  taxonomy of error types encountered in agentic systems, and (3) present a set
  of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and
  grounded in established agentic benchmarks. To ensure ecological validity, we
  curate traces from both single and multi-agent systems, focusing on real-world
  applications such as software engineering and open-world information retrieval.
  Our evaluations reveal that modern long context LLMs perform poorly at trace
  debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our
  dataset and code are made publicly available to support and accelerate future
  research in scalable evaluation for agentic workflows.
paper_ai_summary: Modern long context LLMs struggle with trace debugging of agentic workflows, as evidenced by a new dataset of human-annotated traces.
paper_ai_keywords: LLMs Gemini-2.5-pro trace debugging agentic workflows TRAIL error types single-agent systems multi-agent systems software engineering open-world information retrieval