title: AGENTIF: Benchmarking Instruction Following of Large Language Models in
    Agentic Scenarios
summary: Large Language Models (LLMs) have demonstrated advanced capabilities in
  real-world agentic applications. Growing research efforts aim to develop
  LLM-based agents to address practical demands, introducing a new challenge:
  agentic scenarios often involve lengthy instructions with complex constraints,
  such as extended system prompts and detailed tool specifications. While
  adherence to such instructions is crucial for agentic applications, whether
  LLMs can reliably follow them remains underexplored. In this paper, we
  introduce AgentIF, the first benchmark for systematically evaluating LLM
  instruction following ability in agentic scenarios. AgentIF features three key
  characteristics: (1) Realistic, constructed from 50 real-world agentic
  applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words.
  (3) Complex, averaging 11.9 constraints per instruction, covering diverse
  constraint types, such as tool specifications and condition constraints. To
  construct AgentIF, we collect 707 human-annotated instructions across 50
  agentic tasks from industrial application agents and open-source agentic
  systems. For each instruction, we annotate the associated constraints and
  corresponding evaluation metrics, including code-based evaluation, LLM-based
  evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically
  evaluate existing advanced LLMs. We observe that current models generally
  perform poorly, especially in handling complex constraint structures and tool
  specifications. We further conduct error analysis and analytical experiments on
  instruction length and meta constraints, providing some findings about the
  failure modes of existing LLMs. We have released the code and data to
  facilitate future research.
paper_authors._id: 683023848d2fd6fc7ce9b99a
paper_authors.name: Yunjia Qi
paper_authors._id: 683023848d2fd6fc7ce9b99b
paper_authors.name: Hao Peng
paper_authors._id: 683023848d2fd6fc7ce9b99c
paper_authors.name: Xiaozhi Wang
paper_authors._id: 683023848d2fd6fc7ce9b99d
paper_authors.name: Amy Xin
paper_authors._id: 683023848d2fd6fc7ce9b99e
paper_authors.name: Youfeng Liu
paper_authors._id: 683023848d2fd6fc7ce9b99f
paper_authors.name: Bin Xu
paper_authors._id: 683023848d2fd6fc7ce9b9a0
paper_authors.name: Lei Hou
paper_authors._id: 683023848d2fd6fc7ce9b9a1
paper_authors.name: Juanzi Li
paper_title: AGENTIF: Benchmarking Instruction Following of Large Language Models in
    Agentic Scenarios
paper_summary: Large Language Models (LLMs) have demonstrated advanced capabilities in
  real-world agentic applications. Growing research efforts aim to develop
  LLM-based agents to address practical demands, introducing a new challenge:
  agentic scenarios often involve lengthy instructions with complex constraints,
  such as extended system prompts and detailed tool specifications. While
  adherence to such instructions is crucial for agentic applications, whether
  LLMs can reliably follow them remains underexplored. In this paper, we
  introduce AgentIF, the first benchmark for systematically evaluating LLM
  instruction following ability in agentic scenarios. AgentIF features three key
  characteristics: (1) Realistic, constructed from 50 real-world agentic
  applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words.
  (3) Complex, averaging 11.9 constraints per instruction, covering diverse
  constraint types, such as tool specifications and condition constraints. To
  construct AgentIF, we collect 707 human-annotated instructions across 50
  agentic tasks from industrial application agents and open-source agentic
  systems. For each instruction, we annotate the associated constraints and
  corresponding evaluation metrics, including code-based evaluation, LLM-based
  evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically
  evaluate existing advanced LLMs. We observe that current models generally
  perform poorly, especially in handling complex constraint structures and tool
  specifications. We further conduct error analysis and analytical experiments on
  instruction length and meta constraints, providing some findings about the
  failure modes of existing LLMs. We have released the code and data to
  facilitate future research.
paper_ai_summary: A new benchmark, AgentIF, evaluates Large Language Models' ability to follow complex instructions in realistic agentic scenarios, revealing performance limitations in handling constraints and tool specifications.
paper_ai_keywords: AgentIF Large Language Models (LLMs) agentic applications system prompts tool specifications instruction following constraint structures error analysis