title: Benchmarking Agentic Workflow Generation
summary: Large Language Models (LLMs), with their exceptional ability to handle a wide
  range of tasks, have driven significant advancements in tackling reasoning and
  planning tasks, wherein decomposing complex problems into executable workflows
  is a crucial step in this process. Existing workflow evaluation frameworks
  either focus solely on holistic performance or suffer from limitations such as
  restricted scenario coverage, simplistic workflow structures, and lax
  evaluation standards. To this end, we introduce WorFBench, a unified workflow
  generation benchmark with multi-faceted scenarios and intricate graph workflow
  structures. Additionally, we present WorFEval, a systemic evaluation protocol
  utilizing subsequence and subgraph matching algorithms to accurately quantify
  the LLM agent's workflow generation capabilities. Through comprehensive
  evaluations across different types of LLMs, we discover distinct gaps between
  the sequence planning capabilities and graph planning capabilities of LLM
  agents, with even GPT-4 exhibiting a gap of around 15%. We also train two
  open-source models and evaluate their generalization abilities on held-out
  tasks. Furthermore, we observe that the generated workflows can enhance
  downstream tasks, enabling them to achieve superior performance with less time
  during inference. Code and dataset will be available at
  https://github.com/zjunlp/WorFBench.
paper_authors._id: 6708dea5d5c3f877d9be613d
paper_authors.name: Shuofei Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:33:25.748Z
paper_authors._id: 6447800f30fa4ecb85ddad80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6447800f30fa4ecb85ddad80/NsmXIaMsWctmTNA7tFVkX.jpeg
paper_authors.fullname: Shuofei Qiao
paper_authors.type: user
paper_authors.user: GoooDte
paper_authors._id: 6708dea5d5c3f877d9be613e
paper_authors.name: Runnan Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:33:31.931Z
paper_authors._id: 63d32cd7b734eaa4d4fa410b
paper_authors.avatarUrl: /avatars/68acb80f62bc6493e1ad26506999b6c4.svg
paper_authors.fullname: Runnan Fang
paper_authors.type: user
paper_authors.user: Runnaning
paper_authors._id: 6708dea5d5c3f877d9be613f
paper_authors.name: Zhisong Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:33:37.295Z
paper_authors._id: 65535b54140fc44a74d43635
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MIrD8OzDKF2aI38i7ZPjR.jpeg
paper_authors.fullname: Zhisong Qiu
paper_authors.type: user
paper_authors.user: consultantQ
paper_authors._id: 6708dea5d5c3f877d9be6140
paper_authors.name: Xiaobin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:33:42.698Z
paper_authors._id: 654e24cb12c9597b2099385e
paper_authors.avatarUrl: /avatars/c7505e894fe0b5e3fc1db86d5717da8f.svg
paper_authors.fullname: XIAOBIN WANG
paper_authors.type: user
paper_authors.user: xiaoyuehanbin
paper_authors._id: 6708dea5d5c3f877d9be6141
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T10:08:42.030Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_authors._id: 6708dea5d5c3f877d9be6142
paper_authors.name: Yong Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-20T09:31:08.525Z
paper_authors._id: 678ddd806aa55e76bfffb953
paper_authors.avatarUrl: /avatars/f447936c286a6a2d2874a760210b2f17.svg
paper_authors.fullname: Yong Jiang
paper_authors.type: user
paper_authors.user: yongjiangNLP
paper_authors._id: 6708dea5d5c3f877d9be6143
paper_authors.name: Pengjun Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:33:56.901Z
paper_authors._id: 63a091e42fabbbb89991f5ce
paper_authors.avatarUrl: /avatars/d55485b06461764c36c9edf9d6e8892c.svg
paper_authors.fullname: pengjun xie
paper_authors.type: user
paper_authors.user: xpjandy
paper_authors._id: 6708dea5d5c3f877d9be6144
paper_authors.name: Fei Huang
paper_authors._id: 6708dea5d5c3f877d9be6145
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:34:31.686Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_title: Benchmarking Agentic Workflow Generation
paper_summary: Large Language Models (LLMs), with their exceptional ability to handle a wide
  range of tasks, have driven significant advancements in tackling reasoning and
  planning tasks, wherein decomposing complex problems into executable workflows
  is a crucial step in this process. Existing workflow evaluation frameworks
  either focus solely on holistic performance or suffer from limitations such as
  restricted scenario coverage, simplistic workflow structures, and lax
  evaluation standards. To this end, we introduce WorFBench, a unified workflow
  generation benchmark with multi-faceted scenarios and intricate graph workflow
  structures. Additionally, we present WorFEval, a systemic evaluation protocol
  utilizing subsequence and subgraph matching algorithms to accurately quantify
  the LLM agent's workflow generation capabilities. Through comprehensive
  evaluations across different types of LLMs, we discover distinct gaps between
  the sequence planning capabilities and graph planning capabilities of LLM
  agents, with even GPT-4 exhibiting a gap of around 15%. We also train two
  open-source models and evaluate their generalization abilities on held-out
  tasks. Furthermore, we observe that the generated workflows can enhance
  downstream tasks, enabling them to achieve superior performance with less time
  during inference. Code and dataset will be available at
  https://github.com/zjunlp/WorFBench.
paper_ai_keywords: WorFBench WorFEval subsequence and subgraph matching algorithms workflow generation LLM agent sequence planning graph planning multi-faceted scenarios intricate graph workflow structures