title: Spider2-V: How Far Are Multimodal Agents From Automating Data Science
    and Engineering Workflows?
summary: Data science and engineering workflows often span multiple stages, from
  warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
  vision language models (VLMs) advance in multimodal understanding and code
  generation, VLM-based agents could potentially automate these workflows by
  generating SQL queries, Python code, and GUI operations. This automation can
  improve the productivity of experts while democratizing access to large-scale
  data analysis. In this paper, we introduce Spider2-V, the first multimodal
  agent benchmark focusing on professional data science and engineering
  workflows, featuring 494 real-world tasks in authentic computer environments
  and incorporating 20 enterprise-level professional applications. These tasks,
  derived from real-world use cases, evaluate the ability of a multimodal agent
  to perform data-related tasks by writing code and managing the GUI in
  enterprise data software systems. To balance realistic simulation with
  evaluation simplicity, we devote significant effort to developing automatic
  configurations for task setup and carefully crafting evaluation metrics for
  each task. Furthermore, we supplement multimodal agents with comprehensive
  documents of these enterprise data software systems. Our empirical evaluation
  reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
  automate full data workflows (14.0% success). Even with step-by-step guidance,
  these agents still underperform in tasks that require fine-grained,
  knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
  workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
  multimodal agents to transform the automation of data science and engineering
  workflow. Our code and data are available at https://spider2-v.github.io.
paper_authors._id: 6696c1d3f97e6a8a930f2080
paper_authors.name: Ruisheng Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:48:32.490Z
paper_authors._id: 5fbcf0d28f35b82700205fd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fbcf0d28f35b82700205fd7/KZPwcaIymBAPs0UHXe5fa.jpeg
paper_authors.fullname: Ruisheng Cao
paper_authors.type: user
paper_authors.user: rshcao
paper_authors._id: 6696c1d3f97e6a8a930f2081
paper_authors.name: Fangyu Lei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:58:26.657Z
paper_authors._id: 64104b467a15af878ae6695d
paper_authors.avatarUrl: /avatars/407983918c12411e5ed636bf7435522b.svg
paper_authors.fullname: Fangyu Lei
paper_authors.type: user
paper_authors.user: FangyuLei
paper_authors._id: 6696c1d3f97e6a8a930f2082
paper_authors.name: Haoyuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:30:05.532Z
paper_authors._id: 653927c90dfecb4b26bc5885
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653927c90dfecb4b26bc5885/lM7d75M0CzVrb6HVz1dDD.png
paper_authors.fullname: Fred Wu
paper_authors.type: user
paper_authors.user: HappySixHF
paper_authors._id: 6696c1d3f97e6a8a930f2083
paper_authors.name: Jixuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T14:49:04.929Z
paper_authors._id: 6465941d0e6c7618f615675b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
paper_authors.fullname: Jixuan Chen
paper_authors.type: user
paper_authors.user: Mayome
paper_authors._id: 6696c1d3f97e6a8a930f2084
paper_authors.name: Yeqiao Fu
paper_authors._id: 6696c1d3f97e6a8a930f2085
paper_authors.name: Hongcheng Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:07:52.780Z
paper_authors._id: 62728f4f6253fe2068da1021
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
paper_authors.fullname: Hongcheng Gao
paper_authors.type: user
paper_authors.user: HongchengGao
paper_authors._id: 6696c1d3f97e6a8a930f2086
paper_authors.name: Xinzhuang Xiong
paper_authors._id: 6696c1d3f97e6a8a930f2087
paper_authors.name: Hanchong Zhang
paper_authors._id: 6696c1d3f97e6a8a930f2088
paper_authors.name: Yuchen Mao
paper_authors._id: 6696c1d3f97e6a8a930f2089
paper_authors.name: Wenjing Hu
paper_authors._id: 6696c1d3f97e6a8a930f208a
paper_authors.name: Tianbao Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:53.142Z
paper_authors._id: 618767e4238063b4615d042b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
paper_authors.fullname: Tianbao Xie
paper_authors.type: user
paper_authors.user: tianbaoxiexxx
paper_authors._id: 6696c1d3f97e6a8a930f208b
paper_authors.name: Hongshen Xu
paper_authors._id: 6696c1d3f97e6a8a930f208c
paper_authors.name: Danyang Zhang
paper_authors._id: 6696c1d3f97e6a8a930f208d
paper_authors.name: Sida Wang
paper_authors._id: 6696c1d3f97e6a8a930f208e
paper_authors.name: Ruoxi Sun
paper_authors._id: 6696c1d3f97e6a8a930f208f
paper_authors.name: Pengcheng Yin
paper_authors._id: 6696c1d3f97e6a8a930f2090
paper_authors.name: Caiming Xiong
paper_authors._id: 6696c1d3f97e6a8a930f2091
paper_authors.name: Ansong Ni
paper_authors._id: 6696c1d3f97e6a8a930f2092
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:23.223Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 6696c1d3f97e6a8a930f2093
paper_authors.name: Victor Zhong
paper_authors._id: 6696c1d3f97e6a8a930f2094
paper_authors.name: Lu Chen
paper_authors._id: 6696c1d3f97e6a8a930f2095
paper_authors.name: Kai Yu
paper_authors._id: 6696c1d3f97e6a8a930f2096
paper_authors.name: Tao Yu
paper_title: Spider2-V: How Far Are Multimodal Agents From Automating Data Science
    and Engineering Workflows?
paper_summary: Data science and engineering workflows often span multiple stages, from
  warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As
  vision language models (VLMs) advance in multimodal understanding and code
  generation, VLM-based agents could potentially automate these workflows by
  generating SQL queries, Python code, and GUI operations. This automation can
  improve the productivity of experts while democratizing access to large-scale
  data analysis. In this paper, we introduce Spider2-V, the first multimodal
  agent benchmark focusing on professional data science and engineering
  workflows, featuring 494 real-world tasks in authentic computer environments
  and incorporating 20 enterprise-level professional applications. These tasks,
  derived from real-world use cases, evaluate the ability of a multimodal agent
  to perform data-related tasks by writing code and managing the GUI in
  enterprise data software systems. To balance realistic simulation with
  evaluation simplicity, we devote significant effort to developing automatic
  configurations for task setup and carefully crafting evaluation metrics for
  each task. Furthermore, we supplement multimodal agents with comprehensive
  documents of these enterprise data software systems. Our empirical evaluation
  reveals that existing state-of-the-art LLM/VLM-based agents do not reliably
  automate full data workflows (14.0% success). Even with step-by-step guidance,
  these agents still underperform in tasks that require fine-grained,
  knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted
  workspaces (10.6%). We hope that Spider2-V paves the way for autonomous
  multimodal agents to transform the automation of data science and engineering
  workflow. Our code and data are available at https://spider2-v.github.io.
paper_ai_summary: Spider2-V, a multimodal agent benchmark, evaluates the automation of professional data science workflows using SQL, Python, and GUI interactions, revealing limitations in existing state-of-the-art LLM/VLM-based agents.
paper_ai_keywords: vision language models multimodal understanding multimodal agent SQL queries Python code GUI operations data science engineering workflows enterprise data software systems evaluation metrics