title: DSBench: How Far Are Data Science Agents to Becoming Data Science
    Experts?
summary: Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have
  demonstrated impressive language/vision reasoning abilities, igniting the
  recent trend of building agents for targeted applications such as shopping
  assistants or AI software engineers. Recently, many data science benchmarks
  have been proposed to investigate their performance in the data science domain.
  However, existing data science benchmarks still fall short when compared to
  real-world data science applications due to their simplified settings. To
  bridge this gap, we introduce DSBench, a comprehensive benchmark designed to
  evaluate data science agents with realistic tasks. This benchmark includes 466
  data analysis tasks and 74 data modeling tasks, sourced from Eloquence and
  Kaggle competitions. DSBench offers a realistic setting by encompassing long
  contexts, multimodal task backgrounds, reasoning with large data files and
  multi-table structures, and performing end-to-end data modeling tasks. Our
  evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle
  with most tasks, with the best agent solving only 34.12% of data analysis tasks
  and achieving a 34.74% Relative Performance Gap (RPG). These findings
  underscore the need for further advancements in developing more practical,
  intelligent, and autonomous data science agents.
paper_authors._id: 66e4032d8bcd43ff6cdfd7cf
paper_authors.name: Liqiang Jing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:02:56.727Z
paper_authors._id: 65318cb0cdc25730d4ad1e42
paper_authors.avatarUrl: /avatars/996eb672d532532bd97b685cc9864205.svg
paper_authors.fullname: Liqiang Jing
paper_authors.type: user
paper_authors.user: liqiang888
paper_authors._id: 66e4032d8bcd43ff6cdfd7d0
paper_authors.name: Zhehui Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:04:59.361Z
paper_authors._id: 6262683b08481d1d75ebbaea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668367637437-6262683b08481d1d75ebbaea.jpeg
paper_authors.fullname: Zhehui Huang
paper_authors.type: user
paper_authors.user: Zhehui-Huang
paper_authors._id: 66e4032d8bcd43ff6cdfd7d1
paper_authors.name: Xiaoyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:03:00.221Z
paper_authors._id: 657cd228138b7e391444a65d
paper_authors.avatarUrl: /avatars/c7c984ae483144fab627aa2c54d91d0f.svg
paper_authors.fullname: Xiaoyang Wang
paper_authors.type: user
paper_authors.user: xywang1
paper_authors._id: 66e4032d8bcd43ff6cdfd7d2
paper_authors.name: Wenlin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:05:11.695Z
paper_authors._id: 634f18e4aae4bde2c8e2adca
paper_authors.avatarUrl: /avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
paper_authors.fullname: Wenlin Yao
paper_authors.type: user
paper_authors.user: wenlinyao
paper_authors._id: 66e4032d8bcd43ff6cdfd7d3
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:05:18.254Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 66e4032d8bcd43ff6cdfd7d4
paper_authors.name: Kaixin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:05:24.397Z
paper_authors._id: 64ae4f6280f308a395fd7c19
paper_authors.avatarUrl: /avatars/5f1330f8187cd5e66aa517303659f110.svg
paper_authors.fullname: Kaixin Ma
paper_authors.type: user
paper_authors.user: kaixinm
paper_authors._id: 66e4032d8bcd43ff6cdfd7d5
paper_authors.name: Hongming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-16T07:05:30.586Z
paper_authors._id: 64ed478bec06efeb03034933
paper_authors.avatarUrl: /avatars/cd7dc3165831e90cb36d39d41c3c8157.svg
paper_authors.fullname: Hongming Zhang
paper_authors.type: user
paper_authors.user: Hongming98
paper_authors._id: 66e4032d8bcd43ff6cdfd7d6
paper_authors.name: Xinya Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:02:58.554Z
paper_authors._id: 66c4154fb83a7e94d588ade3
paper_authors.avatarUrl: /avatars/2f1c8a3feb81be628e36eaa198450704.svg
paper_authors.fullname: Xinya Du
paper_authors.type: user
paper_authors.user: xinyadu
paper_authors._id: 66e4032d8bcd43ff6cdfd7d7
paper_authors.name: Dong Yu
paper_title: DSBench: How Far Are Data Science Agents to Becoming Data Science
    Experts?
paper_summary: Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have
  demonstrated impressive language/vision reasoning abilities, igniting the
  recent trend of building agents for targeted applications such as shopping
  assistants or AI software engineers. Recently, many data science benchmarks
  have been proposed to investigate their performance in the data science domain.
  However, existing data science benchmarks still fall short when compared to
  real-world data science applications due to their simplified settings. To
  bridge this gap, we introduce DSBench, a comprehensive benchmark designed to
  evaluate data science agents with realistic tasks. This benchmark includes 466
  data analysis tasks and 74 data modeling tasks, sourced from Eloquence and
  Kaggle competitions. DSBench offers a realistic setting by encompassing long
  contexts, multimodal task backgrounds, reasoning with large data files and
  multi-table structures, and performing end-to-end data modeling tasks. Our
  evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle
  with most tasks, with the best agent solving only 34.12% of data analysis tasks
  and achieving a 34.74% Relative Performance Gap (RPG). These findings
  underscore the need for further advancements in developing more practical,
  intelligent, and autonomous data science agents.
paper_ai_summary: DSBench evaluates large language and vision-language models on realistic data science tasks, revealing significant performance gaps that indicate the need for improved autonomous agents.
paper_ai_keywords: Large Language Models Large Vision-Language Models data science benchmarks data analysis tasks data modeling tasks Eloquence Kaggle long contexts multimodal reasoning large data files multi-table structures Relative Performance Gap data science agents