DSBench: How Far Are Data Science Agents to Becoming Data Science
  Experts?
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have
demonstrated impressive language/vision reasoning abilities, igniting the
recent trend of building agents for targeted applications such as shopping
assistants or AI software engineers. Recently, many data science benchmarks
have been proposed to investigate their performance in the data science domain.
However, existing data science benchmarks still fall short when compared to
real-world data science applications due to their simplified settings. To
bridge this gap, we introduce DSBench, a comprehensive benchmark designed to
evaluate data science agents with realistic tasks. This benchmark includes 466
data analysis tasks and 74 data modeling tasks, sourced from Eloquence and
Kaggle competitions. DSBench offers a realistic setting by encompassing long
contexts, multimodal task backgrounds, reasoning with large data files and
multi-table structures, and performing end-to-end data modeling tasks. Our
evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle
with most tasks, with the best agent solving only 34.12% of data analysis tasks
and achieving a 34.74% Relative Performance Gap (RPG). These findings
underscore the need for further advancements in developing more practical,
intelligent, and autonomous data science agents.
66e4032d8bcd43ff6cdfd7cf
Liqiang Jing
claimed_verified
2024-09-16T07:02:56.727Z
65318cb0cdc25730d4ad1e42
/avatars/996eb672d532532bd97b685cc9864205.svg
Liqiang Jing
user
liqiang888
66e4032d8bcd43ff6cdfd7d0
Zhehui Huang
admin_assigned
2024-09-16T07:04:59.361Z
6262683b08481d1d75ebbaea
https://cdn-avatars.huggingface.co/v1/production/uploads/1668367637437-6262683b08481d1d75ebbaea.jpeg
Zhehui Huang
user
Zhehui-Huang
66e4032d8bcd43ff6cdfd7d1
Xiaoyang Wang
claimed_verified
2024-09-16T07:03:00.221Z
657cd228138b7e391444a65d
/avatars/c7c984ae483144fab627aa2c54d91d0f.svg
Xiaoyang Wang
user
xywang1
66e4032d8bcd43ff6cdfd7d2
Wenlin Yao
admin_assigned
2024-09-16T07:05:11.695Z
634f18e4aae4bde2c8e2adca
/avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
Wenlin Yao
user
wenlinyao
66e4032d8bcd43ff6cdfd7d3
Wenhao Yu
admin_assigned
2024-09-16T07:05:18.254Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
66e4032d8bcd43ff6cdfd7d4
Kaixin Ma
admin_assigned
2024-09-16T07:05:24.397Z
64ae4f6280f308a395fd7c19
/avatars/5f1330f8187cd5e66aa517303659f110.svg
Kaixin Ma
user
kaixinm
66e4032d8bcd43ff6cdfd7d5
Hongming Zhang
admin_assigned
2024-09-16T07:05:30.586Z
64ed478bec06efeb03034933
/avatars/cd7dc3165831e90cb36d39d41c3c8157.svg
Hongming Zhang
user
Hongming98
66e4032d8bcd43ff6cdfd7d6
Xinya Du
claimed_verified
2024-09-16T07:02:58.554Z
66c4154fb83a7e94d588ade3
/avatars/2f1c8a3feb81be628e36eaa198450704.svg
Xinya Du
user
xinyadu
66e4032d8bcd43ff6cdfd7d7
Dong Yu
DSBench: How Far Are Data Science Agents to Becoming Data Science
  Experts?
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have
demonstrated impressive language/vision reasoning abilities, igniting the
recent trend of building agents for targeted applications such as shopping
assistants or AI software engineers. Recently, many data science benchmarks
have been proposed to investigate their performance in the data science domain.
However, existing data science benchmarks still fall short when compared to
real-world data science applications due to their simplified settings. To
bridge this gap, we introduce DSBench, a comprehensive benchmark designed to
evaluate data science agents with realistic tasks. This benchmark includes 466
data analysis tasks and 74 data modeling tasks, sourced from Eloquence and
Kaggle competitions. DSBench offers a realistic setting by encompassing long
contexts, multimodal task backgrounds, reasoning with large data files and
multi-table structures, and performing end-to-end data modeling tasks. Our
evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle
with most tasks, with the best agent solving only 34.12% of data analysis tasks
and achieving a 34.74% Relative Performance Gap (RPG). These findings
underscore the need for further advancements in developing more practical,
intelligent, and autonomous data science agents.
DSBench evaluates large language and vision-language models on realistic data science tasks, revealing significant performance gaps that indicate the need for improved autonomous agents.
Large Language Models Large Vision-Language Models data science benchmarks data analysis tasks data modeling tasks Eloquence Kaggle long contexts multimodal reasoning large data files multi-table structures Relative Performance Gap data science agents