DA-Code: Agent Data Science Code Generation Benchmark for Large Language
  Models
We introduce DA-Code, a code generation benchmark specifically designed to
assess LLMs on agent-based data science tasks. This benchmark features three
core elements: First, the tasks within DA-Code are inherently challenging,
setting them apart from traditional code generation tasks and demanding
advanced coding skills in grounding and planning. Second, examples in DA-Code
are all based on real and diverse data, covering a wide range of complex data
wrangling and analytics tasks. Third, to solve the tasks, the models must
utilize complex data science programming languages, to perform intricate data
processing and derive the answers. We set up the benchmark in a controllable
and executable environment that aligns with real-world data analysis scenarios
and is scalable. The annotators meticulously design the evaluation suite to
ensure the accuracy and robustness of the evaluation. We develop the DA-Agent
baseline. Experiments show that although the baseline performs better than
other existing frameworks, using the current best LLMs achieves only 30.5%
accuracy, leaving ample room for improvement. We release our benchmark at
https://da-code-bench.github.io.
670cdca490a771a1f03eec6c
Yiming Huang
670cdca490a771a1f03eec6d
Jianwen Luo
admin_assigned
2024-10-14T10:51:58.359Z
66adf5cc0c6056d9f4dc308f
/avatars/7689ba5538f6bbf235084d418b3065c1.svg
Jianwen Luo
user
Jianwen2003
670cdca490a771a1f03eec6e
Yan Yu
670cdca490a771a1f03eec6f
Yitong Zhang
670cdca490a771a1f03eec70
Fangyu Lei
admin_assigned
2024-10-14T10:52:29.944Z
64104b467a15af878ae6695d
/avatars/407983918c12411e5ed636bf7435522b.svg
Fangyu Lei
user
FangyuLei
670cdca490a771a1f03eec71
Yifan Wei
admin_assigned
2024-10-14T10:52:36.300Z
665c3d590e92f92b0ee233ad
/avatars/ee4bbf2872ccd5625196966e235f40f7.svg
Yifan Wei
user
bjEdward
670cdca490a771a1f03eec72
Shizhu He
670cdca490a771a1f03eec73
Lifu Huang
670cdca490a771a1f03eec74
Xiao Liu
claimed_verified
2024-10-14T10:47:40.156Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
670cdca490a771a1f03eec75
Jun Zhao
670cdca490a771a1f03eec76
Kang Liu
DA-Code: Agent Data Science Code Generation Benchmark for Large Language
  Models
We introduce DA-Code, a code generation benchmark specifically designed to
assess LLMs on agent-based data science tasks. This benchmark features three
core elements: First, the tasks within DA-Code are inherently challenging,
setting them apart from traditional code generation tasks and demanding
advanced coding skills in grounding and planning. Second, examples in DA-Code
are all based on real and diverse data, covering a wide range of complex data
wrangling and analytics tasks. Third, to solve the tasks, the models must
utilize complex data science programming languages, to perform intricate data
processing and derive the answers. We set up the benchmark in a controllable
and executable environment that aligns with real-world data analysis scenarios
and is scalable. The annotators meticulously design the evaluation suite to
ensure the accuracy and robustness of the evaluation. We develop the DA-Agent
baseline. Experiments show that although the baseline performs better than
other existing frameworks, using the current best LLMs achieves only 30.5%
accuracy, leaving ample room for improvement. We release our benchmark at
https://da-code-bench.github.io.
A code generation benchmark, DA-Code, evaluates LLMs on agent-based data science tasks using real-world, diverse datasets and complex data science programming languages.
LLMs DA-Code agent-based data science tasks data wrangling analytics tasks data science programming languages DA-Agent baseline