title: DA-Code: Agent Data Science Code Generation Benchmark for Large Language
    Models
summary: We introduce DA-Code, a code generation benchmark specifically designed to
  assess LLMs on agent-based data science tasks. This benchmark features three
  core elements: First, the tasks within DA-Code are inherently challenging,
  setting them apart from traditional code generation tasks and demanding
  advanced coding skills in grounding and planning. Second, examples in DA-Code
  are all based on real and diverse data, covering a wide range of complex data
  wrangling and analytics tasks. Third, to solve the tasks, the models must
  utilize complex data science programming languages, to perform intricate data
  processing and derive the answers. We set up the benchmark in a controllable
  and executable environment that aligns with real-world data analysis scenarios
  and is scalable. The annotators meticulously design the evaluation suite to
  ensure the accuracy and robustness of the evaluation. We develop the DA-Agent
  baseline. Experiments show that although the baseline performs better than
  other existing frameworks, using the current best LLMs achieves only 30.5%
  accuracy, leaving ample room for improvement. We release our benchmark at
  https://da-code-bench.github.io.
paper_authors._id: 670cdca490a771a1f03eec6c
paper_authors.name: Yiming Huang
paper_authors._id: 670cdca490a771a1f03eec6d
paper_authors.name: Jianwen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T10:51:58.359Z
paper_authors._id: 66adf5cc0c6056d9f4dc308f
paper_authors.avatarUrl: /avatars/7689ba5538f6bbf235084d418b3065c1.svg
paper_authors.fullname: Jianwen Luo
paper_authors.type: user
paper_authors.user: Jianwen2003
paper_authors._id: 670cdca490a771a1f03eec6e
paper_authors.name: Yan Yu
paper_authors._id: 670cdca490a771a1f03eec6f
paper_authors.name: Yitong Zhang
paper_authors._id: 670cdca490a771a1f03eec70
paper_authors.name: Fangyu Lei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T10:52:29.944Z
paper_authors._id: 64104b467a15af878ae6695d
paper_authors.avatarUrl: /avatars/407983918c12411e5ed636bf7435522b.svg
paper_authors.fullname: Fangyu Lei
paper_authors.type: user
paper_authors.user: FangyuLei
paper_authors._id: 670cdca490a771a1f03eec71
paper_authors.name: Yifan Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T10:52:36.300Z
paper_authors._id: 665c3d590e92f92b0ee233ad
paper_authors.avatarUrl: /avatars/ee4bbf2872ccd5625196966e235f40f7.svg
paper_authors.fullname: Yifan Wei
paper_authors.type: user
paper_authors.user: bjEdward
paper_authors._id: 670cdca490a771a1f03eec72
paper_authors.name: Shizhu He
paper_authors._id: 670cdca490a771a1f03eec73
paper_authors.name: Lifu Huang
paper_authors._id: 670cdca490a771a1f03eec74
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:47:40.156Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 670cdca490a771a1f03eec75
paper_authors.name: Jun Zhao
paper_authors._id: 670cdca490a771a1f03eec76
paper_authors.name: Kang Liu
paper_title: DA-Code: Agent Data Science Code Generation Benchmark for Large Language
    Models
paper_summary: We introduce DA-Code, a code generation benchmark specifically designed to
  assess LLMs on agent-based data science tasks. This benchmark features three
  core elements: First, the tasks within DA-Code are inherently challenging,
  setting them apart from traditional code generation tasks and demanding
  advanced coding skills in grounding and planning. Second, examples in DA-Code
  are all based on real and diverse data, covering a wide range of complex data
  wrangling and analytics tasks. Third, to solve the tasks, the models must
  utilize complex data science programming languages, to perform intricate data
  processing and derive the answers. We set up the benchmark in a controllable
  and executable environment that aligns with real-world data analysis scenarios
  and is scalable. The annotators meticulously design the evaluation suite to
  ensure the accuracy and robustness of the evaluation. We develop the DA-Agent
  baseline. Experiments show that although the baseline performs better than
  other existing frameworks, using the current best LLMs achieves only 30.5%
  accuracy, leaving ample room for improvement. We release our benchmark at
  https://da-code-bench.github.io.
paper_ai_summary: A code generation benchmark, DA-Code, evaluates LLMs on agent-based data science tasks using real-world, diverse datasets and complex data science programming languages.
paper_ai_keywords: LLMs DA-Code agent-based data science tasks data wrangling analytics tasks data science programming languages DA-Agent baseline