title: Can Models Help Us Create Better Models? Evaluating LLMs as Data
    Scientists
summary: We present a benchmark for large language models designed to tackle one of
  the most knowledge-intensive tasks in data science: writing feature engineering
  code, which requires domain knowledge in addition to a deep understanding of
  the underlying problem and data structure. The model is provided with a dataset
  description in a prompt and asked to generate code transforming it. The
  evaluation score is derived from the improvement achieved by an XGBoost model
  fit on the modified dataset compared to the original data. By an extensive
  evaluation of state-of-the-art models and comparison to well-established
  benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
  efficiently assess the broad capabilities of LLMs, in contrast to the existing
  methods.
paper_authors._id: 67242c0223ae9645e6021fbe
paper_authors.name: Michał Pietruszka
paper_authors._id: 67242c0223ae9645e6021fbf
paper_authors.name: Łukasz Borchmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:38.885Z
paper_authors._id: 600b381d3cc3b87db94bc0ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/600b381d3cc3b87db94bc0ce/I3xpr4gzcG1uXawXBpWpD.jpeg
paper_authors.fullname: Łukasz Borchmann
paper_authors.type: user
paper_authors.user: Borchmann
paper_authors._id: 67242c0223ae9645e6021fc0
paper_authors.name: Aleksander Jędrosz
paper_authors._id: 67242c0223ae9645e6021fc1
paper_authors.name: Paweł Morawiecki
paper_title: Can Models Help Us Create Better Models? Evaluating LLMs as Data
    Scientists
paper_summary: We present a benchmark for large language models designed to tackle one of
  the most knowledge-intensive tasks in data science: writing feature engineering
  code, which requires domain knowledge in addition to a deep understanding of
  the underlying problem and data structure. The model is provided with a dataset
  description in a prompt and asked to generate code transforming it. The
  evaluation score is derived from the improvement achieved by an XGBoost model
  fit on the modified dataset compared to the original data. By an extensive
  evaluation of state-of-the-art models and comparison to well-established
  benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
  efficiently assess the broad capabilities of LLMs, in contrast to the existing
  methods.
paper_ai_keywords: large language models feature engineering XGBoost model FeatEng