title: MLE-bench: Evaluating Machine Learning Agents on Machine Learning
    Engineering
summary: We introduce MLE-bench, a benchmark for measuring how well AI agents perform
  at machine learning engineering. To this end, we curate 75 ML
  engineering-related competitions from Kaggle, creating a diverse set of
  challenging tasks that test real-world ML engineering skills such as training
  models, preparing datasets, and running experiments. We establish human
  baselines for each competition using Kaggle's publicly available leaderboards.
  We use open-source agent scaffolds to evaluate several frontier language models
  on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
  with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
  16.9% of competitions. In addition to our main results, we investigate various
  forms of resource scaling for AI agents and the impact of contamination from
  pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
  to facilitate future research in understanding the ML engineering capabilities
  of AI agents.
paper_authors._id: 67077d44503533522daed22f
paper_authors.name: Jun Shern Chan
paper_authors._id: 67077d44503533522daed230
paper_authors.name: Neil Chowdhury
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:44:54.162Z
paper_authors._id: 63ec754359a46fd5f4b81a2f
paper_authors.avatarUrl: /avatars/aaa4a63b25bd03c366f855427ba9cad4.svg
paper_authors.fullname: Neil Chowdhury
paper_authors.type: user
paper_authors.user: nch0w
paper_authors._id: 67077d44503533522daed231
paper_authors.name: Oliver Jaffe
paper_authors._id: 67077d44503533522daed232
paper_authors.name: James Aung
paper_authors._id: 67077d44503533522daed233
paper_authors.name: Dane Sherburn
paper_authors._id: 67077d44503533522daed234
paper_authors.name: Evan Mays
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:44:33.433Z
paper_authors._id: 636d9548da13cd91f922e0d0
paper_authors.avatarUrl: /avatars/488f6a4b12f20809d3a4c3d40ae82b53.svg
paper_authors.fullname: Evan Mays
paper_authors.type: user
paper_authors.user: evanmays
paper_authors._id: 67077d44503533522daed235
paper_authors.name: Giulio Starace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:44:26.853Z
paper_authors._id: 627b8bc83974b0ed6b28db67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665158475637-627b8bc83974b0ed6b28db67.jpeg
paper_authors.fullname: Giulio Starace
paper_authors.type: user
paper_authors.user: thesofakillers
paper_authors._id: 67077d44503533522daed236
paper_authors.name: Kevin Liu
paper_authors._id: 67077d44503533522daed237
paper_authors.name: Leon Maksin
paper_authors._id: 67077d44503533522daed238
paper_authors.name: Tejal Patwardhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:44:12.948Z
paper_authors._id: 64d4f504887f55fb6eedec74
paper_authors.avatarUrl: /avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
paper_authors.fullname: Tejal Patwardhan
paper_authors.type: user
paper_authors.user: tejalp
paper_authors._id: 67077d44503533522daed239
paper_authors.name: Lilian Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:44:07.283Z
paper_authors._id: 5fb2dc322a79d831217aa0cb
paper_authors.avatarUrl: /avatars/c4bdc2f907443e34d75116730105510c.svg
paper_authors.fullname: Lilian Weng
paper_authors.type: user
paper_authors.user: lilianweng
paper_authors._id: 67077d44503533522daed23a
paper_authors.name: Aleksander Mądry
paper_title: MLE-bench: Evaluating Machine Learning Agents on Machine Learning
    Engineering
paper_summary: We introduce MLE-bench, a benchmark for measuring how well AI agents perform
  at machine learning engineering. To this end, we curate 75 ML
  engineering-related competitions from Kaggle, creating a diverse set of
  challenging tasks that test real-world ML engineering skills such as training
  models, preparing datasets, and running experiments. We establish human
  baselines for each competition using Kaggle's publicly available leaderboards.
  We use open-source agent scaffolds to evaluate several frontier language models
  on our benchmark, finding that the best-performing setup--OpenAI's o1-preview
  with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in
  16.9% of competitions. In addition to our main results, we investigate various
  forms of resource scaling for AI agents and the impact of contamination from
  pre-training. We open-source our benchmark code (github.com/openai/mle-bench/)
  to facilitate future research in understanding the ML engineering capabilities
  of AI agents.
paper_ai_keywords: MLE-bench benchmark AI agents machine learning engineering ML engineering-related competitions Kaggle training models preparing datasets running experiments human baselines leaderboards OpenAI's o1-preview AIDE scaffolding bronze medal resource language models resource scaling contamination from pre-training open-source benchmark code