MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
<|reference_start|>MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering: We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.<|reference_end|>
@article{chan2024mle-bench:,
    title={MLE-bench: Evaluating Machine Learning Agents on Machine Learning
  Engineering},
    author={Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane
  Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal
  Patwardhan, Lilian Weng, Aleksander Mk{a}dry},
    journal={arXiv preprint arXiv:2410.07095},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07095},
    primaryClass={cs.CL}
}