title: MLGym: A New Framework and Benchmark for Advancing AI Research Agents
summary: We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for
  evaluating and developing LLM agents on AI research tasks. This is the first
  Gym environment for machine learning (ML) tasks, enabling research on
  reinforcement learning (RL) algorithms for training such agents. MLGym-bench
  consists of 13 diverse and open-ended AI research tasks from diverse domains
  such as computer vision, natural language processing, reinforcement learning,
  and game theory. Solving these tasks requires real-world AI research skills
  such as generating new ideas and hypotheses, creating and processing data,
  implementing ML methods, training models, running experiments, analyzing the
  results, and iterating through this process to improve on a given task. We
  evaluate a number of frontier large language models (LLMs) on our benchmarks
  such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5
  Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate
  models or agents, generate synthetic data at scale, as well as develop new
  learning algorithms for training agents on AI research tasks. We find that
  current frontier models can improve on the given baselines, usually by finding
  better hyperparameters, but do not generate novel hypotheses, algorithms,
  architectures, or substantial improvements. We open-source our framework and
  benchmark to facilitate future research in advancing the AI research
  capabilities of LLM agents.
paper_authors._id: 67b7ee1dfedfe971271dcca0
paper_authors.name: Deepak Nathani
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-21T07:20:46.836Z
paper_authors._id: 6114c9fae7a2566ae7d1a1a7
paper_authors.avatarUrl: /avatars/c71ab1850322fcf5ef239cb8d31cb137.svg
paper_authors.fullname: Deepak Nathani
paper_authors.type: user
paper_authors.user: dnathani
paper_authors._id: 67b7ee1dfedfe971271dcca1
paper_authors.name: Lovish Madaan
paper_authors._id: 67b7ee1dfedfe971271dcca2
paper_authors.name: Nicholas Roberts
paper_authors._id: 67b7ee1dfedfe971271dcca3
paper_authors.name: Nikolay Bashlykov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:42:58.738Z
paper_authors._id: 633476edc3cb9eda9328e556
paper_authors.avatarUrl: /avatars/a127e270a606c18623fe00cd723313f6.svg
paper_authors.fullname: Nikolay B
paper_authors.type: user
paper_authors.user: bashnick
paper_authors._id: 67b7ee1dfedfe971271dcca4
paper_authors.name: Ajay Menon
paper_authors._id: 67b7ee1dfedfe971271dcca5
paper_authors.name: Vincent Moens
paper_authors._id: 67b7ee1dfedfe971271dcca6
paper_authors.name: Amar Budhiraja
paper_authors._id: 67b7ee1dfedfe971271dcca7
paper_authors.name: Despoina Magka
paper_authors._id: 67b7ee1dfedfe971271dcca8
paper_authors.name: Vladislav Vorotilov
paper_authors._id: 67b7ee1dfedfe971271dcca9
paper_authors.name: Gaurav Chaurasia
paper_authors._id: 67b7ee1dfedfe971271dccaa
paper_authors.name: Dieuwke Hupkes
paper_authors._id: 67b7ee1dfedfe971271dccab
paper_authors.name: Ricardo Silveira Cabral
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:42:55.449Z
paper_authors._id: 67b8749ffa8442592bce008e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ctbzupAxRhcNXDka75ANi.png
paper_authors.fullname: Ricardo Silveira Cabral
paper_authors.type: user
paper_authors.user: rscabral
paper_authors._id: 67b7ee1dfedfe971271dccac
paper_authors.name: Tatiana Shavrina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:43:03.777Z
paper_authors._id: 60dc2eb60037b630c5df57aa
paper_authors.avatarUrl: /avatars/fbe707b1231a3d9dc6e87ec011e0e738.svg
paper_authors.fullname: Tatiana
paper_authors.type: user
paper_authors.user: Shavrina
paper_authors._id: 67b7ee1dfedfe971271dccad
paper_authors.name: Jakob Foerster
paper_authors._id: 67b7ee1dfedfe971271dccae
paper_authors.name: Yoram Bachrach
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:43:01.304Z
paper_authors._id: 671b9e3ba54e59639d597fcb
paper_authors.avatarUrl: /avatars/95201b98d1cf2ffa68d23f3b74e387fb.svg
paper_authors.fullname: Yoram Bachrach
paper_authors.type: user
paper_authors.user: yorambac
paper_authors._id: 67b7ee1dfedfe971271dccaf
paper_authors.name: William Yang Wang
paper_authors._id: 67b7ee1dfedfe971271dccb0
paper_authors.name: Roberta Raileanu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-21T03:08:15.471Z
paper_authors._id: 633e94793a17ab61de8e2b9c
paper_authors.avatarUrl: /avatars/5f2f58ddeed211393660ada6b135f0d5.svg
paper_authors.fullname: Roberta Raileanu
paper_authors.type: user
paper_authors.user: rraileanu
paper_title: MLGym: A New Framework and Benchmark for Advancing AI Research Agents
paper_summary: We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for
  evaluating and developing LLM agents on AI research tasks. This is the first
  Gym environment for machine learning (ML) tasks, enabling research on
  reinforcement learning (RL) algorithms for training such agents. MLGym-bench
  consists of 13 diverse and open-ended AI research tasks from diverse domains
  such as computer vision, natural language processing, reinforcement learning,
  and game theory. Solving these tasks requires real-world AI research skills
  such as generating new ideas and hypotheses, creating and processing data,
  implementing ML methods, training models, running experiments, analyzing the
  results, and iterating through this process to improve on a given task. We
  evaluate a number of frontier large language models (LLMs) on our benchmarks
  such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5
  Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate
  models or agents, generate synthetic data at scale, as well as develop new
  learning algorithms for training agents on AI research tasks. We find that
  current frontier models can improve on the given baselines, usually by finding
  better hyperparameters, but do not generate novel hypotheses, algorithms,
  architectures, or substantial improvements. We open-source our framework and
  benchmark to facilitate future research in advancing the AI research
  capabilities of LLM agents.
paper_ai_keywords: Meta MLGym MLGym-Bench LLM agents AI research tasks Gym environment machine learning (ML) tasks reinforcement learning (RL) algorithms CV tasks natural language processing (NLP) tasks game theory tasks ML methods training models logging experiments LLMs Claude-3.5-Sonnet Llama-3.1 405B GPT-4o o1-preview Gemini-1.5 Pro learning algorithms