MLGym: A New Framework and Benchmark for Advancing AI Research Agents
We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for
evaluating and developing LLM agents on AI research tasks. This is the first
Gym environment for machine learning (ML) tasks, enabling research on
reinforcement learning (RL) algorithms for training such agents. MLGym-bench
consists of 13 diverse and open-ended AI research tasks from diverse domains
such as computer vision, natural language processing, reinforcement learning,
and game theory. Solving these tasks requires real-world AI research skills
such as generating new ideas and hypotheses, creating and processing data,
implementing ML methods, training models, running experiments, analyzing the
results, and iterating through this process to improve on a given task. We
evaluate a number of frontier large language models (LLMs) on our benchmarks
such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5
Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate
models or agents, generate synthetic data at scale, as well as develop new
learning algorithms for training agents on AI research tasks. We find that
current frontier models can improve on the given baselines, usually by finding
better hyperparameters, but do not generate novel hypotheses, algorithms,
architectures, or substantial improvements. We open-source our framework and
benchmark to facilitate future research in advancing the AI research
capabilities of LLM agents.
67b7ee1dfedfe971271dcca0
Deepak Nathani
extracted_confirmed
2025-02-21T07:20:46.836Z
6114c9fae7a2566ae7d1a1a7
/avatars/c71ab1850322fcf5ef239cb8d31cb137.svg
Deepak Nathani
user
dnathani
67b7ee1dfedfe971271dcca1
Lovish Madaan
67b7ee1dfedfe971271dcca2
Nicholas Roberts
67b7ee1dfedfe971271dcca3
Nikolay Bashlykov
claimed_verified
2025-02-21T14:42:58.738Z
633476edc3cb9eda9328e556
/avatars/a127e270a606c18623fe00cd723313f6.svg
Nikolay B
user
bashnick
67b7ee1dfedfe971271dcca4
Ajay Menon
67b7ee1dfedfe971271dcca5
Vincent Moens
67b7ee1dfedfe971271dcca6
Amar Budhiraja
67b7ee1dfedfe971271dcca7
Despoina Magka
claimed_verified
2025-07-01T10:16:26.513Z
673dc2345b07d224bff10173
/avatars/2b7b1491de77bc3193fd912de76e176d.svg
Despoina Magka
user
marlamagka
67b7ee1dfedfe971271dcca8
Vladislav Vorotilov
claimed_verified
2025-04-08T06:58:23.089Z
67f0f493b814154649fdf184
/avatars/9ee3c6f48efee0544081cd9caa78f483.svg
Vladislav Vorotilov
user
vladvo
67b7ee1dfedfe971271dcca9
Gaurav Chaurasia
67b7ee1dfedfe971271dccaa
Dieuwke Hupkes
67b7ee1dfedfe971271dccab
Ricardo Silveira Cabral
claimed_verified
2025-02-21T14:42:55.449Z
67b8749ffa8442592bce008e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ctbzupAxRhcNXDka75ANi.png
Ricardo Silveira Cabral
user
rscabral
67b7ee1dfedfe971271dccac
Tatiana Shavrina
claimed_verified
2025-02-21T14:43:03.777Z
60dc2eb60037b630c5df57aa
/avatars/fbe707b1231a3d9dc6e87ec011e0e738.svg
Tatiana
user
Shavrina
67b7ee1dfedfe971271dccad
Jakob Foerster
67b7ee1dfedfe971271dccae
Yoram Bachrach
claimed_verified
2025-02-21T14:43:01.304Z
671b9e3ba54e59639d597fcb
/avatars/95201b98d1cf2ffa68d23f3b74e387fb.svg
Yoram Bachrach
user
yorambac
67b7ee1dfedfe971271dccaf
William Yang Wang
67b7ee1dfedfe971271dccb0
Roberta Raileanu
extracted_pending
2025-02-21T03:08:15.471Z
633e94793a17ab61de8e2b9c
/avatars/5f2f58ddeed211393660ada6b135f0d5.svg
Roberta Raileanu
user
rraileanu
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for
evaluating and developing LLM agents on AI research tasks. This is the first
Gym environment for machine learning (ML) tasks, enabling research on
reinforcement learning (RL) algorithms for training such agents. MLGym-bench
consists of 13 diverse and open-ended AI research tasks from diverse domains
such as computer vision, natural language processing, reinforcement learning,
and game theory. Solving these tasks requires real-world AI research skills
such as generating new ideas and hypotheses, creating and processing data,
implementing ML methods, training models, running experiments, analyzing the
results, and iterating through this process to improve on a given task. We
evaluate a number of frontier large language models (LLMs) on our benchmarks
such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5
Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate
models or agents, generate synthetic data at scale, as well as develop new
learning algorithms for training agents on AI research tasks. We find that
current frontier models can improve on the given baselines, usually by finding
better hyperparameters, but do not generate novel hypotheses, algorithms,
architectures, or substantial improvements. We open-source our framework and
benchmark to facilitate future research in advancing the AI research
capabilities of LLM agents.
A new framework and benchmark, MLGym and MLGym-Bench, is introduced for evaluating and developing LLM agents across diverse AI research tasks using reinforcement learning.
Gym reinforcement learning (RL) large language models (LLMs) Claude-3.5-Sonnet Llama-3.1 405B GPT-4o o1-preview Gemini-1.5 Pro machine learning (ML) computer vision natural language processing game theory hyperparameters synthetic data learning algorithms