MALT: Improving Reasoning with Multi-Agent LLM Training
Enabling effective collaboration among LLMs is a crucial step toward
developing autonomous systems capable of solving complex problems. While LLMs
are typically used as single-model generators, where humans critique and refine
their outputs, the potential for jointly-trained collaborative models remains
largely unexplored. Despite promising results in multi-agent communication and
debate settings, little progress has been made in training models to work
together on tasks. In this paper, we present a first step toward "Multi-agent
LLM training" (MALT) on reasoning problems. Our approach employs a sequential
multi-agent setup with heterogeneous LLMs assigned specialized roles: a
generator, verifier, and refinement model iteratively solving problems. We
propose a trajectory-expansion-based synthetic data generation process and a
credit assignment strategy driven by joint outcome based rewards. This enables
our post-training setup to utilize both positive and negative trajectories to
autonomously improve each model's specialized capabilities as part of a joint
sequential system. We evaluate our approach across MATH, GSM8k, and CQA, where
MALT on Llama 3.1 8B models achieves relative improvements of 14.14%, 7.12%,
and 9.40% respectively over the same baseline model. This demonstrates an early
advance in multi-agent cooperative capabilities for performance on mathematical
and common sense reasoning questions. More generally, our work provides a
concrete direction for research around multi-agent LLM training approaches.
67500fddeac35e016f941d6e
Sumeet Ramesh Motwani
67500fddeac35e016f941d6f
Chandler Smith
67500fddeac35e016f941d70
Rocktim Jyoti Das
admin_assigned
2024-12-04T13:06:35.900Z
665ed23ee4f3744138d302e6
/avatars/34567cf4c57fa8b9ddd7338df7e12a11.svg
Rocktim Jyoti Das
user
RocktimMBZ
67500fddeac35e016f941d71
Markian Rybchuk
67500fddeac35e016f941d72
Philip H. S. Torr
admin_assigned
2024-12-04T13:07:13.104Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
67500fddeac35e016f941d73
Ivan Laptev
67500fddeac35e016f941d74
Fabio Pizzati
67500fddeac35e016f941d75
Ronald Clark
admin_assigned
2024-12-04T13:02:31.896Z
6305ee63d70693fdf1c7dbb8
/avatars/0e81ed3757b4e65be82063b538c3fe49.svg
Ronald Clark
user
r0nn13
67500fddeac35e016f941d76
Christian Schroeder de Witt
admin_assigned
2024-12-04T13:02:15.884Z
62cd901ff27cc94a7594f7f9
/avatars/239d4a2a850e3bb819dac8a1ca3bbd77.svg
Christian Schroeder de Witt
user
csdw
MALT: Improving Reasoning with Multi-Agent LLM Training
Enabling effective collaboration among LLMs is a crucial step toward
developing autonomous systems capable of solving complex problems. While LLMs
are typically used as single-model generators, where humans critique and refine
their outputs, the potential for jointly-trained collaborative models remains
largely unexplored. Despite promising results in multi-agent communication and
debate settings, little progress has been made in training models to work
together on tasks. In this paper, we present a first step toward "Multi-agent
LLM training" (MALT) on reasoning problems. Our approach employs a sequential
multi-agent setup with heterogeneous LLMs assigned specialized roles: a
generator, verifier, and refinement model iteratively solving problems. We
propose a trajectory-expansion-based synthetic data generation process and a
credit assignment strategy driven by joint outcome based rewards. This enables
our post-training setup to utilize both positive and negative trajectories to
autonomously improve each model's specialized capabilities as part of a joint
sequential system. We evaluate our approach across MATH, GSM8k, and CQA, where
MALT on Llama 3.1 8B models achieves relative improvements of 14.14%, 7.12%,
and 9.40% respectively over the same baseline model. This demonstrates an early
advance in multi-agent cooperative capabilities for performance on mathematical
and common sense reasoning questions. More generally, our work provides a
concrete direction for research around multi-agent LLM training approaches.
Multi-agent LLM training improves performance on reasoning tasks by assigning specialized roles and utilizing joint outcome-based rewards to enhance collaboration among models.
sequential multi-agent setup heterogeneous LLMs generator verifier refinement model trajectory-expansion credit assignment strategy joint outcome based rewards MATH GSM8k CQA Llama 3.1 8B models multi-agent cooperative capabilities