title: MALT: Improving Reasoning with Multi-Agent LLM Training
summary: Enabling effective collaboration among LLMs is a crucial step toward
  developing autonomous systems capable of solving complex problems. While LLMs
  are typically used as single-model generators, where humans critique and refine
  their outputs, the potential for jointly-trained collaborative models remains
  largely unexplored. Despite promising results in multi-agent communication and
  debate settings, little progress has been made in training models to work
  together on tasks. In this paper, we present a first step toward "Multi-agent
  LLM training" (MALT) on reasoning problems. Our approach employs a sequential
  multi-agent setup with heterogeneous LLMs assigned specialized roles: a
  generator, verifier, and refinement model iteratively solving problems. We
  propose a trajectory-expansion-based synthetic data generation process and a
  credit assignment strategy driven by joint outcome based rewards. This enables
  our post-training setup to utilize both positive and negative trajectories to
  autonomously improve each model's specialized capabilities as part of a joint
  sequential system. We evaluate our approach across MATH, GSM8k, and CQA, where
  MALT on Llama 3.1 8B models achieves relative improvements of 14.14%, 7.12%,
  and 9.40% respectively over the same baseline model. This demonstrates an early
  advance in multi-agent cooperative capabilities for performance on mathematical
  and common sense reasoning questions. More generally, our work provides a
  concrete direction for research around multi-agent LLM training approaches.
paper_authors._id: 67500fddeac35e016f941d6e
paper_authors.name: Sumeet Ramesh Motwani
paper_authors._id: 67500fddeac35e016f941d6f
paper_authors.name: Chandler Smith
paper_authors._id: 67500fddeac35e016f941d70
paper_authors.name: Rocktim Jyoti Das
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T13:06:35.900Z
paper_authors._id: 665ed23ee4f3744138d302e6
paper_authors.avatarUrl: /avatars/34567cf4c57fa8b9ddd7338df7e12a11.svg
paper_authors.fullname: Rocktim Jyoti Das
paper_authors.type: user
paper_authors.user: RocktimMBZ
paper_authors._id: 67500fddeac35e016f941d71
paper_authors.name: Markian Rybchuk
paper_authors._id: 67500fddeac35e016f941d72
paper_authors.name: Philip H. S. Torr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T13:07:13.104Z
paper_authors._id: 6565ed28a5ec0231cb07225f
paper_authors.avatarUrl: /avatars/7f95bba9aa7811d56eecb380827abfac.svg
paper_authors.fullname: prof philip torr
paper_authors.type: user
paper_authors.user: philiptorr
paper_authors._id: 67500fddeac35e016f941d73
paper_authors.name: Ivan Laptev
paper_authors._id: 67500fddeac35e016f941d74
paper_authors.name: Fabio Pizzati
paper_authors._id: 67500fddeac35e016f941d75
paper_authors.name: Ronald Clark
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T13:02:31.896Z
paper_authors._id: 6305ee63d70693fdf1c7dbb8
paper_authors.avatarUrl: /avatars/0e81ed3757b4e65be82063b538c3fe49.svg
paper_authors.fullname: Ronald Clark
paper_authors.type: user
paper_authors.user: r0nn13
paper_authors._id: 67500fddeac35e016f941d76
paper_authors.name: Christian Schroeder de Witt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T13:02:15.884Z
paper_authors._id: 62cd901ff27cc94a7594f7f9
paper_authors.avatarUrl: /avatars/239d4a2a850e3bb819dac8a1ca3bbd77.svg
paper_authors.fullname: Christian Schroeder de Witt
paper_authors.type: user
paper_authors.user: csdw
paper_title: MALT: Improving Reasoning with Multi-Agent LLM Training
paper_summary: Enabling effective collaboration among LLMs is a crucial step toward
  developing autonomous systems capable of solving complex problems. While LLMs
  are typically used as single-model generators, where humans critique and refine
  their outputs, the potential for jointly-trained collaborative models remains
  largely unexplored. Despite promising results in multi-agent communication and
  debate settings, little progress has been made in training models to work
  together on tasks. In this paper, we present a first step toward "Multi-agent
  LLM training" (MALT) on reasoning problems. Our approach employs a sequential
  multi-agent setup with heterogeneous LLMs assigned specialized roles: a
  generator, verifier, and refinement model iteratively solving problems. We
  propose a trajectory-expansion-based synthetic data generation process and a
  credit assignment strategy driven by joint outcome based rewards. This enables
  our post-training setup to utilize both positive and negative trajectories to
  autonomously improve each model's specialized capabilities as part of a joint
  sequential system. We evaluate our approach across MATH, GSM8k, and CQA, where
  MALT on Llama 3.1 8B models achieves relative improvements of 14.14%, 7.12%,
  and 9.40% respectively over the same baseline model. This demonstrates an early
  advance in multi-agent cooperative capabilities for performance on mathematical
  and common sense reasoning questions. More generally, our work provides a
  concrete direction for research around multi-agent LLM training approaches.
paper_ai_keywords: Multi-agent LLM training (MALT) sequential multi-agent setup heterogeneous LLMs generator verifier refinement model logical trajectory-expansion-based synthetic data generation credit assignment strategy joint outcome based rewards post-training setup MALT on Llama 3.1 8B models M relative improvements MALT on Llama 3.1 8B models mathematical reasoning common sense reasoning