title: Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
summary: Large language models (LLMs) have achieved remarkable performance in recent
  years but are fundamentally limited by the underlying training data. To improve
  models beyond the training data, recent works have explored how LLMs can be
  used to generate synthetic data for autonomous self-improvement. However,
  successive steps of self-improvement can reach a point of diminishing returns.
  In this work, we propose a complementary approach towards self-improvement
  where finetuning is applied to a multiagent society of language models. A group
  of language models, all starting from the same base model, are independently
  specialized by updating each one using data generated through multiagent
  interactions among the models. By training each model on independent sets of
  data, we illustrate how this approach enables specialization across models and
  diversification over the set of models. As a result, our overall system is able
  to preserve diverse reasoning chains and autonomously improve over many more
  rounds of fine-tuning than single-agent self-improvement methods. We
  quantitatively illustrate the efficacy of the approach across a wide suite of
  reasoning tasks.
paper_authors._id: 6784a4b4cf658d598c274e52
paper_authors.name: Vighnesh Subramaniam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:34:56.291Z
paper_authors._id: 63dc6b595d310b018be1a6c7
paper_authors.avatarUrl: /avatars/f6bb94fe302b1627b27f533ba6c32ed3.svg
paper_authors.fullname: Vighnesh Subramaniam
paper_authors.type: user
paper_authors.user: vsub851
paper_authors._id: 6784a4b4cf658d598c274e53
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:35:03.427Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 6784a4b4cf658d598c274e54
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 6784a4b4cf658d598c274e55
paper_authors.name: Antonio Torralba
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:35:36.299Z
paper_authors._id: 6744c6ec6ec99a37d4ba9235
paper_authors.avatarUrl: /avatars/a5384b63bb615192f6fa157c6ea89e92.svg
paper_authors.fullname: Antonio
paper_authors.type: user
paper_authors.user: Antoniotorralbaborruel
paper_authors._id: 6784a4b4cf658d598c274e56
paper_authors.name: Shuang Li
paper_authors._id: 6784a4b4cf658d598c274e57
paper_authors.name: Igor Mordatch
paper_title: Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
paper_summary: Large language models (LLMs) have achieved remarkable performance in recent
  years but are fundamentally limited by the underlying training data. To improve
  models beyond the training data, recent works have explored how LLMs can be
  used to generate synthetic data for autonomous self-improvement. However,
  successive steps of self-improvement can reach a point of diminishing returns.
  In this work, we propose a complementary approach towards self-improvement
  where finetuning is applied to a multiagent society of language models. A group
  of language models, all starting from the same base model, are independently
  specialized by updating each one using data generated through multiagent
  interactions among the models. By training each model on independent sets of
  data, we illustrate how this approach enables specialization across models and
  diversification over the set of models. As a result, our overall system is able
  to preserve diverse reasoning chains and autonomously improve over many more
  rounds of fine-tuning than single-agent self-improvement methods. We
  quantitatively illustrate the efficacy of the approach across a wide suite of
  reasoning tasks.
paper_ai_keywords: large language models (LLMs) synthetic data multiagent society language models multiagent interactions independent specialization data generated fine-tuning quantitative illustration reasoning tasks