title: Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models
    Beneficial?
summary: Ensembling outputs from diverse sources is a straightforward yet effective
  approach to boost performance. Mixture-of-Agents (MoA) is one such popular
  ensemble method that aggregates outputs from multiple different Large Language
  Models (LLMs). This paper raises the question in the context of language
  models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an
  ensemble method that aggregates outputs from only the single top-performing
  LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms
  standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA
  achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an
  average of 3.8% improvement across various benchmarks, including MMLU, CRUX,
  and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0
  directly achieves the new state-of-the-art performance on the leaderboard. To
  understand the effectiveness of Self-MoA, we systematically investigate the
  trade-off between diversity and quality of outputs under various MoA settings.
  We confirm that the MoA performance is rather sensitive to the quality, and
  mixing different LLMs often lowers the average quality of the models. To
  complement the study, we identify the scenarios where mixing different LLMs
  could be helpful. This paper further introduces a sequential version of
  Self-MoA, that is capable of aggregating a large number of LLM outputs
  on-the-fly over multiple rounds, and is as effective as aggregating all outputs
  at once.
paper_authors._id: 67a362c9b9a2bb11fdba4b9f
paper_authors.name: Wenzhe Li
paper_authors._id: 67a362c9b9a2bb11fdba4ba0
paper_authors.name: Yong Lin
paper_authors._id: 67a362c9b9a2bb11fdba4ba1
paper_authors.name: Mengzhou Xia
paper_authors._id: 67a362c9b9a2bb11fdba4ba2
paper_authors.name: Chi Jin
paper_title: Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models
    Beneficial?
paper_summary: Ensembling outputs from diverse sources is a straightforward yet effective
  approach to boost performance. Mixture-of-Agents (MoA) is one such popular
  ensemble method that aggregates outputs from multiple different Large Language
  Models (LLMs). This paper raises the question in the context of language
  models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an
  ensemble method that aggregates outputs from only the single top-performing
  LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms
  standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA
  achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an
  average of 3.8% improvement across various benchmarks, including MMLU, CRUX,
  and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0
  directly achieves the new state-of-the-art performance on the leaderboard. To
  understand the effectiveness of Self-MoA, we systematically investigate the
  trade-off between diversity and quality of outputs under various MoA settings.
  We confirm that the MoA performance is rather sensitive to the quality, and
  mixing different LLMs often lowers the average quality of the models. To
  complement the study, we identify the scenarios where mixing different LLMs
  could be helpful. This paper further introduces a sequential version of
  Self-MoA, that is capable of aggregating a large number of LLM outputs
  on-the-fly over multiple rounds, and is as effective as aggregating all outputs
  at once.
paper_ai_summary: Self-MoA, an ensemble method using outputs from a single top-performing LLM, outperforms standard MoA that aggregates multiple LLMs across various benchmarks.
paper_ai_keywords: Mixture-of-Agents Large Language Models AlpacaEval 2.0 MMLU CRUX MATH sequential version of Self-MoA