title: Optima: Optimizing Effectiveness and Efficiency for LLM-Based
    Multi-Agent System
summary: Large Language Model (LLM) based multi-agent systems (MAS) show remarkable
  potential in collaborative problem-solving, yet they still face critical
  challenges: low communication efficiency, poor scalability, and a lack of
  effective parameter-updating optimization methods. We present Optima, a novel
  framework that addresses these issues by significantly enhancing both
  communication efficiency and task effectiveness in LLM-based MAS through LLM
  training. Optima employs an iterative generate, rank, select, and train
  paradigm with a reward function balancing task performance, token efficiency,
  and communication readability. We explore various RL algorithms, including
  Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid
  approaches, providing insights into their effectiveness-efficiency trade-offs.
  We integrate Monte Carlo Tree Search-inspired techniques for DPO data
  generation, treating conversation turns as tree nodes to explore diverse
  interaction paths. Evaluated on common multi-agent tasks, including
  information-asymmetric question answering and complex reasoning, Optima shows
  consistent and substantial improvements over single-agent baselines and vanilla
  MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than
  10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's
  efficiency gains open new possibilities for leveraging inference-compute more
  effectively, leading to improved inference-time scaling laws. By addressing
  fundamental challenges in LLM-based MAS, Optima shows the potential towards
  scalable, efficient, and effective MAS
  (https://chenweize1998.github.io/optima-project-page).
paper_authors._id: 67088f119e319cca02be09f2
paper_authors.name: Weize Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:24:46.121Z
paper_authors._id: 648312243b7fe59c876c0dca
paper_authors.avatarUrl: /avatars/c26ad76cd213529e4670bb599b8199bb.svg
paper_authors.fullname: weize
paper_authors.type: user
paper_authors.user: weizechen
paper_authors._id: 67088f119e319cca02be09f3
paper_authors.name: Jiarui Yuan
paper_authors._id: 67088f119e319cca02be09f4
paper_authors.name: Chen Qian
paper_authors._id: 67088f119e319cca02be09f5
paper_authors.name: Cheng Yang
paper_authors._id: 67088f119e319cca02be09f6
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:24:01.889Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 67088f119e319cca02be09f7
paper_authors.name: Maosong Sun
paper_title: Optima: Optimizing Effectiveness and Efficiency for LLM-Based
    Multi-Agent System
paper_summary: Large Language Model (LLM) based multi-agent systems (MAS) show remarkable
  potential in collaborative problem-solving, yet they still face critical
  challenges: low communication efficiency, poor scalability, and a lack of
  effective parameter-updating optimization methods. We present Optima, a novel
  framework that addresses these issues by significantly enhancing both
  communication efficiency and task effectiveness in LLM-based MAS through LLM
  training. Optima employs an iterative generate, rank, select, and train
  paradigm with a reward function balancing task performance, token efficiency,
  and communication readability. We explore various RL algorithms, including
  Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid
  approaches, providing insights into their effectiveness-efficiency trade-offs.
  We integrate Monte Carlo Tree Search-inspired techniques for DPO data
  generation, treating conversation turns as tree nodes to explore diverse
  interaction paths. Evaluated on common multi-agent tasks, including
  information-asymmetric question answering and complex reasoning, Optima shows
  consistent and substantial improvements over single-agent baselines and vanilla
  MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than
  10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's
  efficiency gains open new possibilities for leveraging inference-compute more
  effectively, leading to improved inference-time scaling laws. By addressing
  fundamental challenges in LLM-based MAS, Optima shows the potential towards
  scalable, efficient, and effective MAS
  (https://chenweize1998.github.io/optima-project-page).
paper_ai_summary: Optima framework improves communication efficiency and task performance in LLM-based multi-agent systems through iterative training with reward functions and RL algorithms, achieving substantial gains over single-agent baselines.
paper_ai_keywords: LLM-based multi-agent systems communication efficiency task effectiveness iterative generate rank select train paradigm reward function Supervised Fine-Tuning Direct Preference Optimization Monte Carlo Tree Search information-asymmetric question answering complex reasoning inference-compute scaling laws