title: Self-MoE: Towards Compositional Large Language Models with
    Self-Specialized Experts
summary: We present Self-MoE, an approach that transforms a monolithic LLM into a
  compositional, modular system of self-specialized experts, named MiXSE (MiXture
  of Self-specialized Experts). Our approach leverages self-specialization, which
  constructs expert modules using self-generated synthetic data, each equipped
  with a shared base LLM and incorporating self-optimized routing. This allows
  for dynamic and capability-specific handling of various target tasks, enhancing
  overall capabilities, without extensive human-labeled data and added
  parameters. Our empirical results reveal that specializing LLMs may exhibit
  potential trade-offs in performances on non-specialized tasks. On the other
  hand, our Self-MoE demonstrates substantial improvements over the base LLM
  across diverse benchmarks such as knowledge, reasoning, math, and coding. It
  also consistently outperforms other methods, including instance merging and
  weight merging, while offering better flexibility and interpretability by
  design with semantic experts and routing. Our findings highlight the critical
  role of modularity and the potential of self-improvement in achieving
  efficient, scalable, and adaptable systems.
paper_authors._id: 6672fd271dbdf7da831ba57d
paper_authors.name: Junmo Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:18:21.605Z
paper_authors._id: 638967e68a5dbe2f3dcbf7ea
paper_authors.avatarUrl: /avatars/dba92b8d58e8c5f56771e3103abda93f.svg
paper_authors.fullname: Junmo Kang
paper_authors.type: user
paper_authors.user: jm-kang
paper_authors._id: 6672fd271dbdf7da831ba57e
paper_authors.name: Leonid Karlinsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:05.801Z
paper_authors._id: 62558f7cebfc059817b06690
paper_authors.avatarUrl: /avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
paper_authors.fullname: Leonid Karlinsky
paper_authors.type: user
paper_authors.user: leokarlin
paper_authors._id: 6672fd271dbdf7da831ba57f
paper_authors.name: Hongyin Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:11.740Z
paper_authors._id: 62bc495016897fa76cdcda18
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
paper_authors.fullname: Hongyin Luo
paper_authors.type: user
paper_authors.user: luohy
paper_authors._id: 6672fd271dbdf7da831ba580
paper_authors.name: Zhen Wang
paper_authors._id: 6672fd271dbdf7da831ba581
paper_authors.name: Jacob Hansen
paper_authors._id: 6672fd271dbdf7da831ba582
paper_authors.name: James Glass
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:31.173Z
paper_authors._id: 62a793d33e101ec156cc58c4
paper_authors.avatarUrl: /avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
paper_authors.fullname: James Glass
paper_authors.type: user
paper_authors.user: OmegaLittleBob
paper_authors._id: 6672fd271dbdf7da831ba583
paper_authors.name: David Cox
paper_authors._id: 6672fd271dbdf7da831ba584
paper_authors.name: Rameswar Panda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:42.297Z
paper_authors._id: 648a7542ac1fbb8a1a654645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
paper_authors.fullname: Rameswar Panda
paper_authors.type: user
paper_authors.user: rpand002
paper_authors._id: 6672fd271dbdf7da831ba585
paper_authors.name: Rogerio Feris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:48.676Z
paper_authors._id: 6426053ca5ec4a5cbc516567
paper_authors.avatarUrl: /avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
paper_authors.fullname: Rogerio Feris
paper_authors.type: user
paper_authors.user: rferis
paper_authors._id: 6672fd271dbdf7da831ba586
paper_authors.name: Alan Ritter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:56:54.133Z
paper_authors._id: 6377da9065ce6b04bd493cf8
paper_authors.avatarUrl: /avatars/6be2632d6da49857928f8bcce8278f49.svg
paper_authors.fullname: Alan Ritter
paper_authors.type: user
paper_authors.user: rittera
paper_title: Self-MoE: Towards Compositional Large Language Models with
    Self-Specialized Experts
paper_summary: We present Self-MoE, an approach that transforms a monolithic LLM into a
  compositional, modular system of self-specialized experts, named MiXSE (MiXture
  of Self-specialized Experts). Our approach leverages self-specialization, which
  constructs expert modules using self-generated synthetic data, each equipped
  with a shared base LLM and incorporating self-optimized routing. This allows
  for dynamic and capability-specific handling of various target tasks, enhancing
  overall capabilities, without extensive human-labeled data and added
  parameters. Our empirical results reveal that specializing LLMs may exhibit
  potential trade-offs in performances on non-specialized tasks. On the other
  hand, our Self-MoE demonstrates substantial improvements over the base LLM
  across diverse benchmarks such as knowledge, reasoning, math, and coding. It
  also consistently outperforms other methods, including instance merging and
  weight merging, while offering better flexibility and interpretability by
  design with semantic experts and routing. Our findings highlight the critical
  role of modularity and the potential of self-improvement in achieving
  efficient, scalable, and adaptable systems.
paper_ai_keywords: Self-MoE MiXSE (MiXture of Self-specialized Experts) self-specialization self-generated synthetic data shared base LLM self-optimized routing capability-specific handling diverse benchmarks knowledge reasoning math coding modularity self-improvement efficiency scalability adaptability