Self-MoE: Towards Compositional Large Language Models with
  Self-Specialized Experts
We present Self-MoE, an approach that transforms a monolithic LLM into a
compositional, modular system of self-specialized experts, named MiXSE (MiXture
of Self-specialized Experts). Our approach leverages self-specialization, which
constructs expert modules using self-generated synthetic data, each equipped
with a shared base LLM and incorporating self-optimized routing. This allows
for dynamic and capability-specific handling of various target tasks, enhancing
overall capabilities, without extensive human-labeled data and added
parameters. Our empirical results reveal that specializing LLMs may exhibit
potential trade-offs in performances on non-specialized tasks. On the other
hand, our Self-MoE demonstrates substantial improvements over the base LLM
across diverse benchmarks such as knowledge, reasoning, math, and coding. It
also consistently outperforms other methods, including instance merging and
weight merging, while offering better flexibility and interpretability by
design with semantic experts and routing. Our findings highlight the critical
role of modularity and the potential of self-improvement in achieving
efficient, scalable, and adaptable systems.
6672fd271dbdf7da831ba57d
Junmo Kang
claimed_verified
2024-06-19T20:18:21.605Z
638967e68a5dbe2f3dcbf7ea
/avatars/dba92b8d58e8c5f56771e3103abda93f.svg
Junmo Kang
user
jm-kang
6672fd271dbdf7da831ba57e
Leonid Karlinsky
admin_assigned
2024-06-20T09:56:05.801Z
62558f7cebfc059817b06690
/avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
Leonid Karlinsky
user
leokarlin
6672fd271dbdf7da831ba57f
Hongyin Luo
admin_assigned
2024-06-20T09:56:11.740Z
62bc495016897fa76cdcda18
https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
Hongyin Luo
user
luohy
6672fd271dbdf7da831ba580
Zhen Wang
6672fd271dbdf7da831ba581
Jacob Hansen
6672fd271dbdf7da831ba582
James Glass
admin_assigned
2024-06-20T09:56:31.173Z
62a793d33e101ec156cc58c4
/avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
James Glass
user
OmegaLittleBob
6672fd271dbdf7da831ba583
David Cox
6672fd271dbdf7da831ba584
Rameswar Panda
admin_assigned
2024-06-20T09:56:42.297Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
6672fd271dbdf7da831ba585
Rogerio Feris
admin_assigned
2024-06-20T09:56:48.676Z
6426053ca5ec4a5cbc516567
/avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
Rogerio Feris
user
rferis
6672fd271dbdf7da831ba586
Alan Ritter
admin_assigned
2024-06-20T09:56:54.133Z
6377da9065ce6b04bd493cf8
/avatars/6be2632d6da49857928f8bcce8278f49.svg
Alan Ritter
user
rittera
Self-MoE: Towards Compositional Large Language Models with
  Self-Specialized Experts
We present Self-MoE, an approach that transforms a monolithic LLM into a
compositional, modular system of self-specialized experts, named MiXSE (MiXture
of Self-specialized Experts). Our approach leverages self-specialization, which
constructs expert modules using self-generated synthetic data, each equipped
with a shared base LLM and incorporating self-optimized routing. This allows
for dynamic and capability-specific handling of various target tasks, enhancing
overall capabilities, without extensive human-labeled data and added
parameters. Our empirical results reveal that specializing LLMs may exhibit
potential trade-offs in performances on non-specialized tasks. On the other
hand, our Self-MoE demonstrates substantial improvements over the base LLM
across diverse benchmarks such as knowledge, reasoning, math, and coding. It
also consistently outperforms other methods, including instance merging and
weight merging, while offering better flexibility and interpretability by
design with semantic experts and routing. Our findings highlight the critical
role of modularity and the potential of self-improvement in achieving
efficient, scalable, and adaptable systems.
Self-MoE transforms a monolithic LLM into a modular system of self-specialized experts, enhancing performance across diverse benchmarks with improved flexibility and interpretability.
Self-MoE MiXSE MiXture of Self-specialized Experts self-specialization synthetic data self-optimized routing knowledge reasoning math coding instance merging weight merging