LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
  Large Language Models
Mixture of Experts (MoEs) plays an important role in the development of more
efficient and effective large language models (LLMs). Due to the enormous
resource requirements, studying large scale MoE algorithms remain in-accessible
to many researchers. This work develops LibMoE, a comprehensive and
modular framework to streamline the research, training, and evaluation of MoE
algorithms. Built upon three core principles: (i) modular design, (ii)
efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
more accessible to a wide range of researchers by standardizing the training
and evaluation pipelines. Using LibMoE, we extensively benchmarked five
state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
the zero-shot setting. The results show that despite the unique
characteristics, all MoE algorithms perform roughly similar when averaged
across a wide range of tasks. With the modular design and extensive evaluation,
we believe LibMoE will be invaluable for researchers to make meaningful
progress towards the next generation of MoE and LLMs. Project page:
https://fsoft-aic.github.io/fsoft-LibMoE.github.io.
672979dafc78f157e47a8ad0
Nam V. Nguyen
claimed_verified
2024-11-05T07:58:16.335Z
64c2bea2ada7df214276913b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
Nguyen Van Nam
user
DavidNguyen
672979dafc78f157e47a8ad1
Thong T. Doan
claimed_verified
2024-11-05T07:58:14.469Z
63e8ad2accae1fe5c61878d6
https://cdn-avatars.huggingface.co/v1/production/uploads/1676193023976-noauth.jpeg
Doan Tien Thong
user
doantienthongbku
672979dafc78f157e47a8ad2
Luong Tran
672979dafc78f157e47a8ad3
Van Nguyen
672979dafc78f157e47a8ad4
Quang Pham
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
  Large Language Models
Mixture of Experts (MoEs) plays an important role in the development of more
efficient and effective large language models (LLMs). Due to the enormous
resource requirements, studying large scale MoE algorithms remain in-accessible
to many researchers. This work develops LibMoE, a comprehensive and
modular framework to streamline the research, training, and evaluation of MoE
algorithms. Built upon three core principles: (i) modular design, (ii)
efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
more accessible to a wide range of researchers by standardizing the training
and evaluation pipelines. Using LibMoE, we extensively benchmarked five
state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
the zero-shot setting. The results show that despite the unique
characteristics, all MoE algorithms perform roughly similar when averaged
across a wide range of tasks. With the modular design and extensive evaluation,
we believe LibMoE will be invaluable for researchers to make meaningful
progress towards the next generation of MoE and LLMs. Project page:
https://fsoft-aic.github.io/fsoft-LibMoE.github.io.
LibMoE provides a modular framework for accessible research, training, and evaluation of Mixture of Experts algorithms in large language models, enabling standardized pipelines and benchmarking across various tasks.
Mixture of Experts large language models LibMoE modular design efficient training comprehensive evaluation zero-shot setting