title: LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
    Large Language Models
summary: Mixture of Experts (MoEs) plays an important role in the development of more
  efficient and effective large language models (LLMs). Due to the enormous
  resource requirements, studying large scale MoE algorithms remain in-accessible
  to many researchers. This work develops LibMoE, a comprehensive and
  modular framework to streamline the research, training, and evaluation of MoE
  algorithms. Built upon three core principles: (i) modular design, (ii)
  efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
  more accessible to a wide range of researchers by standardizing the training
  and evaluation pipelines. Using LibMoE, we extensively benchmarked five
  state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
  the zero-shot setting. The results show that despite the unique
  characteristics, all MoE algorithms perform roughly similar when averaged
  across a wide range of tasks. With the modular design and extensive evaluation,
  we believe LibMoE will be invaluable for researchers to make meaningful
  progress towards the next generation of MoE and LLMs. Project page:
  https://fsoft-aic.github.io/fsoft-LibMoE.github.io.
paper_authors._id: 672979dafc78f157e47a8ad0
paper_authors.name: Nam V. Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:16.335Z
paper_authors._id: 64c2bea2ada7df214276913b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
paper_authors.fullname: Nguyen Van Nam
paper_authors.type: user
paper_authors.user: DavidNguyen
paper_authors._id: 672979dafc78f157e47a8ad1
paper_authors.name: Thong T. Doan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:14.469Z
paper_authors._id: 63e8ad2accae1fe5c61878d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676193023976-noauth.jpeg
paper_authors.fullname: Doan Tien Thong
paper_authors.type: user
paper_authors.user: doantienthongbku
paper_authors._id: 672979dafc78f157e47a8ad2
paper_authors.name: Luong Tran
paper_authors._id: 672979dafc78f157e47a8ad3
paper_authors.name: Van Nguyen
paper_authors._id: 672979dafc78f157e47a8ad4
paper_authors.name: Quang Pham
paper_title: LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in
    Large Language Models
paper_summary: Mixture of Experts (MoEs) plays an important role in the development of more
  efficient and effective large language models (LLMs). Due to the enormous
  resource requirements, studying large scale MoE algorithms remain in-accessible
  to many researchers. This work develops LibMoE, a comprehensive and
  modular framework to streamline the research, training, and evaluation of MoE
  algorithms. Built upon three core principles: (i) modular design, (ii)
  efficient training; (iii) comprehensive evaluation, LibMoE brings MoE in LLMs
  more accessible to a wide range of researchers by standardizing the training
  and evaluation pipelines. Using LibMoE, we extensively benchmarked five
  state-of-the-art MoE algorithms over three different LLMs and 11 datasets under
  the zero-shot setting. The results show that despite the unique
  characteristics, all MoE algorithms perform roughly similar when averaged
  across a wide range of tasks. With the modular design and extensive evaluation,
  we believe LibMoE will be invaluable for researchers to make meaningful
  progress towards the next generation of MoE and LLMs. Project page:
  https://fsoft-aic.github.io/fsoft-LibMoE.github.io.
paper_ai_keywords: Mixture of Experts (MoEs) large language models (LLMs) LibMoE modular design efficient training comprehensive evaluation zero-shot setting