title: FLAME-MoE: A Transparent End-to-End Research Platform for
    Mixture-of-Experts Language Models
summary: Recent large language models such as Gemini-1.5, DeepSeek-V3, and Llama-4
  increasingly adopt Mixture-of-Experts (MoE) architectures, which offer strong
  efficiency-performance trade-offs by activating only a fraction of the model
  per token. Yet academic researchers still lack a fully open, end-to-end MoE
  platform for investigating scaling, routing, and expert behavior. We release
  FLAME-MoE, a completely open-source research suite composed of seven
  decoder-only models, ranging from 38M to 1.7B active parameters, whose
  architecture--64 experts with top-8 gating and 2 shared experts--closely
  reflects modern production LLMs. All training data pipelines, scripts, logs,
  and checkpoints are publicly available to enable reproducible experimentation.
  Across six evaluation tasks, FLAME-MoE improves average accuracy by up to 3.4
  points over dense baselines trained with identical FLOPs. Leveraging full
  training trace transparency, we present initial analyses showing that (i)
  experts increasingly specialize on distinct token subsets, (ii) co-activation
  matrices remain sparse, reflecting diverse expert usage, and (iii) routing
  behavior stabilizes early in training. All code, training logs, and model
  checkpoints are available at https://github.com/cmu-flame/FLAME-MoE.
paper_authors._id: 6835d87e46c0a9c0f2cac425
paper_authors.name: Hao Kang
paper_authors._id: 6835d87e46c0a9c0f2cac426
paper_authors.name: Zichun Yu
paper_authors._id: 6835d87e46c0a9c0f2cac427
paper_authors.name: Chenyan Xiong
paper_title: FLAME-MoE: A Transparent End-to-End Research Platform for
    Mixture-of-Experts Language Models
paper_summary: Recent large language models such as Gemini-1.5, DeepSeek-V3, and Llama-4
  increasingly adopt Mixture-of-Experts (MoE) architectures, which offer strong
  efficiency-performance trade-offs by activating only a fraction of the model
  per token. Yet academic researchers still lack a fully open, end-to-end MoE
  platform for investigating scaling, routing, and expert behavior. We release
  FLAME-MoE, a completely open-source research suite composed of seven
  decoder-only models, ranging from 38M to 1.7B active parameters, whose
  architecture--64 experts with top-8 gating and 2 shared experts--closely
  reflects modern production LLMs. All training data pipelines, scripts, logs,
  and checkpoints are publicly available to enable reproducible experimentation.
  Across six evaluation tasks, FLAME-MoE improves average accuracy by up to 3.4
  points over dense baselines trained with identical FLOPs. Leveraging full
  training trace transparency, we present initial analyses showing that (i)
  experts increasingly specialize on distinct token subsets, (ii) co-activation
  matrices remain sparse, reflecting diverse expert usage, and (iii) routing
  behavior stabilizes early in training. All code, training logs, and model
  checkpoints are available at https://github.com/cmu-flame/FLAME-MoE.
paper_ai_summary: FLAME-MoE is an open-source research suite for MoE architectures in LLMs, providing tools to investigate scaling, routing, and expert behavior with reproducible experiments.
paper_ai_keywords: Mixture-of-Experts (MoE) decoder-only models top-8 gating shared experts training trace transparency co-activation matrices routing behavior