title: Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
    Language Family Experts
summary: Adapting medical Large Language Models to local languages can reduce barriers
  to accessing healthcare services, but data scarcity remains a significant
  challenge, particularly for low-resource languages. To address this, we first
  construct a high-quality medical dataset and conduct analysis to ensure its
  quality. In order to leverage the generalization capability of multilingual
  LLMs to efficiently scale to more resource-constrained languages, we explore
  the internal information flow of LLMs from a multilingual perspective using
  Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
  routing method that employs language-specific experts and cross-lingual
  routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
  in the End information flow mechanism: while earlier layers concentrate
  cross-lingual information flow, the later layers exhibit language-specific
  divergence. This insight directly led to the development of the Post-MoE
  architecture, which applies sparse routing only in the later layers while
  maintaining dense others. Experimental results demonstrate that this approach
  enhances the generalization of multilingual models to other languages while
  preserving interpretability. Finally, to efficiently scale the model to 50
  languages, we introduce the concept of language family experts, drawing on
  linguistic priors, which enables scaling the number of languages without adding
  additional parameters.
paper_authors._id: 670de62008c0ea11e451cc97
paper_authors.name: Guorui Zheng
paper_authors._id: 670de62008c0ea11e451cc98
paper_authors.name: Xidong Wang
paper_authors._id: 670de62008c0ea11e451cc99
paper_authors.name: Juhao Liang
paper_authors._id: 670de62008c0ea11e451cc9a
paper_authors.name: Nuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:58.296Z
paper_authors._id: 641ac2207c21ab946bf036e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641ac2207c21ab946bf036e8/r6c9gpOrul0eC59d9e2Mo.png
paper_authors.fullname: Nuo Chen
paper_authors.type: user
paper_authors.user: nuojohnchen
paper_authors._id: 670de62008c0ea11e451cc9b
paper_authors.name: Yuping Zheng
paper_authors._id: 670de62008c0ea11e451cc9c
paper_authors.name: Benyou Wang
paper_title: Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
    Language Family Experts
paper_summary: Adapting medical Large Language Models to local languages can reduce barriers
  to accessing healthcare services, but data scarcity remains a significant
  challenge, particularly for low-resource languages. To address this, we first
  construct a high-quality medical dataset and conduct analysis to ensure its
  quality. In order to leverage the generalization capability of multilingual
  LLMs to efficiently scale to more resource-constrained languages, we explore
  the internal information flow of LLMs from a multilingual perspective using
  Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
  routing method that employs language-specific experts and cross-lingual
  routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
  in the End information flow mechanism: while earlier layers concentrate
  cross-lingual information flow, the later layers exhibit language-specific
  divergence. This insight directly led to the development of the Post-MoE
  architecture, which applies sparse routing only in the later layers while
  maintaining dense others. Experimental results demonstrate that this approach
  enhances the generalization of multilingual models to other languages while
  preserving interpretability. Finally, to efficiently scale the model to 50
  languages, we introduce the concept of language family experts, drawing on
  linguistic priors, which enables scaling the number of languages without adding
  additional parameters.
paper_ai_summary: A novel MoE-based routing method and language family experts enhance the generalization of multilingual medical LLMs to low-resource languages.
paper_ai_keywords: Mixture of Experts (MoE) multilingual LLMs Spread Out in the End Post-MoE architecture sparse routing language family experts