Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
  Language Family Experts
Adapting medical Large Language Models to local languages can reduce barriers
to accessing healthcare services, but data scarcity remains a significant
challenge, particularly for low-resource languages. To address this, we first
construct a high-quality medical dataset and conduct analysis to ensure its
quality. In order to leverage the generalization capability of multilingual
LLMs to efficiently scale to more resource-constrained languages, we explore
the internal information flow of LLMs from a multilingual perspective using
Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
routing method that employs language-specific experts and cross-lingual
routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
in the End information flow mechanism: while earlier layers concentrate
cross-lingual information flow, the later layers exhibit language-specific
divergence. This insight directly led to the development of the Post-MoE
architecture, which applies sparse routing only in the later layers while
maintaining dense others. Experimental results demonstrate that this approach
enhances the generalization of multilingual models to other languages while
preserving interpretability. Finally, to efficiently scale the model to 50
languages, we introduce the concept of language family experts, drawing on
linguistic priors, which enables scaling the number of languages without adding
additional parameters.
670de62008c0ea11e451cc97
Guorui Zheng
670de62008c0ea11e451cc98
Xidong Wang
670de62008c0ea11e451cc99
Juhao Liang
670de62008c0ea11e451cc9a
Nuo Chen
claimed_verified
2025-04-03T08:29:58.296Z
641ac2207c21ab946bf036e8
https://cdn-avatars.huggingface.co/v1/production/uploads/641ac2207c21ab946bf036e8/r6c9gpOrul0eC59d9e2Mo.png
Nuo Chen
user
nuojohnchen
670de62008c0ea11e451cc9b
Yuping Zheng
670de62008c0ea11e451cc9c
Benyou Wang
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
  Language Family Experts
Adapting medical Large Language Models to local languages can reduce barriers
to accessing healthcare services, but data scarcity remains a significant
challenge, particularly for low-resource languages. To address this, we first
construct a high-quality medical dataset and conduct analysis to ensure its
quality. In order to leverage the generalization capability of multilingual
LLMs to efficiently scale to more resource-constrained languages, we explore
the internal information flow of LLMs from a multilingual perspective using
Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
routing method that employs language-specific experts and cross-lingual
routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
in the End information flow mechanism: while earlier layers concentrate
cross-lingual information flow, the later layers exhibit language-specific
divergence. This insight directly led to the development of the Post-MoE
architecture, which applies sparse routing only in the later layers while
maintaining dense others. Experimental results demonstrate that this approach
enhances the generalization of multilingual models to other languages while
preserving interpretability. Finally, to efficiently scale the model to 50
languages, we introduce the concept of language family experts, drawing on
linguistic priors, which enables scaling the number of languages without adding
additional parameters.
A novel MoE-based routing method and language family experts enhance the generalization of multilingual medical LLMs to low-resource languages.
Mixture of Experts (MoE) multilingual LLMs Spread Out in the End Post-MoE architecture sparse routing language family experts