Multilingual Encoder Knows more than You Realize: Shared Weights
  Pretraining for Extremely Low-Resource Languages
While multilingual language models like XLM-R have advanced multilingualism
in NLP, they still perform poorly in extremely low-resource languages. This
situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen
support far fewer languages than XLM-R, making text generation models
non-existent for many languages in the world. To tackle this challenge, we
propose a novel framework for adapting multilingual encoders to text generation
in extremely low-resource languages. By reusing the weights between the encoder
and the decoder, our framework allows the model to leverage the learned
semantic space of the encoder, enabling efficient learning and effective
generalization in low-resource languages. Applying this framework to four
Chinese minority languages, we present XLM-SWCM, and demonstrate its superior
performance on various downstream tasks even when compared with much larger
models.
67b55321f703732d151de666
Zeli Su
67b55321f703732d151de667
Ziyin Zhang
admin_assigned
2025-02-19T15:10:05.400Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
67b55321f703732d151de668
Guixian Xu
admin_assigned
2025-02-19T15:09:59.600Z
6747329d228a652d5707e094
/avatars/f33e118950e329ce5612877413806e49.svg
GUIXIAN XU
user
Stuart-Xu
67b55321f703732d151de669
Jianing Liu
67b55321f703732d151de66a
XU Han
67b55321f703732d151de66b
Ting Zhang
67b55321f703732d151de66c
Yushuang Dong
Multilingual Encoder Knows more than You Realize: Shared Weights
  Pretraining for Extremely Low-Resource Languages
While multilingual language models like XLM-R have advanced multilingualism
in NLP, they still perform poorly in extremely low-resource languages. This
situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen
support far fewer languages than XLM-R, making text generation models
non-existent for many languages in the world. To tackle this challenge, we
propose a novel framework for adapting multilingual encoders to text generation
in extremely low-resource languages. By reusing the weights between the encoder
and the decoder, our framework allows the model to leverage the learned
semantic space of the encoder, enabling efficient learning and effective
generalization in low-resource languages. Applying this framework to four
Chinese minority languages, we present XLM-SWCM, and demonstrate its superior
performance on various downstream tasks even when compared with much larger
models.
A framework that reuses weights between encoder and decoder enhances text generation in low-resource languages, demonstrating superior performance on various tasks.
multilingual language models XLM-R LLaMA Qwen multilingual encoders text generation low-resource languages encoder-decoder reuse semantic space efficient learning generalization XLM-SWCM