title: Multilingual Encoder Knows more than You Realize: Shared Weights
    Pretraining for Extremely Low-Resource Languages
summary: While multilingual language models like XLM-R have advanced multilingualism
  in NLP, they still perform poorly in extremely low-resource languages. This
  situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen
  support far fewer languages than XLM-R, making text generation models
  non-existent for many languages in the world. To tackle this challenge, we
  propose a novel framework for adapting multilingual encoders to text generation
  in extremely low-resource languages. By reusing the weights between the encoder
  and the decoder, our framework allows the model to leverage the learned
  semantic space of the encoder, enabling efficient learning and effective
  generalization in low-resource languages. Applying this framework to four
  Chinese minority languages, we present XLM-SWCM, and demonstrate its superior
  performance on various downstream tasks even when compared with much larger
  models.
paper_authors._id: 67b55321f703732d151de666
paper_authors.name: Zeli Su
paper_authors._id: 67b55321f703732d151de667
paper_authors.name: Ziyin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:10:05.400Z
paper_authors._id: 6430bdd8cd31d174a9f900fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
paper_authors.fullname: Ziyin Zhang
paper_authors.type: user
paper_authors.user: Geralt-Targaryen
paper_authors._id: 67b55321f703732d151de668
paper_authors.name: Guixian Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:09:59.600Z
paper_authors._id: 6747329d228a652d5707e094
paper_authors.avatarUrl: /avatars/f33e118950e329ce5612877413806e49.svg
paper_authors.fullname: GUIXIAN XU
paper_authors.type: user
paper_authors.user: Stuart-Xu
paper_authors._id: 67b55321f703732d151de669
paper_authors.name: Jianing Liu
paper_authors._id: 67b55321f703732d151de66a
paper_authors.name: XU Han
paper_authors._id: 67b55321f703732d151de66b
paper_authors.name: Ting Zhang
paper_authors._id: 67b55321f703732d151de66c
paper_authors.name: Yushuang Dong
paper_title: Multilingual Encoder Knows more than You Realize: Shared Weights
    Pretraining for Extremely Low-Resource Languages
paper_summary: While multilingual language models like XLM-R have advanced multilingualism
  in NLP, they still perform poorly in extremely low-resource languages. This
  situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen
  support far fewer languages than XLM-R, making text generation models
  non-existent for many languages in the world. To tackle this challenge, we
  propose a novel framework for adapting multilingual encoders to text generation
  in extremely low-resource languages. By reusing the weights between the encoder
  and the decoder, our framework allows the model to leverage the learned
  semantic space of the encoder, enabling efficient learning and effective
  generalization in low-resource languages. Applying this framework to four
  Chinese minority languages, we present XLM-SWCM, and demonstrate its superior
  performance on various downstream tasks even when compared with much larger
  models.
paper_ai_summary: A framework that reuses weights between encoder and decoder enhances text generation in low-resource languages, demonstrating superior performance on various tasks.
paper_ai_keywords: multilingual language models XLM-R LLaMA Qwen multilingual encoders text generation low-resource languages encoder-decoder reuse semantic space efficient learning generalization XLM-SWCM