title: Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting
    LLMs Across Languages and Resources
summary: Large Language Models (LLMs) exhibit significant disparities in performance
  across languages, primarily benefiting high-resource languages while
  marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as
  a promising approach to address this imbalance, although the relative
  effectiveness of monolingual, bilingual, and code-augmented data strategies
  remains unclear. This study systematically evaluates 36 CPT configurations
  involving three multilingual base models, across 30+ languages categorized as
  altruistic, selfish, and stagnant, spanning various resource levels. Our
  findings reveal three major insights: (1) Bilingual CPT improves multilingual
  classification but often causes language mixing issues during generation. (2)
  Including programming code data during CPT consistently enhances multilingual
  classification accuracy, particularly benefiting low-resource languages, but
  introduces a trade-off by slightly degrading generation quality. (3) Contrary
  to prior work, we observe substantial deviations from language classifications
  according to their impact on cross-lingual transfer: Languages classified as
  altruistic often negatively affect related languages, selfish languages show
  conditional and configuration-dependent behavior, and stagnant languages
  demonstrate surprising adaptability under certain CPT conditions. These nuanced
  interactions emphasize the complexity of multilingual representation learning,
  underscoring the importance of systematic studies on generalizable language
  classification to inform future multilingual CPT strategies.
paper_authors._id: 67f50d48e848388899ffb867
paper_authors.name: Zihao Li
paper_authors._id: 67f50d48e848388899ffb868
paper_authors.name: Shaoxiong Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:51:54.077Z
paper_authors._id: 617a92e16f37340367d5d791
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
paper_authors.fullname: Shaoxiong
paper_authors.type: user
paper_authors.user: jisx
paper_authors._id: 67f50d48e848388899ffb869
paper_authors.name: Hengyu Luo
paper_authors._id: 67f50d48e848388899ffb86a
paper_authors.name: Jörg Tiedemann
paper_title: Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting
    LLMs Across Languages and Resources
paper_summary: Large Language Models (LLMs) exhibit significant disparities in performance
  across languages, primarily benefiting high-resource languages while
  marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as
  a promising approach to address this imbalance, although the relative
  effectiveness of monolingual, bilingual, and code-augmented data strategies
  remains unclear. This study systematically evaluates 36 CPT configurations
  involving three multilingual base models, across 30+ languages categorized as
  altruistic, selfish, and stagnant, spanning various resource levels. Our
  findings reveal three major insights: (1) Bilingual CPT improves multilingual
  classification but often causes language mixing issues during generation. (2)
  Including programming code data during CPT consistently enhances multilingual
  classification accuracy, particularly benefiting low-resource languages, but
  introduces a trade-off by slightly degrading generation quality. (3) Contrary
  to prior work, we observe substantial deviations from language classifications
  according to their impact on cross-lingual transfer: Languages classified as
  altruistic often negatively affect related languages, selfish languages show
  conditional and configuration-dependent behavior, and stagnant languages
  demonstrate surprising adaptability under certain CPT conditions. These nuanced
  interactions emphasize the complexity of multilingual representation learning,
  underscoring the importance of systematic studies on generalizable language
  classification to inform future multilingual CPT strategies.
paper_ai_summary: Continual pretraining strategies involving monolingual, bilingual, and code-augmented data show varied effectiveness across languages, impacting classification and generation tasks differently.
paper_ai_keywords: Continual Pretraining (CPT) multilingual base models multilingual classification language mixing generation quality altruistic languages selfish languages stagnant languages cross-lingual transfer generalizable language classification