Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting
  LLMs Across Languages and Resources
Large Language Models (LLMs) exhibit significant disparities in performance
across languages, primarily benefiting high-resource languages while
marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as
a promising approach to address this imbalance, although the relative
effectiveness of monolingual, bilingual, and code-augmented data strategies
remains unclear. This study systematically evaluates 36 CPT configurations
involving three multilingual base models, across 30+ languages categorized as
altruistic, selfish, and stagnant, spanning various resource levels. Our
findings reveal three major insights: (1) Bilingual CPT improves multilingual
classification but often causes language mixing issues during generation. (2)
Including programming code data during CPT consistently enhances multilingual
classification accuracy, particularly benefiting low-resource languages, but
introduces a trade-off by slightly degrading generation quality. (3) Contrary
to prior work, we observe substantial deviations from language classifications
according to their impact on cross-lingual transfer: Languages classified as
altruistic often negatively affect related languages, selfish languages show
conditional and configuration-dependent behavior, and stagnant languages
demonstrate surprising adaptability under certain CPT conditions. These nuanced
interactions emphasize the complexity of multilingual representation learning,
underscoring the importance of systematic studies on generalizable language
classification to inform future multilingual CPT strategies.
67f50d48e848388899ffb867
Zihao Li
67f50d48e848388899ffb868
Shaoxiong Ji
claimed_verified
2025-06-03T08:51:54.077Z
617a92e16f37340367d5d791
https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
Shaoxiong
user
jisx
67f50d48e848388899ffb869
Hengyu Luo
67f50d48e848388899ffb86a
Jörg Tiedemann
Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting
  LLMs Across Languages and Resources
Large Language Models (LLMs) exhibit significant disparities in performance
across languages, primarily benefiting high-resource languages while
marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as
a promising approach to address this imbalance, although the relative
effectiveness of monolingual, bilingual, and code-augmented data strategies
remains unclear. This study systematically evaluates 36 CPT configurations
involving three multilingual base models, across 30+ languages categorized as
altruistic, selfish, and stagnant, spanning various resource levels. Our
findings reveal three major insights: (1) Bilingual CPT improves multilingual
classification but often causes language mixing issues during generation. (2)
Including programming code data during CPT consistently enhances multilingual
classification accuracy, particularly benefiting low-resource languages, but
introduces a trade-off by slightly degrading generation quality. (3) Contrary
to prior work, we observe substantial deviations from language classifications
according to their impact on cross-lingual transfer: Languages classified as
altruistic often negatively affect related languages, selfish languages show
conditional and configuration-dependent behavior, and stagnant languages
demonstrate surprising adaptability under certain CPT conditions. These nuanced
interactions emphasize the complexity of multilingual representation learning,
underscoring the importance of systematic studies on generalizable language
classification to inform future multilingual CPT strategies.
Continual pretraining strategies involving monolingual, bilingual, and code-augmented data show varied effectiveness across languages, impacting classification and generation tasks differently.
Continual Pretraining (CPT) multilingual base models multilingual classification language mixing generation quality altruistic languages selfish languages stagnant languages cross-lingual transfer generalizable language classification