Massively Multilingual Adaptation of Large Language Models Using
  Bilingual Translation Data
This paper investigates a critical design decision in the practice of
massively multilingual continual pre-training -- the inclusion of parallel
data. Specifically, we study the impact of bilingual translation data for
massively multilingual language adaptation of the Llama3 family of models to
500 languages. To this end, we construct the MaLA bilingual translation corpus,
containing data from more than 2,500 language pairs. Subsequently, we develop
the EMMA-500 Llama 3 suite of four massively multilingual models -- continually
pre-trained from the Llama 3 family of base models extensively on diverse data
mixes up to 671B tokens -- and explore the effect of continual pre-training
with or without bilingual translation data. Comprehensive evaluation across 7
tasks and 12 benchmarks demonstrates that bilingual data tends to enhance
language transfer and performance, particularly for low-resource languages. We
open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model
generations.
683e9e0a1c5320ac91b85a19
Shaoxiong Ji
claimed_verified
2025-06-03T08:37:04.382Z
617a92e16f37340367d5d791
https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
Shaoxiong
user
jisx
683e9e0a1c5320ac91b85a1a
Zihao Li
683e9e0a1c5320ac91b85a1b
Jaakko Paavola
admin_assigned
2025-06-03T12:43:07.193Z
6611225516a9b10274a2ea98
/avatars/60b90a165673ba3fa0736d3c48108fb3.svg
Jaakko Paavola
user
JaakkoP
683e9e0a1c5320ac91b85a1c
Indraneil Paul
admin_assigned
2025-06-03T12:42:57.237Z
62054c0b522e40b4a18d8744
https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
Indraneil Paul
user
iNeil77
683e9e0a1c5320ac91b85a1d
Hengyu Luo
admin_assigned
2025-06-03T12:42:49.625Z
64de06e5a8b70ab6b35a7c89
/avatars/45f287a9c07d3fe04c0782a6a99e9743.svg
Hengyu Luo
user
hengyu-luo
683e9e0a1c5320ac91b85a1e
Jörg Tiedemann
admin_assigned
2025-06-03T12:42:25.922Z
5eac04968a595438195ab4a6
/avatars/0223584657947ca7393ba46d211d6970.svg
Jörg Tiedemann
user
tiedeman
Massively Multilingual Adaptation of Large Language Models Using
  Bilingual Translation Data
This paper investigates a critical design decision in the practice of
massively multilingual continual pre-training -- the inclusion of parallel
data. Specifically, we study the impact of bilingual translation data for
massively multilingual language adaptation of the Llama3 family of models to
500 languages. To this end, we construct the MaLA bilingual translation corpus,
containing data from more than 2,500 language pairs. Subsequently, we develop
the EMMA-500 Llama 3 suite of four massively multilingual models -- continually
pre-trained from the Llama 3 family of base models extensively on diverse data
mixes up to 671B tokens -- and explore the effect of continual pre-training
with or without bilingual translation data. Comprehensive evaluation across 7
tasks and 12 benchmarks demonstrates that bilingual data tends to enhance
language transfer and performance, particularly for low-resource languages. We
open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model
generations.
Bilingual translation data enhances language transfer and performance in massively multilingual language adaptation of the Llama3 family of models.
massively multilingual continual pre-training bilingual translation data Llama3 MaLA bilingual translation corpus EMMA-500 Llama 3 suite continual pre-training language transfer low-resource languages