title: Massively Multilingual Adaptation of Large Language Models Using
    Bilingual Translation Data
summary: This paper investigates a critical design decision in the practice of
  massively multilingual continual pre-training -- the inclusion of parallel
  data. Specifically, we study the impact of bilingual translation data for
  massively multilingual language adaptation of the Llama3 family of models to
  500 languages. To this end, we construct the MaLA bilingual translation corpus,
  containing data from more than 2,500 language pairs. Subsequently, we develop
  the EMMA-500 Llama 3 suite of four massively multilingual models -- continually
  pre-trained from the Llama 3 family of base models extensively on diverse data
  mixes up to 671B tokens -- and explore the effect of continual pre-training
  with or without bilingual translation data. Comprehensive evaluation across 7
  tasks and 12 benchmarks demonstrates that bilingual data tends to enhance
  language transfer and performance, particularly for low-resource languages. We
  open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model
  generations.
paper_authors._id: 683e9e0a1c5320ac91b85a19
paper_authors.name: Shaoxiong Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:37:04.382Z
paper_authors._id: 617a92e16f37340367d5d791
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
paper_authors.fullname: Shaoxiong
paper_authors.type: user
paper_authors.user: jisx
paper_authors._id: 683e9e0a1c5320ac91b85a1a
paper_authors.name: Zihao Li
paper_authors._id: 683e9e0a1c5320ac91b85a1b
paper_authors.name: Jaakko Paavola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:43:07.193Z
paper_authors._id: 6611225516a9b10274a2ea98
paper_authors.avatarUrl: /avatars/60b90a165673ba3fa0736d3c48108fb3.svg
paper_authors.fullname: Jaakko Paavola
paper_authors.type: user
paper_authors.user: JaakkoP
paper_authors._id: 683e9e0a1c5320ac91b85a1c
paper_authors.name: Indraneil Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:42:57.237Z
paper_authors._id: 62054c0b522e40b4a18d8744
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
paper_authors.fullname: Indraneil Paul
paper_authors.type: user
paper_authors.user: iNeil77
paper_authors._id: 683e9e0a1c5320ac91b85a1d
paper_authors.name: Hengyu Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:42:49.625Z
paper_authors._id: 64de06e5a8b70ab6b35a7c89
paper_authors.avatarUrl: /avatars/45f287a9c07d3fe04c0782a6a99e9743.svg
paper_authors.fullname: Hengyu Luo
paper_authors.type: user
paper_authors.user: zuenmin
paper_authors._id: 683e9e0a1c5320ac91b85a1e
paper_authors.name: Jörg Tiedemann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:42:25.922Z
paper_authors._id: 5eac04968a595438195ab4a6
paper_authors.avatarUrl: /avatars/0223584657947ca7393ba46d211d6970.svg
paper_authors.fullname: Jörg Tiedemann
paper_authors.type: user
paper_authors.user: tiedeman
paper_title: Massively Multilingual Adaptation of Large Language Models Using
    Bilingual Translation Data
paper_summary: This paper investigates a critical design decision in the practice of
  massively multilingual continual pre-training -- the inclusion of parallel
  data. Specifically, we study the impact of bilingual translation data for
  massively multilingual language adaptation of the Llama3 family of models to
  500 languages. To this end, we construct the MaLA bilingual translation corpus,
  containing data from more than 2,500 language pairs. Subsequently, we develop
  the EMMA-500 Llama 3 suite of four massively multilingual models -- continually
  pre-trained from the Llama 3 family of base models extensively on diverse data
  mixes up to 671B tokens -- and explore the effect of continual pre-training
  with or without bilingual translation data. Comprehensive evaluation across 7
  tasks and 12 benchmarks demonstrates that bilingual data tends to enhance
  language transfer and performance, particularly for low-resource languages. We
  open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model
  generations.
paper_ai_summary: Bilingual translation data enhances language transfer and performance in massively multilingual language adaptation of the Llama3 family of models.
paper_ai_keywords: massively multilingual continual pre-training bilingual translation data Llama3 MaLA bilingual translation corpus EMMA-500 Llama 3 suite continual pre-training language transfer low-resource languages