title: LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation
    Capabilities Beyond 100 Languages
summary: Large Language Models~(LLMs) demonstrate remarkable translation capabilities
  in high-resource language tasks, yet their performance in low-resource
  languages is hindered by insufficient multilingual data during pre-training. To
  address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting
  extensive multilingual continual pre-training on the LLaMA series models,
  enabling translation support across more than 100 languages. Through a
  comprehensive analysis of training strategies, such as vocabulary expansion and
  data augmentation, we develop LLaMAX. Remarkably, without sacrificing its
  generalization ability, LLaMAX achieves significantly higher translation
  performance compared to existing open-source LLMs~(by more than 10 spBLEU
  points) and performs on-par with specialized translation model~(M2M-100-12B) on
  the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve
  as a robust multilingual foundation model. The
  code~\url{https://github.com/CONE-MT/LLaMAX/.} and
  models~\url{https://huggingface.co/LLaMAX/.} are publicly available.
paper_authors._id: 668cab02a249df5a139d0ec1
paper_authors.name: Yinquan Lu
paper_authors._id: 668cab02a249df5a139d0ec2
paper_authors.name: Wenhao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T09:18:40.502Z
paper_authors._id: 649d7d8968586ca9bf7f5fe6
paper_authors.avatarUrl: /avatars/b444240770d4025dea41871cf38126dc.svg
paper_authors.fullname: Wenhao Zhu
paper_authors.type: user
paper_authors.user: Wenhao97
paper_authors._id: 668cab02a249df5a139d0ec3
paper_authors.name: Lei Li
paper_authors._id: 668cab02a249df5a139d0ec4
paper_authors.name: Yu Qiao
paper_authors._id: 668cab02a249df5a139d0ec5
paper_authors.name: Fei Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:27:51.394Z
paper_authors._id: 65fed45b08d35929362dd651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
paper_authors.fullname: FeiYuan
paper_authors.type: user
paper_authors.user: FeYuan
paper_title: LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation
    Capabilities Beyond 100 Languages
paper_summary: Large Language Models~(LLMs) demonstrate remarkable translation capabilities
  in high-resource language tasks, yet their performance in low-resource
  languages is hindered by insufficient multilingual data during pre-training. To
  address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting
  extensive multilingual continual pre-training on the LLaMA series models,
  enabling translation support across more than 100 languages. Through a
  comprehensive analysis of training strategies, such as vocabulary expansion and
  data augmentation, we develop LLaMAX. Remarkably, without sacrificing its
  generalization ability, LLaMAX achieves significantly higher translation
  performance compared to existing open-source LLMs~(by more than 10 spBLEU
  points) and performs on-par with specialized translation model~(M2M-100-12B) on
  the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve
  as a robust multilingual foundation model. The
  code~\url{https://github.com/CONE-MT/LLaMAX/.} and
  models~\url{https://huggingface.co/LLaMAX/.} are publicly available.
paper_ai_summary: LLaMAX enhances multilingual translation by extensive continual pre-training, achieving superior performance in 100+ languages without sacrificing generalization.
paper_ai_keywords: Large Language Models LLama series multilingual continual pre-training vocabulary expansion data augmentation spBLEU Flores-101 benchmark M2M-100-12B multilingual foundation model