title: MaLA-500: Massive Language Adaptation of Large Language Models
summary: Large language models have advanced the state of the art in natural language
  processing. However, their predominant design for English or a limited set of
  languages creates a substantial gap in their effectiveness for low-resource
  languages. To bridge this gap, we introduce MaLA-500, a novel large language
  model designed to cover an extensive range of 534 languages. To train MaLA-500,
  we employ vocabulary extension and continued pretraining on LLaMA 2 with
  Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves
  state-of-the-art in-context learning results. We release MaLA-500 at
  https://huggingface.co/MaLA-LM
paper_authors._id: 65b1e894b4225e3e42341d1b
paper_authors.name: Peiqin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:18:16.574Z
paper_authors._id: 6347e493e3f2de18545cbfe8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6347e493e3f2de18545cbfe8/_Y068r6Qh7QxqqntRDqC-.jpeg
paper_authors.fullname: Peiqin Lin
paper_authors.type: user
paper_authors.user: lpq29743
paper_authors._id: 65b1e894b4225e3e42341d1c
paper_authors.name: Shaoxiong Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:20:12.959Z
paper_authors._id: 617a92e16f37340367d5d791
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
paper_authors.fullname: Shaoxiong
paper_authors.type: user
paper_authors.user: jisx
paper_authors._id: 65b1e894b4225e3e42341d1d
paper_authors.name: Jörg Tiedemann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:18:40.533Z
paper_authors._id: 5eac04968a595438195ab4a6
paper_authors.avatarUrl: /avatars/0223584657947ca7393ba46d211d6970.svg
paper_authors.fullname: Jörg Tiedemann
paper_authors.type: user
paper_authors.user: tiedeman
paper_authors._id: 65b1e894b4225e3e42341d1e
paper_authors.name: André F. T. Martins
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:22:09.339Z
paper_authors._id: 6326de72a28c096b45a8f8da
paper_authors.avatarUrl: /avatars/94c89f31ab5eee80c95b5bbf6688c05f.svg
paper_authors.fullname: Andre Martins
paper_authors.type: user
paper_authors.user: andre-martins
paper_authors._id: 65b1e894b4225e3e42341d1f
paper_authors.name: Hinrich Schütze
paper_title: MaLA-500: Massive Language Adaptation of Large Language Models
paper_summary: Large language models have advanced the state of the art in natural language
  processing. However, their predominant design for English or a limited set of
  languages creates a substantial gap in their effectiveness for low-resource
  languages. To bridge this gap, we introduce MaLA-500, a novel large language
  model designed to cover an extensive range of 534 languages. To train MaLA-500,
  we employ vocabulary extension and continued pretraining on LLaMA 2 with
  Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves
  state-of-the-art in-context learning results. We release MaLA-500 at
  https://huggingface.co/MaLA-LM
paper_ai_keywords: large language models natural language processing low-resource languages MaLA-500 vocabulary extension continued pretraining LLaMA 2 Glot500-c SIB-200 in-context learning