title: SambaLingo: Teaching Large Language Models New Languages
summary: Despite the widespread availability of LLMs, there remains a substantial gap
  in their capabilities and availability across diverse languages. One approach
  to address these issues has been to take an existing pre-trained LLM and
  continue to train it on new languages. While prior works have experimented with
  language adaptation, many questions around best practices and methodology have
  not been covered. In this paper, we present a comprehensive investigation into
  the adaptation of LLMs to new languages. Our study covers the key components in
  this process, including vocabulary extension, direct preference optimization
  and the data scarcity problem for human alignment in low-resource languages. We
  scale these experiments across 9 languages and 2 parameter scales (7B and 70B).
  We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing
  language experts, outperforming all prior published baselines. Additionally,
  all evaluation code and checkpoints are made public to facilitate future
  research.
paper_authors._id: 6615f38116917dfdc679328b
paper_authors.name: Zoltan Csaki
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-10T02:03:46.864Z
paper_authors._id: 645ba903ca5d8a2977148389
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/zxdZvpuAP6qEhk3vyRO3_.jpeg
paper_authors.fullname: Zoltan Csaki
paper_authors.type: user
paper_authors.user: zolicsaki
paper_authors._id: 6615f38116917dfdc679328c
paper_authors.name: Bo Li
paper_authors._id: 6615f38116917dfdc679328d
paper_authors.name: Jonathan Li
paper_authors._id: 6615f38116917dfdc679328e
paper_authors.name: Qiantong Xu
paper_authors._id: 6615f38116917dfdc679328f
paper_authors.name: Pian Pawakapan
paper_authors._id: 6615f38116917dfdc6793290
paper_authors.name: Leon Zhang
paper_authors._id: 6615f38116917dfdc6793291
paper_authors.name: Yun Du
paper_authors._id: 6615f38116917dfdc6793292
paper_authors.name: Hengyu Zhao
paper_authors._id: 6615f38116917dfdc6793293
paper_authors.name: Changran Hu
paper_authors._id: 6615f38116917dfdc6793294
paper_authors.name: Urmish Thakker
paper_title: SambaLingo: Teaching Large Language Models New Languages
paper_summary: Despite the widespread availability of LLMs, there remains a substantial gap
  in their capabilities and availability across diverse languages. One approach
  to address these issues has been to take an existing pre-trained LLM and
  continue to train it on new languages. While prior works have experimented with
  language adaptation, many questions around best practices and methodology have
  not been covered. In this paper, we present a comprehensive investigation into
  the adaptation of LLMs to new languages. Our study covers the key components in
  this process, including vocabulary extension, direct preference optimization
  and the data scarcity problem for human alignment in low-resource languages. We
  scale these experiments across 9 languages and 2 parameter scales (7B and 70B).
  We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing
  language experts, outperforming all prior published baselines. Additionally,
  all evaluation code and checkpoints are made public to facilitate future
  research.
paper_ai_summary: This investigation explores comprehensive techniques for adapting large language models to new languages, addressing vocabulary extension, preference optimization, and data scarcity issues across multiple languages and model sizes.
paper_ai_keywords: vocabulary extension direct preference optimization data scarcity human alignment low-resource languages language adaptation large language models