SambaLingo: Teaching Large Language Models New Languages
Despite the widespread availability of LLMs, there remains a substantial gap
in their capabilities and availability across diverse languages. One approach
to address these issues has been to take an existing pre-trained LLM and
continue to train it on new languages. While prior works have experimented with
language adaptation, many questions around best practices and methodology have
not been covered. In this paper, we present a comprehensive investigation into
the adaptation of LLMs to new languages. Our study covers the key components in
this process, including vocabulary extension, direct preference optimization
and the data scarcity problem for human alignment in low-resource languages. We
scale these experiments across 9 languages and 2 parameter scales (7B and 70B).
We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing
language experts, outperforming all prior published baselines. Additionally,
all evaluation code and checkpoints are made public to facilitate future
research.
6615f38116917dfdc679328b
Zoltan Csaki
extracted_pending
2024-04-10T02:03:46.864Z
645ba903ca5d8a2977148389
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/zxdZvpuAP6qEhk3vyRO3_.jpeg
Zoltan Csaki
user
zolicsaki
6615f38116917dfdc679328c
Bo Li
6615f38116917dfdc679328d
Jonathan Li
6615f38116917dfdc679328e
Qiantong Xu
6615f38116917dfdc679328f
Pian Pawakapan
6615f38116917dfdc6793290
Leon Zhang
6615f38116917dfdc6793291
Yun Du
6615f38116917dfdc6793292
Hengyu Zhao
6615f38116917dfdc6793293
Changran Hu
6615f38116917dfdc6793294
Urmish Thakker
SambaLingo: Teaching Large Language Models New Languages
Despite the widespread availability of LLMs, there remains a substantial gap
in their capabilities and availability across diverse languages. One approach
to address these issues has been to take an existing pre-trained LLM and
continue to train it on new languages. While prior works have experimented with
language adaptation, many questions around best practices and methodology have
not been covered. In this paper, we present a comprehensive investigation into
the adaptation of LLMs to new languages. Our study covers the key components in
this process, including vocabulary extension, direct preference optimization
and the data scarcity problem for human alignment in low-resource languages. We
scale these experiments across 9 languages and 2 parameter scales (7B and 70B).
We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing
language experts, outperforming all prior published baselines. Additionally,
all evaluation code and checkpoints are made public to facilitate future
research.
This investigation explores comprehensive techniques for adapting large language models to new languages, addressing vocabulary extension, preference optimization, and data scarcity issues across multiple languages and model sizes.
vocabulary extension direct preference optimization data scarcity human alignment low-resource languages language adaptation large language models