Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer
<|reference_start|>Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer: This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named \textsc{Llammas}, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.<|reference_end|>
@article{kuulmets2024teaching,
    title={Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer},
    author={Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, Mark Fishel},
    journal={Findings of the Association for Computational Linguistics: NAACL
  2024, pages 3309-3325},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.04042},
    primaryClass={cs.CL}
}