title: Multilingual Instruction Tuning With Just a Pinch of Multilinguality
summary: As instruction-tuned large language models (LLMs) gain global adoption, their
  ability to follow instructions in multiple languages becomes increasingly
  crucial. One promising approach is cross-lingual transfer, where a model
  acquires specific functionality on some language by finetuning on another
  language. In this work, we investigate how multilinguality during instruction
  tuning of a multilingual LLM affects instruction-following across languages. We
  first show that many languages transfer some instruction-following capabilities
  to other languages from even monolingual tuning. Furthermore, we find that only
  40 multilingual examples in an English tuning set substantially improve
  multilingual instruction-following, both in seen and unseen languages during
  tuning. In general, we observe that models tuned on multilingual mixtures
  exhibit comparable or superior performance in several languages compared to
  monolingually tuned models, despite training on 10x fewer examples in those
  languages. Finally, we find that increasing the number of languages in the
  instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual
  generalization. Our results suggest that building massively multilingual
  instruction-tuned models can be done with only a very small set of multilingual
  instruction-responses.
paper_authors._id: 659614d450a303befb4668bf
paper_authors.name: Uri Shaham
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-11T11:36:10.741Z
paper_authors._id: 61265535ed8ae1abb08326d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629996654743-61265535ed8ae1abb08326d4.jpeg
paper_authors.fullname: Uri Shaham
paper_authors.type: user
paper_authors.user: Uri
paper_authors._id: 659614d450a303befb4668c0
paper_authors.name: Jonathan Herzig
paper_authors._id: 659614d450a303befb4668c1
paper_authors.name: Roee Aharoni
paper_authors._id: 659614d450a303befb4668c2
paper_authors.name: Idan Szpektor
paper_authors._id: 659614d450a303befb4668c3
paper_authors.name: Reut Tsarfaty
paper_authors._id: 659614d450a303befb4668c4
paper_authors.name: Matan Eyal
paper_title: Multilingual Instruction Tuning With Just a Pinch of Multilinguality
paper_summary: As instruction-tuned large language models (LLMs) gain global adoption, their
  ability to follow instructions in multiple languages becomes increasingly
  crucial. One promising approach is cross-lingual transfer, where a model
  acquires specific functionality on some language by finetuning on another
  language. In this work, we investigate how multilinguality during instruction
  tuning of a multilingual LLM affects instruction-following across languages. We
  first show that many languages transfer some instruction-following capabilities
  to other languages from even monolingual tuning. Furthermore, we find that only
  40 multilingual examples in an English tuning set substantially improve
  multilingual instruction-following, both in seen and unseen languages during
  tuning. In general, we observe that models tuned on multilingual mixtures
  exhibit comparable or superior performance in several languages compared to
  monolingually tuned models, despite training on 10x fewer examples in those
  languages. Finally, we find that increasing the number of languages in the
  instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual
  generalization. Our results suggest that building massively multilingual
  instruction-tuned models can be done with only a very small set of multilingual
  instruction-responses.
paper_ai_keywords: instruction-tuned large language models (LLMs) cross-lingual transfer multilinguality instruction-following monolingual tuning multilingual instruction-following multilingual mixtures cross-lingual generalization massively multilingual instruction-tuned models