  LLMs such as ChatGPT and PaLM can be utilized to train on a new language and
revitalize low-resource languages. However, it is evidently very costly to
pretrain pr fine-tune LLMs to adopt new languages. Another challenge is the
limitation of benchmark datasets and the metrics used to measure the
performance of models in multilingual settings. This paper proposes
cost-effective solutions to both of the aforementioned challenges. We introduce
the Multilingual Instruction-Tuning Dataset (MITS), which is comprised of the
translation of Alpaca-52K, Dolly-15K, and Vicuna Benchmark in 132 languages.
Also, we propose a new method called \emph{TaCo: Translation-Assisted
Cross-Linguality}, which make uses of translation in a chain-of-thought process
to instruction-tune LLMs on a new languages through a curriculum learning
process. As a proof of concept, we experimented with the instruction-tuned
Guanaco-33B model and performed further instruction tuning using the TaCo
method in three low-resource languages and one high-resource language. Our
results show that the TaCo method impresses the GPT-4 with 82% for a
low-resource language in the Vicuna Benchmark dataset, and boosts performance
by double in contrast to the performance of instruction tuning only. Our
results show that TaCo is a promising method for creating multilingual LLMs,
even for low-resource languages. We have released our datasets and the model
adapters, and encourage the research community to make use of these resources
towards advancing work on multilingual LLMs.

Computation and Language;Artificial Intelligence