title: LLaMA Beyond English: An Empirical Study on Language Capability Transfer
summary: In recent times, substantial advancements have been witnessed in large
  language models (LLMs), exemplified by ChatGPT, showcasing remarkable
  proficiency across a range of complex tasks. However, many mainstream LLMs
  (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their
  performance in other non-English languages. In this paper, we focus on how to
  effectively transfer the capabilities of language generation and following
  instructions to a non-English language. To answer this question, we conduct an
  extensive empirical investigation based on LLaMA, accumulating over 1440 GPU
  hours. We analyze the impact of key factors such as vocabulary extension,
  further pretraining, and instruction tuning on transfer. To accurately assess
  the model's level of knowledge, we employ four widely used standardized testing
  benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a
  comprehensive evaluation of the model's response quality is conducted,
  considering aspects such as accuracy, fluency, informativeness, logical
  coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting
  instruction tasks from 17 diverse categories. Our evaluation results
  demonstrate that comparable performance to state-of-the-art transfer models can
  be achieved with less than 1% of the pretraining data, both in terms of
  knowledge alignment and response quality. Furthermore, the experimental
  outcomes across the thirteen low-resource languages also exhibit similar
  trends. We anticipate that the conclusions revealed by the experiments will aid
  the community in developing non-English LLMs.
paper_authors._id: 6594c64e1adf6d577e5a3a11
paper_authors.name: Jun Zhao
paper_authors._id: 6594c64e1adf6d577e5a3a12
paper_authors.name: Zhihao Zhang
paper_authors._id: 6594c64e1adf6d577e5a3a13
paper_authors.name: Qi Zhang
paper_authors._id: 6594c64e1adf6d577e5a3a14
paper_authors.name: Tao Gui
paper_authors._id: 6594c64e1adf6d577e5a3a15
paper_authors.name: Xuanjing Huang
paper_title: LLaMA Beyond English: An Empirical Study on Language Capability Transfer
paper_summary: In recent times, substantial advancements have been witnessed in large
  language models (LLMs), exemplified by ChatGPT, showcasing remarkable
  proficiency across a range of complex tasks. However, many mainstream LLMs
  (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their
  performance in other non-English languages. In this paper, we focus on how to
  effectively transfer the capabilities of language generation and following
  instructions to a non-English language. To answer this question, we conduct an
  extensive empirical investigation based on LLaMA, accumulating over 1440 GPU
  hours. We analyze the impact of key factors such as vocabulary extension,
  further pretraining, and instruction tuning on transfer. To accurately assess
  the model's level of knowledge, we employ four widely used standardized testing
  benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a
  comprehensive evaluation of the model's response quality is conducted,
  considering aspects such as accuracy, fluency, informativeness, logical
  coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting
  instruction tasks from 17 diverse categories. Our evaluation results
  demonstrate that comparable performance to state-of-the-art transfer models can
  be achieved with less than 1% of the pretraining data, both in terms of
  knowledge alignment and response quality. Furthermore, the experimental
  outcomes across the thirteen low-resource languages also exhibit similar
  trends. We anticipate that the conclusions revealed by the experiments will aid
  the community in developing non-English LLMs.
paper_ai_summary: A comprehensive study on transferring English-dominant LLMs to non-English languages using LLaMA demonstrates that comparable performance can be achieved with minimal pretraining data across various language capabilities and response quality metrics.
paper_ai_keywords: large language models LLMs ChatGPT LLaMA vocabulary extension further pretraining instruction tuning C-Eval MMLU AGI-Eval GAOKAO-Bench LLM-Eval non-English languages low-resource languages