Dynamic data sampler for cross-language transfer learning in large
  language models
Large Language Models (LLMs) have gained significant attention in the field
of natural language processing (NLP) due to their wide range of applications.
However, training LLMs for languages other than English poses significant
challenges, due to the difficulty in acquiring large-scale corpus and the
requisite computing resources. In this paper, we propose ChatFlow, a
cross-language transfer-based LLM, to address these challenges and train large
Chinese language models in a cost-effective manner. We employ a mix of Chinese,
English, and parallel corpus to continuously train the LLaMA2 model, aiming to
align cross-language representations and facilitate the knowledge transfer
specifically to the Chinese language model. In addition, we use a dynamic data
sampler to progressively transition the model from unsupervised pre-training to
supervised fine-tuning. Experimental results demonstrate that our approach
accelerates model convergence and achieves superior performance. We evaluate
ChatFlow on popular Chinese and English benchmarks, the results indicate that
it outperforms other Chinese models post-trained on LLaMA-2-7B.
664b675698abf89a9c6a90cf
Yudong Li
664b675698abf89a9c6a90d0
Yuhao Feng
664b675698abf89a9c6a90d1
Wen Zhou
664b675698abf89a9c6a90d2
Zhe Zhao
664b675698abf89a9c6a90d3
Linlin Shen
664b675698abf89a9c6a90d4
Cheng Hou
664b675698abf89a9c6a90d5
Xianxu Hou
admin_assigned
2024-05-21T09:38:35.905Z
633643d2d184e6b53c4e11a1
/avatars/4b19d7acdf84a04f186b9a08c715a464.svg
Hou
user
houxianxu
Dynamic data sampler for cross-language transfer learning in large
  language models
Large Language Models (LLMs) have gained significant attention in the field
of natural language processing (NLP) due to their wide range of applications.
However, training LLMs for languages other than English poses significant
challenges, due to the difficulty in acquiring large-scale corpus and the
requisite computing resources. In this paper, we propose ChatFlow, a
cross-language transfer-based LLM, to address these challenges and train large
Chinese language models in a cost-effective manner. We employ a mix of Chinese,
English, and parallel corpus to continuously train the LLaMA2 model, aiming to
align cross-language representations and facilitate the knowledge transfer
specifically to the Chinese language model. In addition, we use a dynamic data
sampler to progressively transition the model from unsupervised pre-training to
supervised fine-tuning. Experimental results demonstrate that our approach
accelerates model convergence and achieves superior performance. We evaluate
ChatFlow on popular Chinese and English benchmarks, the results indicate that
it outperforms other Chinese models post-trained on LLaMA-2-7B.
ChatFlow, a cross-language transfer-based LLM using Chinese, English, and parallel corpus, enhances training efficiency and performance for Chinese language models leveraging LLaMA2.
Large Language Models natural language processing LLMs NLP ChatFlow cross-language transfer LLaMA2 model unsupervised pre-training supervised fine-tuning model convergence