title: Dynamic data sampler for cross-language transfer learning in large
    language models
summary: Large Language Models (LLMs) have gained significant attention in the field
  of natural language processing (NLP) due to their wide range of applications.
  However, training LLMs for languages other than English poses significant
  challenges, due to the difficulty in acquiring large-scale corpus and the
  requisite computing resources. In this paper, we propose ChatFlow, a
  cross-language transfer-based LLM, to address these challenges and train large
  Chinese language models in a cost-effective manner. We employ a mix of Chinese,
  English, and parallel corpus to continuously train the LLaMA2 model, aiming to
  align cross-language representations and facilitate the knowledge transfer
  specifically to the Chinese language model. In addition, we use a dynamic data
  sampler to progressively transition the model from unsupervised pre-training to
  supervised fine-tuning. Experimental results demonstrate that our approach
  accelerates model convergence and achieves superior performance. We evaluate
  ChatFlow on popular Chinese and English benchmarks, the results indicate that
  it outperforms other Chinese models post-trained on LLaMA-2-7B.
paper_authors._id: 664b675698abf89a9c6a90cf
paper_authors.name: Yudong Li
paper_authors._id: 664b675698abf89a9c6a90d0
paper_authors.name: Yuhao Feng
paper_authors._id: 664b675698abf89a9c6a90d1
paper_authors.name: Wen Zhou
paper_authors._id: 664b675698abf89a9c6a90d2
paper_authors.name: Zhe Zhao
paper_authors._id: 664b675698abf89a9c6a90d3
paper_authors.name: Linlin Shen
paper_authors._id: 664b675698abf89a9c6a90d4
paper_authors.name: Cheng Hou
paper_authors._id: 664b675698abf89a9c6a90d5
paper_authors.name: Xianxu Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:38:35.905Z
paper_authors._id: 633643d2d184e6b53c4e11a1
paper_authors.avatarUrl: /avatars/4b19d7acdf84a04f186b9a08c715a464.svg
paper_authors.fullname: Hou
paper_authors.type: user
paper_authors.user: houxianxu
paper_title: Dynamic data sampler for cross-language transfer learning in large
    language models
paper_summary: Large Language Models (LLMs) have gained significant attention in the field
  of natural language processing (NLP) due to their wide range of applications.
  However, training LLMs for languages other than English poses significant
  challenges, due to the difficulty in acquiring large-scale corpus and the
  requisite computing resources. In this paper, we propose ChatFlow, a
  cross-language transfer-based LLM, to address these challenges and train large
  Chinese language models in a cost-effective manner. We employ a mix of Chinese,
  English, and parallel corpus to continuously train the LLaMA2 model, aiming to
  align cross-language representations and facilitate the knowledge transfer
  specifically to the Chinese language model. In addition, we use a dynamic data
  sampler to progressively transition the model from unsupervised pre-training to
  supervised fine-tuning. Experimental results demonstrate that our approach
  accelerates model convergence and achieves superior performance. We evaluate
  ChatFlow on popular Chinese and English benchmarks, the results indicate that
  it outperforms other Chinese models post-trained on LLaMA-2-7B.
paper_ai_keywords: Large Language Models natural language processing (NLP) end-scale corpus computing resources ChatFlow cross-language transfer LLaMA2 model parallel corpus cross-language representations knowledge transfer dynamic data sampler unsupervised pre-training supervised fine-tuning model convergence benchmarks LLaMA-2-7B