title: Ziya2: Data-centric Learning is All LLMs Need
summary: Various large language models (LLMs) have been proposed in recent years,
  including closed- and open-source ones, continually setting new records on
  multiple benchmarks. However, the development of LLMs still faces several
  issues, such as high cost of training models from scratch, and continual
  pre-training leading to catastrophic forgetting, etc. Although many such issues
  are addressed along the line of research on LLMs, an important yet practical
  limitation is that many studies overly pursue enlarging model sizes without
  comprehensively analyzing and optimizing the use of pre-training data in their
  learning process, as well as appropriate organization and leveraging of such
  data in training LLMs under cost-effective settings. In this work, we propose
  Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
  model, and further pre-trained on 700 billion tokens, where we focus on
  pre-training techniques and use data-centric optimization to enhance the
  learning process of Ziya2 on different stages. Experiments show that Ziya2
  significantly outperforms other models in multiple benchmarks especially with
  promising results compared to representative open-source ones. Ziya2 (Base) is
  released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
  https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.
paper_authors._id: 6549b80f44e75a7de4fb301d
paper_authors.name: Ruyi Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:41:15.198Z
paper_authors._id: 612300897d70fa0063f38731
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637806723246-612300897d70fa0063f38731.jpeg
paper_authors.fullname: ganruyi
paper_authors.type: user
paper_authors.user: roygan
paper_authors._id: 6549b80f44e75a7de4fb301e
paper_authors.name: Ziwei Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:41:28.329Z
paper_authors._id: 619f0825678725e00cd9a27f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637812566433-619f0825678725e00cd9a27f.png
paper_authors.fullname: wuziwei
paper_authors.type: user
paper_authors.user: Zimix
paper_authors._id: 6549b80f44e75a7de4fb301f
paper_authors.name: Renliang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:41:36.227Z
paper_authors._id: 653a195b0da86d726c9c580c
paper_authors.avatarUrl: /avatars/61649e1d600fdc1edc50ead0dfa99fdd.svg
paper_authors.fullname: Renliang Sun
paper_authors.type: user
paper_authors.user: RLSNLP
paper_authors._id: 6549b80f44e75a7de4fb3020
paper_authors.name: Junyu Lu
paper_authors._id: 6549b80f44e75a7de4fb3021
paper_authors.name: Xiaojun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T11:43:42.071Z
paper_authors._id: 6352637d0f9bdb641c44e52d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6352637d0f9bdb641c44e52d/mSBRPzcH5pIV68PUmcsHV.png
paper_authors.fullname: wuxiaojun
paper_authors.type: user
paper_authors.user: wuxiaojun
paper_authors._id: 6549b80f44e75a7de4fb3022
paper_authors.name: Dixiang Zhang
paper_authors._id: 6549b80f44e75a7de4fb3023
paper_authors.name: Kunhao Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:42:19.278Z
paper_authors._id: 635262337d071f23d084d8af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666695219265-635262337d071f23d084d8af.jpeg
paper_authors.fullname: Kunhao Pan
paper_authors.type: user
paper_authors.user: pskun
paper_authors._id: 6549b80f44e75a7de4fb3024
paper_authors.name: Ping Yang
paper_authors._id: 6549b80f44e75a7de4fb3025
paper_authors.name: Qi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:42:50.708Z
paper_authors._id: 6215a6efbfcb3893344dd095
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668779559428-6215a6efbfcb3893344dd095.png
paper_authors.fullname: Qi Yang
paper_authors.type: user
paper_authors.user: qiyang
paper_authors._id: 6549b80f44e75a7de4fb3026
paper_authors.name: Jiaxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:42:58.477Z
paper_authors._id: 637854b38f288aba3d01d261
paper_authors.avatarUrl: /avatars/a7f0daf90326bdfea134c75b0bc9bea1.svg
paper_authors.fullname: Jiaxing Zhang
paper_authors.type: user
paper_authors.user: ideazhangjason
paper_authors._id: 6549b80f44e75a7de4fb3027
paper_authors.name: Yan Song
paper_title: Ziya2: Data-centric Learning is All LLMs Need
paper_summary: Various large language models (LLMs) have been proposed in recent years,
  including closed- and open-source ones, continually setting new records on
  multiple benchmarks. However, the development of LLMs still faces several
  issues, such as high cost of training models from scratch, and continual
  pre-training leading to catastrophic forgetting, etc. Although many such issues
  are addressed along the line of research on LLMs, an important yet practical
  limitation is that many studies overly pursue enlarging model sizes without
  comprehensively analyzing and optimizing the use of pre-training data in their
  learning process, as well as appropriate organization and leveraging of such
  data in training LLMs under cost-effective settings. In this work, we propose
  Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
  model, and further pre-trained on 700 billion tokens, where we focus on
  pre-training techniques and use data-centric optimization to enhance the
  learning process of Ziya2 on different stages. Experiments show that Ziya2
  significantly outperforms other models in multiple benchmarks especially with
  promising results compared to representative open-source ones. Ziya2 (Base) is
  released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
  https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.
paper_ai_summary: Ziya2, a 13-billion-parameter language model built on LLaMA2 and further pre-trained on 700 billion tokens, achieves superior performance across multiple benchmarks using data-centric optimization techniques.
paper_ai_keywords: large language models LLMs closed-source open-source catastrophic forgetting pre-training data cost-effective settings data-centric optimization Ziya2 LLaMA2