title: Skywork: A More Open Bilingual Foundation Model
summary: In this technical report, we present Skywork-13B, a family of large language
  models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both
  English and Chinese texts. This bilingual foundation model is the most
  extensively trained and openly published LLMs of comparable size to date. We
  introduce a two-stage training methodology using a segmented corpus, targeting
  general purpose training and then domain-specific enhancement training,
  respectively. We show that our model not only excels on popular benchmarks, but
  also achieves state of the art performance in Chinese language modeling
  on diverse domains. Furthermore, we propose a novel leakage detection method,
  demonstrating that test data contamination is a pressing issue warranting
  further investigation by the LLM community. To spur future research, we release
  Skywork-13B along with checkpoints obtained during intermediate stages of the
  training process. We are also releasing part of our SkyPile corpus, a
  collection of over 150 billion tokens of web text, which is the largest high
  quality open Chinese pre-training corpus to date. We hope Skywork-13B and our
  open corpus will serve as a valuable open-source resource to democratize access
  to high-quality LLMs.
paper_authors._id: 65408e8aff86086e115360b3
paper_authors.name: Tianwen Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T15:04:54.859Z
paper_authors._id: 63d8773a635499f21644601b
paper_authors.avatarUrl: /avatars/07efea204d172e2844dbacc418131da2.svg
paper_authors.fullname: Wei Tianwen
paper_authors.type: user
paper_authors.user: weitianwen
paper_authors._id: 65408e8aff86086e115360b4
paper_authors.name: Liang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:42:15.403Z
paper_authors._id: 64996ce2ed09df36eaf236ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64996ce2ed09df36eaf236ea/wIr1j2zkAezXfpsczTjq_.jpeg
paper_authors.fullname: zhaoliang
paper_authors.type: user
paper_authors.user: zhao1iang
paper_authors._id: 65408e8aff86086e115360b5
paper_authors.name: Lichang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:00:10.939Z
paper_authors._id: 63f3396b4745321de34ed545
paper_authors.avatarUrl: /avatars/f53c05b6ff1d59caeaf092853c04baf8.svg
paper_authors.fullname: lichang zhang
paper_authors.type: user
paper_authors.user: liczhang
paper_authors._id: 65408e8aff86086e115360b6
paper_authors.name: Bo Zhu
paper_authors._id: 65408e8aff86086e115360b7
paper_authors.name: Lijie Wang
paper_authors._id: 65408e8aff86086e115360b8
paper_authors.name: Haihua Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:00:54.659Z
paper_authors._id: 653f24f4cd52f288490775e6
paper_authors.avatarUrl: /avatars/4938e31c7dcbee80861ce948702e831a.svg
paper_authors.fullname: yanghaihua
paper_authors.type: user
paper_authors.user: yanghaihua
paper_authors._id: 65408e8aff86086e115360b9
paper_authors.name: Biye Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:01:09.880Z
paper_authors._id: 636a1719f8d9af4aea17095a
paper_authors.avatarUrl: /avatars/5eb27d368aa18f242e91e2d6ed1c60cf.svg
paper_authors.fullname: Biye Li
paper_authors.type: user
paper_authors.user: dugusword
paper_authors._id: 65408e8aff86086e115360ba
paper_authors.name: Cheng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:31:22.366Z
paper_authors._id: 64535b71bcbd25618f7655da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64535b71bcbd25618f7655da/vDc8nrYQd4rpYgXLWOuIx.jpeg
paper_authors.fullname: cheng cheng
paper_authors.type: user
paper_authors.user: chengtbf
paper_authors._id: 65408e8aff86086e115360bb
paper_authors.name: Weiwei Lü
paper_authors._id: 65408e8aff86086e115360bc
paper_authors.name: Rui Hu
paper_authors._id: 65408e8aff86086e115360bd
paper_authors.name: Chenxia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:02:24.788Z
paper_authors._id: 62cec5bea3a23014aca6c3ee
paper_authors.avatarUrl: /avatars/b15f262fad2d146481df4328ee42f78b.svg
paper_authors.fullname: lichenxia
paper_authors.type: user
paper_authors.user: lichenxia
paper_authors._id: 65408e8aff86086e115360be
paper_authors.name: Liu Yang
paper_authors._id: 65408e8aff86086e115360bf
paper_authors.name: Xilin Luo
paper_authors._id: 65408e8aff86086e115360c0
paper_authors.name: Xuejie Wu
paper_authors._id: 65408e8aff86086e115360c1
paper_authors.name: Lunan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:13:20.667Z
paper_authors._id: 624eabc864599731f224e87b
paper_authors.avatarUrl: /avatars/c640d4168c359f6f75ffa50b60fb4d7b.svg
paper_authors.fullname: Lunan LIU
paper_authors.type: user
paper_authors.user: rangnide
paper_authors._id: 65408e8aff86086e115360c2
paper_authors.name: Wenjun Cheng
paper_authors._id: 65408e8aff86086e115360c3
paper_authors.name: Peng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T09:24:32.179Z
paper_authors._id: 636b8f4429668f04eaf5d015
paper_authors.avatarUrl: /avatars/84b6209914cb83af7cfce2ffb337d7bd.svg
paper_authors.fullname: CPFLAME
paper_authors.type: user
paper_authors.user: CPFLAME
paper_authors._id: 65408e8aff86086e115360c4
paper_authors.name: Jianhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:13:53.447Z
paper_authors._id: 64235adf23b98d5ee6e9ca85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64235adf23b98d5ee6e9ca85/m_7uwLaAkTXyt3qB2pPLA.png
paper_authors.fullname: Jianhao Zhang
paper_authors.type: user
paper_authors.user: Heerozh
paper_authors._id: 65408e8aff86086e115360c5
paper_authors.name: Xiaoyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T11:51:28.972Z
paper_authors._id: 63647823a1503c61efd4fbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63647823a1503c61efd4fbcb/Df5U59Imobmr9S8wDf77I.png
paper_authors.fullname: Yi Wang
paper_authors.type: user
paper_authors.user: BBuf
paper_authors._id: 65408e8aff86086e115360c6
paper_authors.name: Lei Lin
paper_authors._id: 65408e8aff86086e115360c7
paper_authors.name: Xiaokun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:13:03.978Z
paper_authors._id: 62be9b5aae56e75e4d689e7c
paper_authors.avatarUrl: /avatars/6772bc09d6eeb4e86b1210481be91720.svg
paper_authors.fullname: wangxiaokun
paper_authors.type: user
paper_authors.user: shawn0wang
paper_authors._id: 65408e8aff86086e115360c8
paper_authors.name: Yutuan Ma
paper_authors._id: 65408e8aff86086e115360c9
paper_authors.name: Chuanhai Dong
paper_authors._id: 65408e8aff86086e115360ca
paper_authors.name: Yanqi Sun
paper_authors._id: 65408e8aff86086e115360cb
paper_authors.name: Yifu Chen
paper_authors._id: 65408e8aff86086e115360cc
paper_authors.name: Yongyi Peng
paper_authors._id: 65408e8aff86086e115360cd
paper_authors.name: Xiaojuan Liang
paper_authors._id: 65408e8aff86086e115360ce
paper_authors.name: Shuicheng Yan
paper_authors._id: 65408e8aff86086e115360cf
paper_authors.name: Han Fang
paper_authors._id: 65408e8aff86086e115360d0
paper_authors.name: Yahui Zhou
paper_title: Skywork: A More Open Bilingual Foundation Model
paper_summary: In this technical report, we present Skywork-13B, a family of large language
  models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both
  English and Chinese texts. This bilingual foundation model is the most
  extensively trained and openly published LLMs of comparable size to date. We
  introduce a two-stage training methodology using a segmented corpus, targeting
  general purpose training and then domain-specific enhancement training,
  respectively. We show that our model not only excels on popular benchmarks, but
  also achieves state of the art performance in Chinese language modeling
  on diverse domains. Furthermore, we propose a novel leakage detection method,
  demonstrating that test data contamination is a pressing issue warranting
  further investigation by the LLM community. To spur future research, we release
  Skywork-13B along with checkpoints obtained during intermediate stages of the
  training process. We are also releasing part of our SkyPile corpus, a
  collection of over 150 billion tokens of web text, which is the largest high
  quality open Chinese pre-training corpus to date. We hope Skywork-13B and our
  open corpus will serve as a valuable open-source resource to democratize access
  to high-quality LLMs.
paper_ai_summary: Skywork-13B, a bilingual LLM trained on extensive corpora, excels in benchmarks and achieves state-of-the-art performance in Chinese language modeling, introducing a novel leakage detection method.
paper_ai_keywords: large language models LLMs segmented corpus general purpose training domain-specific enhancement training state of the art Chinese language modeling leakage detection method SkyPile corpus