title: BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
    Deduplication by Introducing a Competitive Large Language Model Baseline
summary: The general capabilities of Large Language Models (LLM) highly rely on the
  composition and selection on extensive pretraining datasets, treated as
  commercial secrets by several institutions. To mitigate this issue, we
  open-source the details of a universally applicable data processing pipeline
  and validate its effectiveness and potential by introducing a competitive LLM
  baseline. Specifically, the data processing pipeline consists of broad
  collection to scale up and reweighting to improve quality. We then pretrain a
  7B model BaichuanSEED with 3T tokens processed by our pipeline without any
  deliberate downstream task-related optimization, followed by an easy but
  effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
  and predictability throughout training and achieves comparable performance on
  comprehensive benchmarks with several commercial advanced large language
  models, such as Qwen1.5 and Llama3. We also conduct several heuristic
  experiments to discuss the potential for further optimization of downstream
  tasks, such as mathematics and coding.
paper_authors._id: 66d008caf1e57767fdf55b89
paper_authors.name: Guosheng Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:21:59.022Z
paper_authors._id: 668d2ff4271bd45dab6a8e71
paper_authors.avatarUrl: /avatars/4e198c35a9f09bbc6551c34148aaf560.svg
paper_authors.fullname: guosheng dong
paper_authors.type: user
paper_authors.user: dongguosheng
paper_authors._id: 66d008caf1e57767fdf55b8a
paper_authors.name: Da Pan
paper_authors._id: 66d008caf1e57767fdf55b8b
paper_authors.name: Yiding Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:26:40.448Z
paper_authors._id: 64896a17238422ccf37975ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nlfoIYrpufqeLeaJLAG71.jpeg
paper_authors.fullname: Yiding Sun
paper_authors.type: user
paper_authors.user: Emanual20
paper_authors._id: 66d008caf1e57767fdf55b8c
paper_authors.name: Shusen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:22:31.178Z
paper_authors._id: 64d9c9bd95cf13a381f6cc1a
paper_authors.avatarUrl: /avatars/ce138298df376511044c69182b6419b9.svg
paper_authors.fullname: zss
paper_authors.type: user
paper_authors.user: zss9606
paper_authors._id: 66d008caf1e57767fdf55b8d
paper_authors.name: Zheng Liang
paper_authors._id: 66d008caf1e57767fdf55b8e
paper_authors.name: Xin Wu
paper_authors._id: 66d008caf1e57767fdf55b8f
paper_authors.name: Yanjun Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:28:16.193Z
paper_authors._id: 64ced7c01720c7a483d0862f
paper_authors.avatarUrl: /avatars/1fae685b7226a0a1c95b2d4993f2ab60.svg
paper_authors.fullname: shenyanjun
paper_authors.type: user
paper_authors.user: zilchshen
paper_authors._id: 66d008caf1e57767fdf55b90
paper_authors.name: Fan Yang
paper_authors._id: 66d008caf1e57767fdf55b91
paper_authors.name: Haoze Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:41.512Z
paper_authors._id: 6436bb0dd58a5ea528c55acb
paper_authors.avatarUrl: /avatars/df17b66780e14e07bbe4625f068a94ad.svg
paper_authors.fullname: Alvin Sun
paper_authors.type: user
paper_authors.user: AlvinSunYooo
paper_authors._id: 66d008caf1e57767fdf55b92
paper_authors.name: Tianpeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:29:12.964Z
paper_authors._id: 64be4c97ef8c0e42bf448469
paper_authors.avatarUrl: /avatars/a2a45b5f17d3e27c59547312bcf7bb52.svg
paper_authors.fullname: Tianpeng Li
paper_authors.type: user
paper_authors.user: TJU-Tianpengli
paper_authors._id: 66d008caf1e57767fdf55b93
paper_authors.name: Mingan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:49:00.811Z
paper_authors._id: 6415947858a690df103af49f
paper_authors.avatarUrl: /avatars/38aec23b869833bceb25b9250809b419.svg
paper_authors.fullname: lma
paper_authors.type: user
paper_authors.user: lin5547
paper_authors._id: 66d008caf1e57767fdf55b94
paper_authors.name: Jianhua Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:29:27.331Z
paper_authors._id: 66042414ded78d7454b9746e
paper_authors.avatarUrl: /avatars/828feab7015b527b5ac5c9238fa3c7e0.svg
paper_authors.fullname: jianhua Xu
paper_authors.type: user
paper_authors.user: ArthurXu2020
paper_authors._id: 66d008caf1e57767fdf55b95
paper_authors.name: Yufan Zhang
paper_authors._id: 66d008caf1e57767fdf55b96
paper_authors.name: Xiaonan Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:30:02.122Z
paper_authors._id: 64b6b81142134e053233c3c0
paper_authors.avatarUrl: /avatars/5c7455d99a7a2648f77a531c9a71eb98.svg
paper_authors.fullname: Xiaonan Nie
paper_authors.type: user
paper_authors.user: codecaution
paper_authors._id: 66d008caf1e57767fdf55b97
paper_authors.name: Lei Su
paper_authors._id: 66d008caf1e57767fdf55b98
paper_authors.name: Bingning Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T15:15:28.995Z
paper_authors._id: 640dd3e0c364a086c6322ad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dd3e0c364a086c6322ad2/z11CvZcbWGSRtWXzvem6I.png
paper_authors.fullname: Daniel Wang
paper_authors.type: user
paper_authors.user: DanielWang
paper_authors._id: 66d008caf1e57767fdf55b99
paper_authors.name: Wentao Zhang
paper_authors._id: 66d008caf1e57767fdf55b9a
paper_authors.name: Jiaxin Mao
paper_authors._id: 66d008caf1e57767fdf55b9b
paper_authors.name: Zenan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:31:03.172Z
paper_authors._id: 668d4e50ed63008dfaa78304
paper_authors.avatarUrl: /avatars/80854a3c6b4b7c70cd46694d4cf7296a.svg
paper_authors.fullname: Zenan Zhou
paper_authors.type: user
paper_authors.user: Zenan11
paper_authors._id: 66d008caf1e57767fdf55b9c
paper_authors.name: Weipeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:29:02.819Z
paper_authors._id: 6501587887b370a56ad2608e
paper_authors.avatarUrl: /avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
paper_authors.fullname: Wei-Peng Chen
paper_authors.type: user
paper_authors.user: whenfra
paper_title: BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
    Deduplication by Introducing a Competitive Large Language Model Baseline
paper_summary: The general capabilities of Large Language Models (LLM) highly rely on the
  composition and selection on extensive pretraining datasets, treated as
  commercial secrets by several institutions. To mitigate this issue, we
  open-source the details of a universally applicable data processing pipeline
  and validate its effectiveness and potential by introducing a competitive LLM
  baseline. Specifically, the data processing pipeline consists of broad
  collection to scale up and reweighting to improve quality. We then pretrain a
  7B model BaichuanSEED with 3T tokens processed by our pipeline without any
  deliberate downstream task-related optimization, followed by an easy but
  effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
  and predictability throughout training and achieves comparable performance on
  comprehensive benchmarks with several commercial advanced large language
  models, such as Qwen1.5 and Llama3. We also conduct several heuristic
  experiments to discuss the potential for further optimization of downstream
  tasks, such as mathematics and coding.
paper_ai_keywords: Large Language Models (LLM) pretraining data processing pipeline token BaichuanSEED dual collection reweighting pretrain supervised fine-tuning consistency predictability comprehensive benchmarks Qwen1.5 Llama3 heuristic experiments downstream tasks mathematics coding