BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
  Deduplication by Introducing a Competitive Large Language Model Baseline
The general capabilities of Large Language Models (LLM) highly rely on the
composition and selection on extensive pretraining datasets, treated as
commercial secrets by several institutions. To mitigate this issue, we
open-source the details of a universally applicable data processing pipeline
and validate its effectiveness and potential by introducing a competitive LLM
baseline. Specifically, the data processing pipeline consists of broad
collection to scale up and reweighting to improve quality. We then pretrain a
7B model BaichuanSEED with 3T tokens processed by our pipeline without any
deliberate downstream task-related optimization, followed by an easy but
effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
and predictability throughout training and achieves comparable performance on
comprehensive benchmarks with several commercial advanced large language
models, such as Qwen1.5 and Llama3. We also conduct several heuristic
experiments to discuss the potential for further optimization of downstream
tasks, such as mathematics and coding.
66d008caf1e57767fdf55b89
Guosheng Dong
admin_assigned
2024-08-29T07:21:59.022Z
668d2ff4271bd45dab6a8e71
/avatars/4e198c35a9f09bbc6551c34148aaf560.svg
guosheng dong
user
dongguosheng
66d008caf1e57767fdf55b8a
Da Pan
66d008caf1e57767fdf55b8b
Yiding Sun
admin_assigned
2024-08-29T07:26:40.448Z
64896a17238422ccf37975ca
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nlfoIYrpufqeLeaJLAG71.jpeg
Yiding Sun
user
Emanual20
66d008caf1e57767fdf55b8c
Shusen Zhang
claimed_verified
2024-08-29T07:22:31.178Z
64d9c9bd95cf13a381f6cc1a
/avatars/ce138298df376511044c69182b6419b9.svg
zss
user
zss9606
66d008caf1e57767fdf55b8d
Zheng Liang
66d008caf1e57767fdf55b8e
Xin Wu
66d008caf1e57767fdf55b8f
Yanjun Shen
admin_assigned
2024-08-29T07:28:16.193Z
64ced7c01720c7a483d0862f
/avatars/1fae685b7226a0a1c95b2d4993f2ab60.svg
shenyanjun
user
zilchshen
66d008caf1e57767fdf55b90
Fan Yang
66d008caf1e57767fdf55b91
Haoze Sun
claimed_verified
2025-01-23T15:05:41.512Z
6436bb0dd58a5ea528c55acb
/avatars/df17b66780e14e07bbe4625f068a94ad.svg
Alvin Sun
user
AlvinSunYooo
66d008caf1e57767fdf55b92
Tianpeng Li
admin_assigned
2024-08-29T07:29:12.964Z
64be4c97ef8c0e42bf448469
/avatars/a2a45b5f17d3e27c59547312bcf7bb52.svg
Tianpeng Li
user
TJU-Tianpengli
66d008caf1e57767fdf55b93
Mingan Lin
claimed_verified
2024-10-14T07:49:00.811Z
6415947858a690df103af49f
/avatars/38aec23b869833bceb25b9250809b419.svg
lma
user
lin5547
66d008caf1e57767fdf55b94
Jianhua Xu
admin_assigned
2024-08-29T07:29:27.331Z
66042414ded78d7454b9746e
/avatars/828feab7015b527b5ac5c9238fa3c7e0.svg
jianhua Xu
user
ArthurXu2020
66d008caf1e57767fdf55b95
Yufan Zhang
66d008caf1e57767fdf55b96
Xiaonan Nie
admin_assigned
2024-08-29T07:30:02.122Z
64b6b81142134e053233c3c0
/avatars/5c7455d99a7a2648f77a531c9a71eb98.svg
Xiaonan Nie
user
codecaution
66d008caf1e57767fdf55b97
Lei Su
66d008caf1e57767fdf55b98
Bingning Wang
claimed_verified
2024-12-10T15:15:28.995Z
640dd3e0c364a086c6322ad2
https://cdn-avatars.huggingface.co/v1/production/uploads/640dd3e0c364a086c6322ad2/z11CvZcbWGSRtWXzvem6I.png
Daniel Wang
user
DanielWang
66d008caf1e57767fdf55b99
Wentao Zhang
66d008caf1e57767fdf55b9a
Jiaxin Mao
66d008caf1e57767fdf55b9b
Zenan Zhou
admin_assigned
2024-08-29T07:31:03.172Z
668d4e50ed63008dfaa78304
/avatars/80854a3c6b4b7c70cd46694d4cf7296a.svg
Zenan Zhou
user
Zenan11
66d008caf1e57767fdf55b9c
Weipeng Chen
admin_assigned
2024-08-29T07:29:02.819Z
6501587887b370a56ad2608e
/avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
Wei-Peng Chen
user
whenfra
BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
  Deduplication by Introducing a Competitive Large Language Model Baseline
The general capabilities of Large Language Models (LLM) highly rely on the
composition and selection on extensive pretraining datasets, treated as
commercial secrets by several institutions. To mitigate this issue, we
open-source the details of a universally applicable data processing pipeline
and validate its effectiveness and potential by introducing a competitive LLM
baseline. Specifically, the data processing pipeline consists of broad
collection to scale up and reweighting to improve quality. We then pretrain a
7B model BaichuanSEED with 3T tokens processed by our pipeline without any
deliberate downstream task-related optimization, followed by an easy but
effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
and predictability throughout training and achieves comparable performance on
comprehensive benchmarks with several commercial advanced large language
models, such as Qwen1.5 and Llama3. We also conduct several heuristic
experiments to discuss the potential for further optimization of downstream
tasks, such as mathematics and coding.
A data processing pipeline and open-sourced details for training a large language model achieve competitive performance with commercial models.
Large Language Models LLM data processing pipeline pretraining datasets open-source broad collection reweighting pretrained model BaichuanSEED supervised fine-tuning Qwen1.5 Llama3 comprehensive benchmarks mathematics coding