How to Synthesize Text Data without Model Collapse?
Model collapse in synthetic data indicates that iterative training on
self-generated data leads to a gradual decline in performance. With the
proliferation of AI models, synthetic data will fundamentally reshape the web
data ecosystem. Future GPT-{n} models will inevitably be trained on a blend
of synthetic and human-produced data. In this paper, we focus on two questions:
what is the impact of synthetic data on language model training, and how to
synthesize data without model collapse? We first pre-train language models
across different proportions of synthetic data, revealing a negative
correlation between the proportion of synthetic data and model performance. We
further conduct statistical analysis on synthetic data to uncover
distributional shift phenomenon and over-concentration of n-gram features.
Inspired by the above findings, we propose token editing on human-produced data
to obtain semi-synthetic data. As a proof of concept, we theoretically
demonstrate that token-level editing can prevent model collapse, as the test
error is constrained by a finite upper bound. We conduct extensive experiments
on pre-training from scratch, continual pre-training, and supervised
fine-tuning. The results validate our theoretical proof that token-level
editing improves data quality and enhances model performance.
6764e1dfc51db09f8c3cd75b
Xuekai Zhu
admin_assigned
2024-12-20T09:20:24.860Z
647ffddeb82adfa7cc1a10d9
/avatars/26aa168d6b2068298ebb16584aa52b6c.svg
zhu
user
xuekai
6764e1dfc51db09f8c3cd75c
Daixuan Cheng
admin_assigned
2024-12-20T09:20:32.479Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
6764e1dfc51db09f8c3cd75d
Hengli Li
admin_assigned
2024-12-20T09:20:41.812Z
63256836ff539edeea8a8660
https://cdn-avatars.huggingface.co/v1/production/uploads/1663395861151-noauth.png
Li Hengli
user
Hengli
6764e1dfc51db09f8c3cd75e
Kaiyan Zhang
admin_assigned
2024-12-20T09:20:57.241Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
6764e1dfc51db09f8c3cd75f
Ermo Hua
claimed_verified
2024-12-30T19:36:29.222Z
6445fa2ffc22e309d78bef3e
https://cdn-avatars.huggingface.co/v1/production/uploads/6445fa2ffc22e309d78bef3e/FQaINLd0PjgY9EnK_APRk.jpeg
Messi Hua
user
Messi-Hua
6764e1dfc51db09f8c3cd760
Xingtai Lv
admin_assigned
2024-12-20T09:21:08.043Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
6764e1dfc51db09f8c3cd761
Ning Ding
admin_assigned
2024-12-20T09:21:13.963Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
6764e1dfc51db09f8c3cd762
Zhouhan Lin
6764e1dfc51db09f8c3cd763
Zilong Zheng
extracted_pending
2024-12-20T03:17:52.795Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
6764e1dfc51db09f8c3cd764
Bowen Zhou
admin_assigned
2024-12-20T09:21:41.799Z
669f614b59adf5b56e05bce3
/avatars/ffd4189efbceb0e63a03db273065a44b.svg
BowenZhou
user
bowenZhou
How to Synthesize Text Data without Model Collapse?
Model collapse in synthetic data indicates that iterative training on
self-generated data leads to a gradual decline in performance. With the
proliferation of AI models, synthetic data will fundamentally reshape the web
data ecosystem. Future GPT-{n} models will inevitably be trained on a blend
of synthetic and human-produced data. In this paper, we focus on two questions:
what is the impact of synthetic data on language model training, and how to
synthesize data without model collapse? We first pre-train language models
across different proportions of synthetic data, revealing a negative
correlation between the proportion of synthetic data and model performance. We
further conduct statistical analysis on synthetic data to uncover
distributional shift phenomenon and over-concentration of n-gram features.
Inspired by the above findings, we propose token editing on human-produced data
to obtain semi-synthetic data. As a proof of concept, we theoretically
demonstrate that token-level editing can prevent model collapse, as the test
error is constrained by a finite upper bound. We conduct extensive experiments
on pre-training from scratch, continual pre-training, and supervised
fine-tuning. The results validate our theoretical proof that token-level
editing improves data quality and enhances model performance.
The use of synthetic data in language model training leads to model collapse, which is mitigated by token-level editing of human-produced data to create semi-synthetic data.
synthetic data model collapse language model training distributional shift n-gram features token editing semi-synthetic data pre-training continual pre-training supervised fine-tuning