title: How to Synthesize Text Data without Model Collapse?
summary: Model collapse in synthetic data indicates that iterative training on
  self-generated data leads to a gradual decline in performance. With the
  proliferation of AI models, synthetic data will fundamentally reshape the web
  data ecosystem. Future GPT-{n} models will inevitably be trained on a blend
  of synthetic and human-produced data. In this paper, we focus on two questions:
  what is the impact of synthetic data on language model training, and how to
  synthesize data without model collapse? We first pre-train language models
  across different proportions of synthetic data, revealing a negative
  correlation between the proportion of synthetic data and model performance. We
  further conduct statistical analysis on synthetic data to uncover
  distributional shift phenomenon and over-concentration of n-gram features.
  Inspired by the above findings, we propose token editing on human-produced data
  to obtain semi-synthetic data. As a proof of concept, we theoretically
  demonstrate that token-level editing can prevent model collapse, as the test
  error is constrained by a finite upper bound. We conduct extensive experiments
  on pre-training from scratch, continual pre-training, and supervised
  fine-tuning. The results validate our theoretical proof that token-level
  editing improves data quality and enhances model performance.
paper_authors._id: 6764e1dfc51db09f8c3cd75b
paper_authors.name: Xuekai Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:20:24.860Z
paper_authors._id: 647ffddeb82adfa7cc1a10d9
paper_authors.avatarUrl: /avatars/26aa168d6b2068298ebb16584aa52b6c.svg
paper_authors.fullname: zhu
paper_authors.type: user
paper_authors.user: xuekai
paper_authors._id: 6764e1dfc51db09f8c3cd75c
paper_authors.name: Daixuan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:20:32.479Z
paper_authors._id: 649e6761f9134a06ed1e0cea
paper_authors.avatarUrl: /avatars/00b5dcb744c54a4aa18fe08efd70d6ff.svg
paper_authors.fullname: Daixuan Cheng
paper_authors.type: user
paper_authors.user: daixuancheng
paper_authors._id: 6764e1dfc51db09f8c3cd75d
paper_authors.name: Hengli Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:20:41.812Z
paper_authors._id: 63256836ff539edeea8a8660
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663395861151-noauth.png
paper_authors.fullname: Li Hengli
paper_authors.type: user
paper_authors.user: Hengli
paper_authors._id: 6764e1dfc51db09f8c3cd75e
paper_authors.name: Kaiyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:20:57.241Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 6764e1dfc51db09f8c3cd75f
paper_authors.name: Ermo Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:29.222Z
paper_authors._id: 6445fa2ffc22e309d78bef3e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445fa2ffc22e309d78bef3e/FQaINLd0PjgY9EnK_APRk.jpeg
paper_authors.fullname: Messi Hua
paper_authors.type: user
paper_authors.user: Messi-Hua
paper_authors._id: 6764e1dfc51db09f8c3cd760
paper_authors.name: Xingtai Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:21:08.043Z
paper_authors._id: 663f07d029be04778ba97871
paper_authors.avatarUrl: /avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
paper_authors.fullname: Xingtai Lv
paper_authors.type: user
paper_authors.user: XingtaiHF
paper_authors._id: 6764e1dfc51db09f8c3cd761
paper_authors.name: Ning Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:21:13.963Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 6764e1dfc51db09f8c3cd762
paper_authors.name: Zhouhan Lin
paper_authors._id: 6764e1dfc51db09f8c3cd763
paper_authors.name: Zilong Zheng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-20T03:17:52.795Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_authors._id: 6764e1dfc51db09f8c3cd764
paper_authors.name: Bowen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:21:41.799Z
paper_authors._id: 669f614b59adf5b56e05bce3
paper_authors.avatarUrl: /avatars/ffd4189efbceb0e63a03db273065a44b.svg
paper_authors.fullname: BowenZhou
paper_authors.type: user
paper_authors.user: bowenZhou
paper_title: How to Synthesize Text Data without Model Collapse?
paper_summary: Model collapse in synthetic data indicates that iterative training on
  self-generated data leads to a gradual decline in performance. With the
  proliferation of AI models, synthetic data will fundamentally reshape the web
  data ecosystem. Future GPT-{n} models will inevitably be trained on a blend
  of synthetic and human-produced data. In this paper, we focus on two questions:
  what is the impact of synthetic data on language model training, and how to
  synthesize data without model collapse? We first pre-train language models
  across different proportions of synthetic data, revealing a negative
  correlation between the proportion of synthetic data and model performance. We
  further conduct statistical analysis on synthetic data to uncover
  distributional shift phenomenon and over-concentration of n-gram features.
  Inspired by the above findings, we propose token editing on human-produced data
  to obtain semi-synthetic data. As a proof of concept, we theoretically
  demonstrate that token-level editing can prevent model collapse, as the test
  error is constrained by a finite upper bound. We conduct extensive experiments
  on pre-training from scratch, continual pre-training, and supervised
  fine-tuning. The results validate our theoretical proof that token-level
  editing improves data quality and enhances model performance.
paper_ai_keywords: model collapse iterative training synthetic data language model distributional shift n-gram features token editing semi-synthetic data token-level editing test error pre-training from scratch continual pre-training model collapse iterative training synthetic data language model distributional shift n-gram features token editing semi-synthetic data token-level editing test error pre-training from scratch continual pre-training