title: A Tale of Tails: Model Collapse as a Change of Scaling Laws
summary: As AI model size grows, neural scaling laws have become a crucial tool to
  predict the improvements of large models when increasing capacity and the size
  of original (human or natural) training data. Yet, the widespread use of
  popular models means that the ecosystem of online data and text will co-evolve
  to progressively contain increased amounts of synthesized data. In this paper
  we ask: How will the scaling laws change in the inevitable regime where
  synthetic data makes its way into the training corpus? Will future models,
  still improve, or be doomed to degenerate up to total (model) collapse? We
  develop a theoretical framework of model collapse through the lens of scaling
  laws. We discover a wide range of decay phenomena, analyzing loss of scaling,
  shifted scaling with number of generations, the ''un-learning" of skills, and
  grokking when mixing human and synthesized data. Our theory is validated by
  large-scale experiments with a transformer on an arithmetic task and text
  generation using the large language model Llama2.
paper_authors._id: 65cad9a451c1738a55841e3c
paper_authors.name: Elvis Dohmatob
paper_authors._id: 65cad9a451c1738a55841e3d
paper_authors.name: Yunzhen Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:42:56.044Z
paper_authors._id: 65cbfa6c968742be942e6cba
paper_authors.avatarUrl: /avatars/1a6cc0983edc28fa92178d3abc283ba1.svg
paper_authors.fullname: Feng
paper_authors.type: user
paper_authors.user: Yunzhen
paper_authors._id: 65cad9a451c1738a55841e3e
paper_authors.name: Pu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:16:59.457Z
paper_authors._id: 64632ebd7e9025b09bd55be2
paper_authors.avatarUrl: /avatars/396935c4fb9c077cea35ca2337620e33.svg
paper_authors.fullname: Pu Yang
paper_authors.type: user
paper_authors.user: yangpuPKU
paper_authors._id: 65cad9a451c1738a55841e3f
paper_authors.name: Francois Charton
paper_authors._id: 65cad9a451c1738a55841e40
paper_authors.name: Julia Kempe
paper_title: A Tale of Tails: Model Collapse as a Change of Scaling Laws
paper_summary: As AI model size grows, neural scaling laws have become a crucial tool to
  predict the improvements of large models when increasing capacity and the size
  of original (human or natural) training data. Yet, the widespread use of
  popular models means that the ecosystem of online data and text will co-evolve
  to progressively contain increased amounts of synthesized data. In this paper
  we ask: How will the scaling laws change in the inevitable regime where
  synthetic data makes its way into the training corpus? Will future models,
  still improve, or be doomed to degenerate up to total (model) collapse? We
  develop a theoretical framework of model collapse through the lens of scaling
  laws. We discover a wide range of decay phenomena, analyzing loss of scaling,
  shifted scaling with number of generations, the ''un-learning" of skills, and
  grokking when mixing human and synthesized data. Our theory is validated by
  large-scale experiments with a transformer on an arithmetic task and text
  generation using the large language model Llama2.
paper_ai_keywords: advanced scaling laws permutation invariant synthesized data model collapse decay phenomena loss of scaling shifted scaling un-learning grokking transformer arithmetic task large language model Llama2