title: Model Dementia: Generated Data Makes Models Forget
summary: Stable Diffusion revolutionised image creation from descriptive text. GPT-2,
  GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of
  language tasks. ChatGPT introduced such language models to the general public.
  It is now clear that large language models (LLMs) are here to stay, and will
  bring about drastic change in the whole ecosystem of online text and images. In
  this paper we consider what the future might hold. What will happen to GPT-{n}
  once LLMs contribute much of the language found online? We find that use of
  model-generated content in training causes irreversible defects in the
  resulting models, where tails of the original content distribution disappear.
  We call this effect model dementia and show that it can occur in Variational
  Autoencoders (VAEs), Gaussian Mixture Models (GMMs) and LLMs. We build
  theoretical intuition behind the phenomenon and portray its ubiquity amongst
  all learned generative models. We demonstrate that it has to be taken seriously
  if we are to sustain the benefits of training from large-scale data scraped
  from the web. Indeed, the value of data collected about genuine human
  interactions with systems will be increasingly valuable in the presence of
  content generated by LLMs in data crawled from the Internet.
paper_authors._id: 64756056b68461d5cf7eb717
paper_authors.name: Ilia Shumailov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T10:07:58.040Z
paper_authors._id: 6475c2794766357252e69e9f
paper_authors.avatarUrl: /avatars/db428715dfd2239df2aeaaff1282323f.svg
paper_authors.fullname: i
paper_authors.type: user
paper_authors.user: iliashum
paper_authors._id: 64756056b68461d5cf7eb718
paper_authors.name: Zakhar Shumaylov
paper_authors._id: 64756056b68461d5cf7eb719
paper_authors.name: Yiren Zhao
paper_authors._id: 64756056b68461d5cf7eb71a
paper_authors.name: Yarin Gal
paper_authors._id: 64756056b68461d5cf7eb71b
paper_authors.name: Nicolas Papernot
paper_authors._id: 64756056b68461d5cf7eb71c
paper_authors.name: Ross Anderson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T10:08:01.131Z
paper_authors._id: 6475c2c9975bc84973211a60
paper_authors.avatarUrl: /avatars/efd314f9404c01dd419c5aa563fca8d3.svg
paper_authors.fullname: Ross Anderson
paper_authors.type: user
paper_authors.user: RossJAnderson
paper_title: Model Dementia: Generated Data Makes Models Forget
paper_summary: Stable Diffusion revolutionised image creation from descriptive text. GPT-2,
  GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of
  language tasks. ChatGPT introduced such language models to the general public.
  It is now clear that large language models (LLMs) are here to stay, and will
  bring about drastic change in the whole ecosystem of online text and images. In
  this paper we consider what the future might hold. What will happen to GPT-{n}
  once LLMs contribute much of the language found online? We find that use of
  model-generated content in training causes irreversible defects in the
  resulting models, where tails of the original content distribution disappear.
  We call this effect model dementia and show that it can occur in Variational
  Autoencoders (VAEs), Gaussian Mixture Models (GMMs) and LLMs. We build
  theoretical intuition behind the phenomenon and portray its ubiquity amongst
  all learned generative models. We demonstrate that it has to be taken seriously
  if we are to sustain the benefits of training from large-scale data scraped
  from the web. Indeed, the value of data collected about genuine human
  interactions with systems will be increasingly valuable in the presence of
  content generated by LLMs in data crawled from the Internet.
paper_ai_keywords: Stable Diffusion large language models (LLMs) GPT-{n} model-generated content model dementia Variational Autoencoders (VAEs) Gaussian Mixture Models (GMMs)