title: Emergent properties with repeated examples
summary: We study the performance of transformers as a function of the number of
  repetitions of training examples with algorithmically generated datasets. On
  three problems of mathematics: the greatest common divisor, modular
  multiplication, and matrix eigenvalues, we show that for a fixed number of
  training steps, models trained on smaller sets of repeated examples outperform
  models trained on larger sets of single-use examples. We also demonstrate that
  two-set training - repeated use of a small random subset of examples, along
  normal sampling on the rest of the training set - provides for faster learning
  and better performance. This highlights that the benefits of repetition can
  outweigh those of data diversity. These datasets and problems provide a
  controlled setting to shed light on the still poorly understood interplay
  between generalization and memorization in deep learning.
paper_authors._id: 67088ce7886c2765f184aebe
paper_authors.name: François Charton
paper_authors._id: 67088ce7886c2765f184aebf
paper_authors.name: Julia Kempe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:08:47.955Z
paper_authors._id: 65ce30e06da01df536eded5a
paper_authors.avatarUrl: /avatars/04c32cba7a3bbaf9ea5dee88c96cf87b.svg
paper_authors.fullname: Julia Kempe
paper_authors.type: user
paper_authors.user: Knykny
paper_title: Emergent properties with repeated examples
paper_summary: We study the performance of transformers as a function of the number of
  repetitions of training examples with algorithmically generated datasets. On
  three problems of mathematics: the greatest common divisor, modular
  multiplication, and matrix eigenvalues, we show that for a fixed number of
  training steps, models trained on smaller sets of repeated examples outperform
  models trained on larger sets of single-use examples. We also demonstrate that
  two-set training - repeated use of a small random subset of examples, along
  normal sampling on the rest of the training set - provides for faster learning
  and better performance. This highlights that the benefits of repetition can
  outweigh those of data diversity. These datasets and problems provide a
  controlled setting to shed light on the still poorly understood interplay
  between generalization and memorization in deep learning.
paper_ai_keywords: transformers algorithmically generated datasets greatest common divisor modular multiplication matrix eigenvalues training steps two-set training cross-attention layers generalization and memorization