title: In-context learning and Occam's razor
summary: The goal of machine learning is generalization. While the No Free Lunch
  Theorem states that we cannot obtain theoretical guarantees for generalization
  without further assumptions, in practice we observe that simple models which
  explain the training data generalize best: a principle called Occam's razor.
  Despite the need for simple models, most current approaches in machine learning
  only minimize the training error, and at best indirectly promote simplicity
  through regularization or architecture design. Here, we draw a connection
  between Occam's razor and in-context learning: an emergent ability of certain
  sequence models like Transformers to learn at inference time from past
  observations in a sequence. In particular, we show that the next-token
  prediction loss used to train in-context learners is directly equivalent to a
  data compression technique called prequential coding, and that minimizing this
  loss amounts to jointly minimizing both the training error and the complexity
  of the model that was implicitly learned from context. Our theory and the
  empirical experiments we use to support it not only provide a normative account
  of in-context learning, but also elucidate the shortcomings of current
  in-context learning methods, suggesting ways in which they can be improved. We
  make our code available at https://github.com/3rdCore/PrequentialCode.
paper_authors._id: 6717ac81ee30e9f032308982
paper_authors.name: Eric Elmoznino
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:54:21.298Z
paper_authors._id: 64fd0dc89ecd05d5bf77d331
paper_authors.avatarUrl: /avatars/1087b977ebbafc89dba1cfb1084df6aa.svg
paper_authors.fullname: Eric Elmoznino
paper_authors.type: user
paper_authors.user: EricElmoznino
paper_authors._id: 6717ac81ee30e9f032308983
paper_authors.name: Tom Marty
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T13:47:06.100Z
paper_authors._id: 646402a6a5af935cfe7226a5
paper_authors.avatarUrl: /avatars/98655f52a05259a5f6478588aa15bda6.svg
paper_authors.fullname: Tom Marty
paper_authors.type: user
paper_authors.user: 3rdCore
paper_authors._id: 6717ac81ee30e9f032308984
paper_authors.name: Tejas Kasetty
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:54:26.971Z
paper_authors._id: 65b2f64d54fa188a2f2a7132
paper_authors.avatarUrl: /avatars/95f218fe0f525f56d12238330a68e593.svg
paper_authors.fullname: Tejas Kasetty
paper_authors.type: user
paper_authors.user: tejaskasetty
paper_authors._id: 6717ac81ee30e9f032308985
paper_authors.name: Leo Gagnon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:54:32.102Z
paper_authors._id: 669580c8643c258b90be7497
paper_authors.avatarUrl: /avatars/19fc666b4d56f71020fa646ca2f173de.svg
paper_authors.fullname: Léo Gagnon
paper_authors.type: user
paper_authors.user: leogagnon
paper_authors._id: 6717ac81ee30e9f032308986
paper_authors.name: Sarthak Mittal
paper_authors._id: 6717ac81ee30e9f032308987
paper_authors.name: Mahan Fathi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:54:53.891Z
paper_authors._id: 64903c2d9b1aaa0738286323
paper_authors.avatarUrl: /avatars/9c4d0734b6f779d0f138951998f1d7b5.svg
paper_authors.fullname: Mahan Fathi
paper_authors.type: user
paper_authors.user: mahanfathi
paper_authors._id: 6717ac81ee30e9f032308988
paper_authors.name: Dhanya Sridhar
paper_authors._id: 6717ac81ee30e9f032308989
paper_authors.name: Guillaume Lajoie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:55:02.206Z
paper_authors._id: 64909d1d6f217a26d91c6d9b
paper_authors.avatarUrl: /avatars/28eeff49dd5c2f01b576e4a050551e17.svg
paper_authors.fullname: guillaume lajoie
paper_authors.type: user
paper_authors.user: glajoie
paper_title: In-context learning and Occam's razor
paper_summary: The goal of machine learning is generalization. While the No Free Lunch
  Theorem states that we cannot obtain theoretical guarantees for generalization
  without further assumptions, in practice we observe that simple models which
  explain the training data generalize best: a principle called Occam's razor.
  Despite the need for simple models, most current approaches in machine learning
  only minimize the training error, and at best indirectly promote simplicity
  through regularization or architecture design. Here, we draw a connection
  between Occam's razor and in-context learning: an emergent ability of certain
  sequence models like Transformers to learn at inference time from past
  observations in a sequence. In particular, we show that the next-token
  prediction loss used to train in-context learners is directly equivalent to a
  data compression technique called prequential coding, and that minimizing this
  loss amounts to jointly minimizing both the training error and the complexity
  of the model that was implicitly learned from context. Our theory and the
  empirical experiments we use to support it not only provide a normative account
  of in-context learning, but also elucidate the shortcomings of current
  in-context learning methods, suggesting ways in which they can be improved. We
  make our code available at https://github.com/3rdCore/PrequentialCode.
paper_ai_keywords: in-context learning Transformers prequential coding