title: In-Context Pretraining: Language Modeling Beyond Document Boundaries
summary: Large language models (LMs) are currently trained to predict tokens given
  document prefixes, enabling them to directly perform long-form generation and
  prompting-style tasks which can be reduced to document completion. Existing
  pretraining pipelines train LMs by concatenating random sets of short documents
  to create input contexts but the prior documents provide no signal for
  predicting the next document. We instead present In-Context Pretraining, a new
  approach where language models are pretrained on a sequence of related
  documents, thereby explicitly encouraging them to read and reason across
  document boundaries. We can do In-Context Pretraining by simply changing the
  document ordering so that each context contains related documents, and directly
  applying existing pretraining pipelines. However, this document sorting problem
  is challenging. There are billions of documents and we would like the sort to
  maximize contextual similarity for every document without repeating any data.
  To do this, we introduce approximate algorithms for finding related documents
  with efficient nearest neighbor search and constructing coherent input contexts
  with a graph traversal algorithm. Our experiments show In-Context Pretraining
  offers a simple and scalable approach to significantly enhance LMs'performance:
  we see notable improvements in tasks that require more complex contextual
  reasoning, including in-context learning (+8%), reading comprehension (+15%),
  faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
  retrieval augmentation (+9%).
paper_authors._id: 652e0cceb86e108d0ff3c2aa
paper_authors.name: Weijia Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:32:01.655Z
paper_authors._id: 6400f2ed568dbe30c9161e47
paper_authors.avatarUrl: /avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
paper_authors.fullname: Weijia Shi
paper_authors.type: user
paper_authors.user: swj0419
paper_authors._id: 652e0cceb86e108d0ff3c2ab
paper_authors.name: Sewon Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:32:08.918Z
paper_authors._id: 63a76d0de27a6dbd485fe863
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
paper_authors.fullname: Sewon Min
paper_authors.type: user
paper_authors.user: sewon
paper_authors._id: 652e0cceb86e108d0ff3c2ac
paper_authors.name: Maria Lomeli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:32:25.253Z
paper_authors._id: 64f86587e7584abc62545b3e
paper_authors.avatarUrl: /avatars/57296279592ad5f77c39d174f6289420.svg
paper_authors.fullname: Maria Lomeli
paper_authors.type: user
paper_authors.user: marialomeli
paper_authors._id: 652e0cceb86e108d0ff3c2ad
paper_authors.name: Chunting Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:32:35.939Z
paper_authors._id: 64f8d07dd493d8b0d2c04ecd
paper_authors.avatarUrl: /avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Chunting
paper_authors._id: 652e0cceb86e108d0ff3c2ae
paper_authors.name: Margaret Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:32:54.740Z
paper_authors._id: 64bf0bcfc05a0df0d2d09d5c
paper_authors.avatarUrl: /avatars/d24bfb4d6b3d9815e68cca710639ab87.svg
paper_authors.fullname: Margaret Li
paper_authors.type: user
paper_authors.user: margsli
paper_authors._id: 652e0cceb86e108d0ff3c2af
paper_authors.name: Victoria Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:33:10.059Z
paper_authors._id: 61df349e34b589fbaf2e93c4
paper_authors.avatarUrl: /avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
paper_authors.fullname: Victoria Lin
paper_authors.type: user
paper_authors.user: VictoriaLinML
paper_authors._id: 652e0cceb86e108d0ff3c2b0
paper_authors.name: Noah A. Smith
paper_authors._id: 652e0cceb86e108d0ff3c2b1
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 652e0cceb86e108d0ff3c2b2
paper_authors.name: Scott Yih
paper_authors._id: 652e0cceb86e108d0ff3c2b3
paper_authors.name: Mike Lewis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:34:54.174Z
paper_authors._id: 5e67eda71009063689407484
paper_authors.avatarUrl: /avatars/6d7688faba888d463584eaa02ad011e3.svg
paper_authors.fullname: Mike Lewis
paper_authors.type: user
paper_authors.user: mikelewis0
paper_title: In-Context Pretraining: Language Modeling Beyond Document Boundaries
paper_summary: Large language models (LMs) are currently trained to predict tokens given
  document prefixes, enabling them to directly perform long-form generation and
  prompting-style tasks which can be reduced to document completion. Existing
  pretraining pipelines train LMs by concatenating random sets of short documents
  to create input contexts but the prior documents provide no signal for
  predicting the next document. We instead present In-Context Pretraining, a new
  approach where language models are pretrained on a sequence of related
  documents, thereby explicitly encouraging them to read and reason across
  document boundaries. We can do In-Context Pretraining by simply changing the
  document ordering so that each context contains related documents, and directly
  applying existing pretraining pipelines. However, this document sorting problem
  is challenging. There are billions of documents and we would like the sort to
  maximize contextual similarity for every document without repeating any data.
  To do this, we introduce approximate algorithms for finding related documents
  with efficient nearest neighbor search and constructing coherent input contexts
  with a graph traversal algorithm. Our experiments show In-Context Pretraining
  offers a simple and scalable approach to significantly enhance LMs'performance:
  we see notable improvements in tasks that require more complex contextual
  reasoning, including in-context learning (+8%), reading comprehension (+15%),
  faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
  retrieval augmentation (+9%).
paper_ai_summary: In-Context Pretraining improves large language models by ordering documents to maximize contextual similarity, leading to better performance in complex reasoning tasks.
paper_ai_keywords: large language models (LMs) token prediction long-form generation prompting-style tasks document prefixes document completion pretraining pipelines In-Context Pretraining related documents contextual similarity nearest neighbor search graph traversal algorithm in-context learning reading comprehension faithfulness long-context reasoning retrieval augmentation