title: Data Engineering for Scaling Language Models to 128K Context
summary: We study the continual pretraining recipe for scaling language models'
  context lengths to 128K, with a focus on data engineering. We hypothesize that
  long context modeling, in particular the ability to utilize information
  at arbitrary input locations, is a capability that is mostly already acquired
  through large-scale pretraining, and that this capability can be readily
  extended to contexts substantially longer than seen during training~(e.g., 4K
  to 128K) through lightweight continual pretraining on appropriate data mixture.
  We investigate the quantity and quality of the data for
  continual pretraining: (1) for quantity, we show that 500 million to 5 billion
  tokens are enough to enable the model to retrieve information anywhere within
  the 128K context; (2) for quality, our results equally emphasize domain
  balance and length upsampling. Concretely, we find that naively
  upsampling longer data on certain domains like books, a common practice of
  existing work, gives suboptimal performance, and that a balanced domain mixture
  is important. We demonstrate that continual pretraining of the full model on
  1B-5B tokens of such data is an effective and affordable strategy for scaling
  the context length of language models to 128K. Our recipe outperforms strong
  open-source long-context models and closes the gap to frontier models like
  GPT-4 128K.
paper_authors._id: 65ced3ba2a811b35593b217b
paper_authors.name: Yao Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T14:29:21.605Z
paper_authors._id: 636c929d5ef5a1a07db4e785
paper_authors.avatarUrl: /avatars/1bb48dbba964f7fa84036586032684cf.svg
paper_authors.fullname: Yao Fu
paper_authors.type: user
paper_authors.user: yaofu
paper_authors._id: 65ced3ba2a811b35593b217c
paper_authors.name: Rameswar Panda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T14:29:28.707Z
paper_authors._id: 648a7542ac1fbb8a1a654645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
paper_authors.fullname: Rameswar Panda
paper_authors.type: user
paper_authors.user: rpand002
paper_authors._id: 65ced3ba2a811b35593b217d
paper_authors.name: Xinyao Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T14:29:34.558Z
paper_authors._id: 627f4c100d2c1c0ba3da8a72
paper_authors.avatarUrl: /avatars/404869a87480ef526c1d5fccb583bfa8.svg
paper_authors.fullname: Xinyao Niu
paper_authors.type: user
paper_authors.user: sirius-ctrl
paper_authors._id: 65ced3ba2a811b35593b217e
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T14:29:59.838Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 65ced3ba2a811b35593b217f
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 65ced3ba2a811b35593b2180
paper_authors.name: Yoon Kim
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-16T03:17:14.629Z
paper_authors._id: 630ee22c05b5c5280e3cdcdf
paper_authors.avatarUrl: /avatars/b282162d6d9ae8512873eba0275f603e.svg
paper_authors.fullname: Yoon Kim
paper_authors.type: user
paper_authors.user: yoon-kim
paper_authors._id: 65ced3ba2a811b35593b2181
paper_authors.name: Hao Peng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-04T15:24:18.817Z
paper_authors._id: 660ec5a2509153ca49775a7c
paper_authors.avatarUrl: /avatars/97570fc245cc8ec7628da9c13bd35b71.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: haopeng01
paper_title: Data Engineering for Scaling Language Models to 128K Context
paper_summary: We study the continual pretraining recipe for scaling language models'
  context lengths to 128K, with a focus on data engineering. We hypothesize that
  long context modeling, in particular the ability to utilize information
  at arbitrary input locations, is a capability that is mostly already acquired
  through large-scale pretraining, and that this capability can be readily
  extended to contexts substantially longer than seen during training~(e.g., 4K
  to 128K) through lightweight continual pretraining on appropriate data mixture.
  We investigate the quantity and quality of the data for
  continual pretraining: (1) for quantity, we show that 500 million to 5 billion
  tokens are enough to enable the model to retrieve information anywhere within
  the 128K context; (2) for quality, our results equally emphasize domain
  balance and length upsampling. Concretely, we find that naively
  upsampling longer data on certain domains like books, a common practice of
  existing work, gives suboptimal performance, and that a balanced domain mixture
  is important. We demonstrate that continual pretraining of the full model on
  1B-5B tokens of such data is an effective and affordable strategy for scaling
  the context length of language models to 128K. Our recipe outperforms strong
  open-source long-context models and closes the gap to frontier models like
  GPT-4 128K.
paper_ai_summary: Continual pretraining with domain-balanced and length-upsampled data effectively scales language models' context length to 128K without significant additional computational cost.
paper_ai_keywords: continual pretraining context lengths long context modeling data engineering data mixture domain balance length upsampling language models