title: LongAlign: A Recipe for Long Context Alignment of Large Language Models
summary: Extending large language models to effectively handle long contexts requires
  instruction fine-tuning on input sequences of similar length. To address this,
  we present LongAlign -- a recipe of the instruction data, training, and
  evaluation for long context alignment. First, we construct a long
  instruction-following dataset using Self-Instruct. To ensure the data
  diversity, it covers a broad range of tasks from various long context sources.
  Second, we adopt the packing and sorted batching strategies to speed up
  supervised fine-tuning on data with varied length distributions. Additionally,
  we develop a loss weighting method to balance the contribution to the loss
  across different sequences during packing training. Third, we introduce the
  LongBench-Chat benchmark for evaluating instruction-following capabilities on
  queries of 10k-100k in length. Experiments show that LongAlign outperforms
  existing recipes for LLMs in long context tasks by up to 30\%, while also
  maintaining their proficiency in handling short, generic tasks. The code, data,
  and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.
paper_authors._id: 65bb14cabd9e29f771f5d0cd
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:44:19.478Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 65bb14cabd9e29f771f5d0ce
paper_authors.name: Xin Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:42:35.924Z
paper_authors._id: 648c64829c935db2b527a764
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
paper_authors.fullname: Xin Lv
paper_authors.type: user
paper_authors.user: davidlvxin
paper_authors._id: 65bb14cabd9e29f771f5d0cf
paper_authors.name: Jiajie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:26:17.362Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 65bb14cabd9e29f771f5d0d0
paper_authors.name: Yuze He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:43:10.123Z
paper_authors._id: 64c903957b4d0d947ce86bc6
paper_authors.avatarUrl: /avatars/61d70a3ba00c83a5950f5c909a1a06f8.svg
paper_authors.fullname: Yuze He
paper_authors.type: user
paper_authors.user: hyz317
paper_authors._id: 65bb14cabd9e29f771f5d0d1
paper_authors.name: Ji Qi
paper_authors._id: 65bb14cabd9e29f771f5d0d2
paper_authors.name: Lei Hou
paper_authors._id: 65bb14cabd9e29f771f5d0d3
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:44:23.312Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_authors._id: 65bb14cabd9e29f771f5d0d4
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:44:29.296Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 65bb14cabd9e29f771f5d0d5
paper_authors.name: Juanzi Li
paper_title: LongAlign: A Recipe for Long Context Alignment of Large Language Models
paper_summary: Extending large language models to effectively handle long contexts requires
  instruction fine-tuning on input sequences of similar length. To address this,
  we present LongAlign -- a recipe of the instruction data, training, and
  evaluation for long context alignment. First, we construct a long
  instruction-following dataset using Self-Instruct. To ensure the data
  diversity, it covers a broad range of tasks from various long context sources.
  Second, we adopt the packing and sorted batching strategies to speed up
  supervised fine-tuning on data with varied length distributions. Additionally,
  we develop a loss weighting method to balance the contribution to the loss
  across different sequences during packing training. Third, we introduce the
  LongBench-Chat benchmark for evaluating instruction-following capabilities on
  queries of 10k-100k in length. Experiments show that LongAlign outperforms
  existing recipes for LLMs in long context tasks by up to 30\%, while also
  maintaining their proficiency in handling short, generic tasks. The code, data,
  and long-aligned models are open-sourced at https://github.com/THUDM/LongAlign.
paper_ai_keywords: LongAlign instruction-following dataset Self-Instruct packing sorted batching strategies supervised fine-tuning loss weighting method LongBench-Chat benchmark instruction-following capabilities long context alignment