title: LongRecipe: Recipe for Efficient Long Context Generalization in Large
    Languge Models
summary: Large language models (LLMs) face significant challenges in handling
  long-context tasks because of their limited effective context window size
  during pretraining, which restricts their ability to generalize over extended
  sequences. Meanwhile, extending the context window in LLMs through
  post-pretraining is highly resource-intensive. To address this, we introduce
  **LongRecipe**, an efficient training strategy for extending the context window
  of LLMs, including impactful token analysis, position index transformation, and
  training optimization strategies. It simulates long-sequence inputs while
  maintaining training efficiency and significantly improves the model's
  understanding of long-range dependencies. Experiments on three types of LLMs
  show that LongRecipe can utilize long sequences while requiring only 30% of the
  target context window size, and reduces computational training resource over
  85% compared to full sequence training. Furthermore, LongRecipe also preserves
  the original LLM's capabilities in general tasks. Ultimately, *we can extend
  the effective context window of open-source LLMs from 8k to 128k, achieving
  performance close to GPT-4 with just one day of dedicated training using a
  single GPU with 80G memory.* Our code is released at the
  [link](https://github.com/zhiyuanhubj/LongRecipe).
paper_authors._id: 66d7dd20b5c8cfdee7d38006
paper_authors.name: Zhiyuan Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T08:33:49.067Z
paper_authors._id: 64351475901c5734bcb64248
paper_authors.avatarUrl: /avatars/12346d4301c1bfb00ce0ea128a93cc15.svg
paper_authors.fullname: Zhiyuan Hu
paper_authors.type: user
paper_authors.user: zhiyuanhucs
paper_authors._id: 66d7dd20b5c8cfdee7d38007
paper_authors.name: Yuliang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:15:02.277Z
paper_authors._id: 6529f79e802e3d1a4f8ec662
paper_authors.avatarUrl: /avatars/d05320c370a6497d8792ef5acb563dd5.svg
paper_authors.fullname: Yuliang Liu
paper_authors.type: user
paper_authors.user: yuliang03181
paper_authors._id: 66d7dd20b5c8cfdee7d38008
paper_authors.name: Jinman Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T16:21:03.953Z
paper_authors._id: 6546ecc9cd0a56213929fd18
paper_authors.avatarUrl: /avatars/9f4d4e369f0ca6e5669b038badc4e622.svg
paper_authors.fullname: jinman
paper_authors.type: user
paper_authors.user: zhaojinm
paper_authors._id: 66d7dd20b5c8cfdee7d38009
paper_authors.name: Suyuchen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:15:03.844Z
paper_authors._id: 62bb1e0f3ff437e49a3088e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/bcUQmH8tKfI6DIWH9IcYp.jpeg
paper_authors.fullname: Suyuchen Wang
paper_authors.type: user
paper_authors.user: sheryc
paper_authors._id: 66d7dd20b5c8cfdee7d3800a
paper_authors.name: Yan Wang
paper_authors._id: 66d7dd20b5c8cfdee7d3800b
paper_authors.name: Wei Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-12T13:30:29.260Z
paper_authors._id: 6468823272d9180d4ac90bdf
paper_authors.avatarUrl: /avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
paper_authors.fullname: Wei Shen
paper_authors.type: user
paper_authors.user: Swtheking
paper_authors._id: 66d7dd20b5c8cfdee7d3800c
paper_authors.name: Qing Gu
paper_authors._id: 66d7dd20b5c8cfdee7d3800d
paper_authors.name: Anh Tuan Luu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:16:02.258Z
paper_authors._id: 655722e80438e0854fae7554
paper_authors.avatarUrl: /avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
paper_authors.fullname: Luu Anh Tuan
paper_authors.type: user
paper_authors.user: anhtuanluu36
paper_authors._id: 66d7dd20b5c8cfdee7d3800e
paper_authors.name: See-Kiong Ng
paper_authors._id: 66d7dd20b5c8cfdee7d3800f
paper_authors.name: Zhiwei Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:15:49.639Z
paper_authors._id: 66446066b5a21591719f3f8e
paper_authors.avatarUrl: /avatars/923e7786da7bdaa69c7e3d1567a6a156.svg
paper_authors.fullname: Jiang Zhiwei
paper_authors.type: user
paper_authors.user: Laziobird
paper_authors._id: 66d7dd20b5c8cfdee7d38010
paper_authors.name: Bryan Hooi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:14:39.419Z
paper_authors._id: 651d8032c50012d33e914f2f
paper_authors.avatarUrl: /avatars/0a44c9f51fc50ce86582e328c361ea00.svg
paper_authors.fullname: Bryan Hooi
paper_authors.type: user
paper_authors.user: bhooi
paper_title: LongRecipe: Recipe for Efficient Long Context Generalization in Large
    Languge Models
paper_summary: Large language models (LLMs) face significant challenges in handling
  long-context tasks because of their limited effective context window size
  during pretraining, which restricts their ability to generalize over extended
  sequences. Meanwhile, extending the context window in LLMs through
  post-pretraining is highly resource-intensive. To address this, we introduce
  **LongRecipe**, an efficient training strategy for extending the context window
  of LLMs, including impactful token analysis, position index transformation, and
  training optimization strategies. It simulates long-sequence inputs while
  maintaining training efficiency and significantly improves the model's
  understanding of long-range dependencies. Experiments on three types of LLMs
  show that LongRecipe can utilize long sequences while requiring only 30% of the
  target context window size, and reduces computational training resource over
  85% compared to full sequence training. Furthermore, LongRecipe also preserves
  the original LLM's capabilities in general tasks. Ultimately, *we can extend
  the effective context window of open-source LLMs from 8k to 128k, achieving
  performance close to GPT-4 with just one day of dedicated training using a
  single GPU with 80G memory.* Our code is released at the
  [link](https://github.com/zhiyuanhubj/LongRecipe).
paper_ai_keywords: LongRecipe Long-context tasks Effective context window size Denoising autoencoders Position index transformation Training optimization strategies Long-sequence inputs Long-range dependencies Open-source LLMs GPT-4