title: Extending Llama-3's Context Ten-Fold Overnight
summary: We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA
  fine-tuning. The entire training cycle is super efficient, which takes 8 hours
  on one 8xA800 (80G) GPU machine. The resulted model exhibits superior
  performances across a broad range of evaluation tasks, such as NIHS, topic
  retrieval, and long-context language understanding; meanwhile, it also well
  preserves the original capability over short contexts. The dramatic context
  extension is mainly attributed to merely 3.5K synthetic training samples
  generated by GPT-4 , which indicates the LLMs' inherent (yet largely
  underestimated) potential to extend its original context length. In fact, the
  context length could be extended far beyond 80K with more computation
  resources. Therefore, the team will publicly release the entire resources
  (including data, model, data generation pipeline, training code) so as to
  facilitate the future research from the community:
  https://github.com/FlagOpen/FlagEmbedding.
paper_authors._id: 6631b628ab69b961afed7bd9
paper_authors.name: Peitian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:11:06.041Z
paper_authors._id: 603fb3ebabc4bc39998ff765
paper_authors.avatarUrl: /avatars/1a20dec6a1e48017f9a650ac2e510b4e.svg
paper_authors.fullname: NaN
paper_authors.type: user
paper_authors.user: namespace-Pt
paper_authors._id: 6631b628ab69b961afed7bda
paper_authors.name: Ninglu Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:13:27.391Z
paper_authors._id: 653272210917932040597047
paper_authors.avatarUrl: /avatars/877a19686dd9780129cfcb85533e2e66.svg
paper_authors.fullname: Ninglu Shao
paper_authors.type: user
paper_authors.user: rainym00d
paper_authors._id: 6631b628ab69b961afed7bdb
paper_authors.name: Zheng Liu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-01T03:25:29.050Z
paper_authors._id: 64a38c590111d5ff6c3d5f2b
paper_authors.avatarUrl: /avatars/ef13dc7ce243819bc0da9b04e778b432.svg
paper_authors.fullname: zhengliu
paper_authors.type: user
paper_authors.user: zl101
paper_authors._id: 6631b628ab69b961afed7bdc
paper_authors.name: Shitao Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:13:36.394Z
paper_authors._id: 62612679bbcbd1c34f1638af
paper_authors.avatarUrl: /avatars/c0675d05a52192ee14e9ab1633353956.svg
paper_authors.fullname: Xiao
paper_authors.type: user
paper_authors.user: Shitao
paper_authors._id: 6631b628ab69b961afed7bdd
paper_authors.name: Hongjin Qian
paper_authors._id: 6631b628ab69b961afed7bde
paper_authors.name: Qiwei Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:13:49.819Z
paper_authors._id: 62e9de1b4fa4bc6de6c36b7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e9de1b4fa4bc6de6c36b7c/JbLi6POVPaTSZGcF-9ddQ.jpeg
paper_authors.fullname: Qiwei Ye
paper_authors.type: user
paper_authors.user: aeros0ul
paper_authors._id: 6631b628ab69b961afed7bdf
paper_authors.name: Zhicheng Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T06:59:00.856Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_title: Extending Llama-3's Context Ten-Fold Overnight
paper_summary: We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA
  fine-tuning. The entire training cycle is super efficient, which takes 8 hours
  on one 8xA800 (80G) GPU machine. The resulted model exhibits superior
  performances across a broad range of evaluation tasks, such as NIHS, topic
  retrieval, and long-context language understanding; meanwhile, it also well
  preserves the original capability over short contexts. The dramatic context
  extension is mainly attributed to merely 3.5K synthetic training samples
  generated by GPT-4 , which indicates the LLMs' inherent (yet largely
  underestimated) potential to extend its original context length. In fact, the
  context length could be extended far beyond 80K with more computation
  resources. Therefore, the team will publicly release the entire resources
  (including data, model, data generation pipeline, training code) so as to
  facilitate the future research from the community:
  https://github.com/FlagOpen/FlagEmbedding.
paper_ai_summary: Llama-3-8B-Instruct's context length is extended from 8K to 80K using QLoRA fine-tuning with minimal additional training samples, demonstrating significant potential for further context extension with increased computational resources.
paper_ai_keywords: QLoRA fine-tuning