Extending Llama-3's Context Ten-Fold Overnight
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA
fine-tuning. The entire training cycle is super efficient, which takes 8 hours
on one 8xA800 (80G) GPU machine. The resulted model exhibits superior
performances across a broad range of evaluation tasks, such as NIHS, topic
retrieval, and long-context language understanding; meanwhile, it also well
preserves the original capability over short contexts. The dramatic context
extension is mainly attributed to merely 3.5K synthetic training samples
generated by GPT-4 , which indicates the LLMs' inherent (yet largely
underestimated) potential to extend its original context length. In fact, the
context length could be extended far beyond 80K with more computation
resources. Therefore, the team will publicly release the entire resources
(including data, model, data generation pipeline, training code) so as to
facilitate the future research from the community:
https://github.com/FlagOpen/FlagEmbedding.
6631b628ab69b961afed7bd9
Peitian Zhang
claimed_verified
2024-05-02T07:11:06.041Z
603fb3ebabc4bc39998ff765
/avatars/1a20dec6a1e48017f9a650ac2e510b4e.svg
NaN
user
namespace-Pt
6631b628ab69b961afed7bda
Ninglu Shao
admin_assigned
2024-05-02T08:13:27.391Z
653272210917932040597047
/avatars/877a19686dd9780129cfcb85533e2e66.svg
Ninglu Shao
user
rainym00d
6631b628ab69b961afed7bdb
Zheng Liu
extracted_pending
2024-05-01T03:25:29.050Z
64a38c590111d5ff6c3d5f2b
/avatars/ef13dc7ce243819bc0da9b04e778b432.svg
zhengliu
user
zl101
6631b628ab69b961afed7bdc
Shitao Xiao
admin_assigned
2024-05-02T08:13:36.394Z
62612679bbcbd1c34f1638af
/avatars/c0675d05a52192ee14e9ab1633353956.svg
Xiao
user
Shitao
6631b628ab69b961afed7bdd
Hongjin Qian
6631b628ab69b961afed7bde
Qiwei Ye
admin_assigned
2024-05-02T08:13:49.819Z
62e9de1b4fa4bc6de6c36b7c
https://cdn-avatars.huggingface.co/v1/production/uploads/62e9de1b4fa4bc6de6c36b7c/JbLi6POVPaTSZGcF-9ddQ.jpeg
Qiwei Ye
user
aeros0ul
6631b628ab69b961afed7bdf
Zhicheng Dou
claimed_verified
2024-09-23T06:59:00.856Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
Extending Llama-3's Context Ten-Fold Overnight
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA
fine-tuning. The entire training cycle is super efficient, which takes 8 hours
on one 8xA800 (80G) GPU machine. The resulted model exhibits superior
performances across a broad range of evaluation tasks, such as NIHS, topic
retrieval, and long-context language understanding; meanwhile, it also well
preserves the original capability over short contexts. The dramatic context
extension is mainly attributed to merely 3.5K synthetic training samples
generated by GPT-4 , which indicates the LLMs' inherent (yet largely
underestimated) potential to extend its original context length. In fact, the
context length could be extended far beyond 80K with more computation
resources. Therefore, the team will publicly release the entire resources
(including data, model, data generation pipeline, training code) so as to
facilitate the future research from the community:
https://github.com/FlagOpen/FlagEmbedding.
Llama-3-8B-Instruct's context length is extended from 8K to 80K using QLoRA fine-tuning with minimal additional training samples, demonstrating significant potential for further context extension with increased computational resources.
QLoRA fine-tuning