title: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
summary: We present LongLoRA, an efficient fine-tuning approach that extends the
  context sizes of pre-trained large language models (LLMs), with limited
  computation cost. Typically, training LLMs with long context sizes is
  computationally expensive, requiring extensive training hours and GPU
  resources. For example, training on the context length of 8192 needs 16x
  computational costs in self-attention layers as that of 2048. In this paper, we
  speed up the context extension of LLMs in two aspects. On the one hand,
  although dense global attention is needed during inference, fine-tuning the
  model can be effectively and efficiently done by sparse local attention. The
  proposed shift short attention effectively enables context extension, leading
  to non-trivial computation saving with similar performance to fine-tuning with
  vanilla attention. Particularly, it can be implemented with only two lines of
  code in training, while being optional in inference. On the other hand, we
  revisit the parameter-efficient fine-tuning regime for context expansion.
  Notably, we find that LoRA for context extension works well under the premise
  of trainable embedding and normalization. LongLoRA demonstrates strong
  empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
  LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
  single 8x A100 machine. LongLoRA extends models' context while retaining their
  original architectures, and is compatible with most existing techniques, like
  FlashAttention-2. In addition, to make LongLoRA practical, we collect a
  dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
  context question-answer pairs.
paper_authors._id: 650ce7f6cbd0c7d550c64d9c
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T14:54:21.276Z
paper_authors._id: 62919485a29097b211bc7b83
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653710384819-62919485a29097b211bc7b83.png
paper_authors.fullname: YukangChen
paper_authors.type: user
paper_authors.user: Yukang
paper_authors._id: 650ce7f6cbd0c7d550c64d9d
paper_authors.name: Shengju Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T14:54:29.002Z
paper_authors._id: 6380580f42cedbc20c7bef71
paper_authors.avatarUrl: /avatars/8d710e0de551cd2bf545cc31fcaf099d.svg
paper_authors.fullname: Shengju Qian
paper_authors.type: user
paper_authors.user: thesouthfrog
paper_authors._id: 650ce7f6cbd0c7d550c64d9e
paper_authors.name: Haotian Tang
paper_authors._id: 650ce7f6cbd0c7d550c64d9f
paper_authors.name: Xin Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T14:54:47.313Z
paper_authors._id: 642bddc1fc41757877f68327
paper_authors.avatarUrl: /avatars/f275237f36a112624d59a7e3f73237d3.svg
paper_authors.fullname: Xin Lai
paper_authors.type: user
paper_authors.user: xinlai
paper_authors._id: 650ce7f6cbd0c7d550c64da0
paper_authors.name: Zhijian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-22T15:32:56.869Z
paper_authors._id: 650dac79b959b0e1d41d7378
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
paper_authors.fullname: Zhijian Liu
paper_authors.type: user
paper_authors.user: zhijianliu
paper_authors._id: 650ce7f6cbd0c7d550c64da1
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T14:55:46.154Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_authors._id: 650ce7f6cbd0c7d550c64da2
paper_authors.name: Jiaya Jia
paper_title: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
paper_summary: We present LongLoRA, an efficient fine-tuning approach that extends the
  context sizes of pre-trained large language models (LLMs), with limited
  computation cost. Typically, training LLMs with long context sizes is
  computationally expensive, requiring extensive training hours and GPU
  resources. For example, training on the context length of 8192 needs 16x
  computational costs in self-attention layers as that of 2048. In this paper, we
  speed up the context extension of LLMs in two aspects. On the one hand,
  although dense global attention is needed during inference, fine-tuning the
  model can be effectively and efficiently done by sparse local attention. The
  proposed shift short attention effectively enables context extension, leading
  to non-trivial computation saving with similar performance to fine-tuning with
  vanilla attention. Particularly, it can be implemented with only two lines of
  code in training, while being optional in inference. On the other hand, we
  revisit the parameter-efficient fine-tuning regime for context expansion.
  Notably, we find that LoRA for context extension works well under the premise
  of trainable embedding and normalization. LongLoRA demonstrates strong
  empirical results on various tasks on LLaMA2 models from 7B/13B to 70B.
  LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a
  single 8x A100 machine. LongLoRA extends models' context while retaining their
  original architectures, and is compatible with most existing techniques, like
  FlashAttention-2. In addition, to make LongLoRA practical, we collect a
  dataset, LongQA, for supervised fine-tuning. It contains more than 3k long
  context question-answer pairs.
paper_ai_summary: LongLoRA extends LLM context sizes efficiently using sparse local attention and parameter-efficient fine-tuning, demonstrating strong results with minimal computational cost.
paper_ai_keywords: LongLoRA dense global attention sparse local attention shift short attention parameter-efficient fine-tuning LoRA trainable embedding normalization LongQA