title: Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
summary: The utilization of long contexts poses a big challenge for large language
  models due to their limited context window length. Although the context window
  can be extended through fine-tuning, it will result in a considerable cost at
  both training and inference time, and exert an unfavorable impact to the LLM's
  original capabilities. In this work, we propose Activation Beacon, which
  condenses LLM's raw activations into more compact forms such that it can
  perceive a much longer context with a limited context window. Activation Beacon
  is introduced as a plug-and-play module for the LLM. It fully preserves the
  LLM's original capability on short contexts while extending the new capability
  on processing longer contexts. Besides, it works with short sliding windows to
  process the long context, which achieves a competitive memory and time
  efficiency in both training and inference. Activation Beacon is learned by the
  auto-regression task conditioned on a mixture of beacons with diversified
  condensing ratios. Thanks to such a treatment, it can be efficiently trained
  purely with short-sequence data in just 10K steps, which consumes less than 9
  hours on a single 8xA800 GPU machine. The experimental studies show that
  Activation Beacon is able to extend Llama-2-7B's context length by times100
  times (from 4K to 400K), meanwhile achieving a superior result on both
  long-context generation and understanding tasks. Our model and code will be
  available at the BGE repository.
paper_authors._id: 659cc210c80023a02e24bd54
paper_authors.name: Peitian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-09T09:16:51.563Z
paper_authors._id: 603fb3ebabc4bc39998ff765
paper_authors.avatarUrl: /avatars/1a20dec6a1e48017f9a650ac2e510b4e.svg
paper_authors.fullname: NaN
paper_authors.type: user
paper_authors.user: namespace-Pt
paper_authors._id: 659cc210c80023a02e24bd55
paper_authors.name: Zheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-25T08:04:55.862Z
paper_authors._id: 64a38c590111d5ff6c3d5f2b
paper_authors.avatarUrl: /avatars/ef13dc7ce243819bc0da9b04e778b432.svg
paper_authors.fullname: zhengliu
paper_authors.type: user
paper_authors.user: zl101
paper_authors._id: 659cc210c80023a02e24bd56
paper_authors.name: Shitao Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:22:34.399Z
paper_authors._id: 62612679bbcbd1c34f1638af
paper_authors.avatarUrl: /avatars/c0675d05a52192ee14e9ab1633353956.svg
paper_authors.fullname: Xiao
paper_authors.type: user
paper_authors.user: Shitao
paper_authors._id: 659cc210c80023a02e24bd57
paper_authors.name: Ninglu Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:22:44.167Z
paper_authors._id: 653272210917932040597047
paper_authors.avatarUrl: /avatars/877a19686dd9780129cfcb85533e2e66.svg
paper_authors.fullname: Ninglu Shao
paper_authors.type: user
paper_authors.user: rainym00d
paper_authors._id: 659cc210c80023a02e24bd58
paper_authors.name: Qiwei Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:22:51.963Z
paper_authors._id: 62e9de1b4fa4bc6de6c36b7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e9de1b4fa4bc6de6c36b7c/JbLi6POVPaTSZGcF-9ddQ.jpeg
paper_authors.fullname: Qiwei Ye
paper_authors.type: user
paper_authors.user: aeros0ul
paper_authors._id: 659cc210c80023a02e24bd59
paper_authors.name: Zhicheng Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T06:59:35.122Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_title: Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
paper_summary: The utilization of long contexts poses a big challenge for large language
  models due to their limited context window length. Although the context window
  can be extended through fine-tuning, it will result in a considerable cost at
  both training and inference time, and exert an unfavorable impact to the LLM's
  original capabilities. In this work, we propose Activation Beacon, which
  condenses LLM's raw activations into more compact forms such that it can
  perceive a much longer context with a limited context window. Activation Beacon
  is introduced as a plug-and-play module for the LLM. It fully preserves the
  LLM's original capability on short contexts while extending the new capability
  on processing longer contexts. Besides, it works with short sliding windows to
  process the long context, which achieves a competitive memory and time
  efficiency in both training and inference. Activation Beacon is learned by the
  auto-regression task conditioned on a mixture of beacons with diversified
  condensing ratios. Thanks to such a treatment, it can be efficiently trained
  purely with short-sequence data in just 10K steps, which consumes less than 9
  hours on a single 8xA800 GPU machine. The experimental studies show that
  Activation Beacon is able to extend Llama-2-7B's context length by times100
  times (from 4K to 400K), meanwhile achieving a superior result on both
  long-context generation and understanding tasks. Our model and code will be
  available at the BGE repository.
paper_ai_summary: Activation Beacon enhances large language models' ability to process long contexts using compact activation forms, maintaining original capabilities and improving efficiency.
paper_ai_keywords: long contexts large language models context window fine-tuning activation beacon raw activations plug-and-play module short contexts short sliding windows memory efficiency time efficiency auto-regression task diversified condensing ratios BGE repository