title: From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence
    Generation up to 100K Tokens
summary: Generating ultra-long sequences with large language models (LLMs) has become
  increasingly crucial but remains a highly time-intensive task, particularly for
  sequences up to 100K tokens. While traditional speculative decoding methods
  exist, simply extending their generation limits fails to accelerate the process
  and can be detrimental. Through an in-depth analysis, we identify three major
  challenges hindering efficient generation: frequent model reloading, dynamic
  key-value (KV) management and repetitive generation. To address these issues,
  we introduce TOKENSWIFT, a novel framework designed to substantially accelerate
  the generation process of ultra-long sequences while maintaining the target
  model's inherent quality. Experimental results demonstrate that TOKENSWIFT
  achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B,
  14B) and architectures (MHA, GQA). This acceleration translates to hours of
  time savings for ultra-long sequence generation, establishing TOKENSWIFT as a
  scalable and effective solution at unprecedented lengths. Code can be found at
  https://github.com/bigai-nlco/TokenSwift.
paper_authors._id: 67c6cbd6e52534aa6ada2e26
paper_authors.name: Tong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:58:45.670Z
paper_authors._id: 668f7fee5156d55f72af4f21
paper_authors.avatarUrl: /avatars/02edf8d7d5f288d80dc665b18dda4d0a.svg
paper_authors.fullname: TongWu
paper_authors.type: user
paper_authors.user: TongWu
paper_authors._id: 67c6cbd6e52534aa6ada2e27
paper_authors.name: Junzhe Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:58:27.834Z
paper_authors._id: 6530c9d7d107f378e105d667
paper_authors.avatarUrl: /avatars/889dfcb6514c90351802bebb4a34a78f.svg
paper_authors.fullname: Junzhe Shen
paper_authors.type: user
paper_authors.user: JunzheS
paper_authors._id: 67c6cbd6e52534aa6ada2e28
paper_authors.name: Zixia Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:58:34.128Z
paper_authors._id: 64b7ae6cf53ae848e72b997d
paper_authors.avatarUrl: /avatars/b55dd3d6fcb3ccac2e3880d01a9bdc63.svg
paper_authors.fullname: Zixia Jia
paper_authors.type: user
paper_authors.user: vickyandkekey
paper_authors._id: 67c6cbd6e52534aa6ada2e29
paper_authors.name: Yuxuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:12:27.432Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 67c6cbd6e52534aa6ada2e2a
paper_authors.name: Zilong Zheng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-04T09:45:59.571Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_title: From Hours to Minutes: Lossless Acceleration of Ultra Long Sequence
    Generation up to 100K Tokens
paper_summary: Generating ultra-long sequences with large language models (LLMs) has become
  increasingly crucial but remains a highly time-intensive task, particularly for
  sequences up to 100K tokens. While traditional speculative decoding methods
  exist, simply extending their generation limits fails to accelerate the process
  and can be detrimental. Through an in-depth analysis, we identify three major
  challenges hindering efficient generation: frequent model reloading, dynamic
  key-value (KV) management and repetitive generation. To address these issues,
  we introduce TOKENSWIFT, a novel framework designed to substantially accelerate
  the generation process of ultra-long sequences while maintaining the target
  model's inherent quality. Experimental results demonstrate that TOKENSWIFT
  achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B,
  14B) and architectures (MHA, GQA). This acceleration translates to hours of
  time savings for ultra-long sequence generation, establishing TOKENSWIFT as a
  scalable and effective solution at unprecedented lengths. Code can be found at
  https://github.com/bigai-nlco/TokenSwift.
paper_ai_keywords: speculative decoding frequent model reloading dynamic key-value (KV) management repetitive generation TOKENSWIFT ultra-long sequences speedup MHA GQA scalability large language models (LLMs)