title: SepLLM: Accelerate Large Language Models by Compressing One Segment into
    One Separator
summary: Large Language Models (LLMs) have exhibited exceptional performance across a
  spectrum of natural language processing tasks. However, their substantial sizes
  pose considerable challenges, particularly in computational demands and
  inference speed, due to their quadratic complexity. In this work, we have
  identified a key pattern: certain seemingly meaningless special tokens (i.e.,
  separators) contribute disproportionately to attention scores compared to
  semantically meaningful tokens. This observation suggests that information of
  the segments between these separator tokens can be effectively condensed into
  the separator tokens themselves without significant information loss. Guided by
  this insight, we introduce SepLLM, a plug-and-play framework that accelerates
  inference by compressing these segments and eliminating redundant tokens.
  Additionally, we implement efficient kernels for training acceleration.
  Experimental results across training-free, training-from-scratch, and
  post-training settings demonstrate SepLLM's effectiveness. Notably, using the
  Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the
  GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in
  streaming settings, SepLLM effectively processes sequences of up to 4 million
  tokens or more while maintaining consistent language modeling capabilities.
paper_authors._id: 67615d54402071c9e07c0c42
paper_authors.name: Guoxuan Chen
paper_authors._id: 67615d54402071c9e07c0c43
paper_authors.name: Han Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T11:18:01.539Z
paper_authors._id: 637751ad7169ed981703d26a
paper_authors.avatarUrl: /avatars/9823f115040a3d6b0240038010e46cb6.svg
paper_authors.fullname: Han Shi
paper_authors.type: user
paper_authors.user: shihan96
paper_authors._id: 67615d54402071c9e07c0c44
paper_authors.name: Jiawei Li
paper_authors._id: 67615d54402071c9e07c0c45
paper_authors.name: Yihang Gao
paper_authors._id: 67615d54402071c9e07c0c46
paper_authors.name: Xiaozhe Ren
paper_authors._id: 67615d54402071c9e07c0c47
paper_authors.name: Yimeng Chen
paper_authors._id: 67615d54402071c9e07c0c48
paper_authors.name: Xin Jiang
paper_authors._id: 67615d54402071c9e07c0c49
paper_authors.name: Zhenguo Li
paper_authors._id: 67615d54402071c9e07c0c4a
paper_authors.name: Weiyang Liu
paper_authors._id: 67615d54402071c9e07c0c4b
paper_authors.name: Chao Huang
paper_title: SepLLM: Accelerate Large Language Models by Compressing One Segment into
    One Separator
paper_summary: Large Language Models (LLMs) have exhibited exceptional performance across a
  spectrum of natural language processing tasks. However, their substantial sizes
  pose considerable challenges, particularly in computational demands and
  inference speed, due to their quadratic complexity. In this work, we have
  identified a key pattern: certain seemingly meaningless special tokens (i.e.,
  separators) contribute disproportionately to attention scores compared to
  semantically meaningful tokens. This observation suggests that information of
  the segments between these separator tokens can be effectively condensed into
  the separator tokens themselves without significant information loss. Guided by
  this insight, we introduce SepLLM, a plug-and-play framework that accelerates
  inference by compressing these segments and eliminating redundant tokens.
  Additionally, we implement efficient kernels for training acceleration.
  Experimental results across training-free, training-from-scratch, and
  post-training settings demonstrate SepLLM's effectiveness. Notably, using the
  Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the
  GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in
  streaming settings, SepLLM effectively processes sequences of up to 4 million
  tokens or more while maintaining consistent language modeling capabilities.
paper_ai_keywords: Large Language Models (LLMs) natural language processing tasks computational demands inference speed quadratic complexity special tokens separators attention scores semantically meaningful tokens SegLLM plug-and-play framework inference acceleration compressing segments eliminating redundant tokens efficient kernels training acceleration training-free training-from-scratch post-training settings GSM8K-CoT benchmark KV cache streaming settings language modeling capabilities