title: Star Attention: Efficient LLM Inference over Long Sequences
summary: Inference with Transformer-based Large Language Models (LLMs) on long
  sequences is both costly and slow due to the quadratic complexity of the
  self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
  approximation that improves computational efficiency by sharding attention
  across multiple hosts while minimizing communication overhead. In the first
  phase, the context is processed using blockwise-local attention across hosts,
  in parallel. In the second phase, query and response tokens attend to all prior
  cached tokens through sequence-global attention. Star Attention integrates
  seamlessly with most Transformer-based LLMs trained with global attention,
  reducing memory requirements and inference time by up to 11x while preserving
  95-100% of accuracy.
paper_authors._id: 67467818ee80399924ef20cc
paper_authors.name: Shantanu Acharya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:27.963Z
paper_authors._id: 626c5978f451470f86251117
paper_authors.avatarUrl: /avatars/c2ecdf2d4eb8ca1c14ba427933cc38b0.svg
paper_authors.fullname: Shantanu Acharya
paper_authors.type: user
paper_authors.user: shantanuacharya
paper_authors._id: 67467818ee80399924ef20cd
paper_authors.name: Fei Jia
paper_authors._id: 67467818ee80399924ef20ce
paper_authors.name: Boris Ginsburg
paper_title: Star Attention: Efficient LLM Inference over Long Sequences
paper_summary: Inference with Transformer-based Large Language Models (LLMs) on long
  sequences is both costly and slow due to the quadratic complexity of the
  self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
  approximation that improves computational efficiency by sharding attention
  across multiple hosts while minimizing communication overhead. In the first
  phase, the context is processed using blockwise-local attention across hosts,
  in parallel. In the second phase, query and response tokens attend to all prior
  cached tokens through sequence-global attention. Star Attention integrates
  seamlessly with most Transformer-based LLMs trained with global attention,
  reducing memory requirements and inference time by up to 11x while preserving
  95-100% of accuracy.
paper_ai_keywords: Transformer-based Large Language Models (LLMs) self-attention Star Attention two-phase block-sparse approximation sharding attention communication overhead blockwise-local attention sequence sequence-global attention memory requirements inference time accuracy