Star Attention: Efficient LLM Inference over Long Sequences
Inference with Transformer-based Large Language Models (LLMs) on long
sequences is both costly and slow due to the quadratic complexity of the
self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
approximation that improves computational efficiency by sharding attention
across multiple hosts while minimizing communication overhead. In the first
phase, the context is processed using blockwise-local attention across hosts,
in parallel. In the second phase, query and response tokens attend to all prior
cached tokens through sequence-global attention. Star Attention integrates
seamlessly with most Transformer-based LLMs trained with global attention,
reducing memory requirements and inference time by up to 11x while preserving
95-100% of accuracy.
67467818ee80399924ef20cc
Shantanu Acharya
claimed_verified
2024-11-27T22:20:27.963Z
626c5978f451470f86251117
https://cdn-avatars.huggingface.co/v1/production/uploads/626c5978f451470f86251117/KgtS2aFEMsSF1z79T3sob.jpeg
Shantanu Acharya
user
shantanuacharya
67467818ee80399924ef20cd
Fei Jia
67467818ee80399924ef20ce
Boris Ginsburg
Star Attention: Efficient LLM Inference over Long Sequences
Inference with Transformer-based Large Language Models (LLMs) on long
sequences is both costly and slow due to the quadratic complexity of the
self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
approximation that improves computational efficiency by sharding attention
across multiple hosts while minimizing communication overhead. In the first
phase, the context is processed using blockwise-local attention across hosts,
in parallel. In the second phase, query and response tokens attend to all prior
cached tokens through sequence-global attention. Star Attention integrates
seamlessly with most Transformer-based LLMs trained with global attention,
reducing memory requirements and inference time by up to 11x while preserving
95-100% of accuracy.
Star Attention improves inference efficiency of large language models on long sequences by using block-sparse approximation, reducing memory and time without significant accuracy loss.
Transformer-based Large Language Models (LLMs) self-attention mechanism block-sparse approximation Star Attention blockwise-local attention sequence-global attention memory requirements inference time accuracy