HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
Transformer-based large language models (LLMs) demonstrate impressive
performance in long context generation. Extending the context length has
disproportionately shifted the memory footprint of LLMs during inference to the
key-value cache (KV cache). In this paper, we propose HEADINFER, which offloads
the KV cache to CPU RAM while avoiding the need to fully store the KV cache for
any transformer layer on the GPU. HEADINFER employs a fine-grained, head-wise
offloading strategy, maintaining only selective attention heads KV cache on the
GPU while computing attention output dynamically. Through roofline analysis, we
demonstrate that HEADINFER maintains computational efficiency while
significantly reducing memory footprint. We evaluate HEADINFER on the
Llama-3-8B model with a 1-million-token sequence, reducing the GPU memory
footprint of the KV cache from 128 GB to 1 GB and the total GPU memory usage
from 207 GB to 17 GB, achieving a 92% reduction compared to BF16 baseline
inference. Notably, HEADINFER enables 4-million-token inference with an 8B
model on a single consumer GPU with 24GB memory (e.g., NVIDIA RTX 4090) without
approximation methods.
67b547f555d0424a31b9c384
Cheng Luo
admin_assigned
2025-02-19T09:40:25.130Z
64cb48f7667f4f808535107e
/avatars/8f77f378ad665b246e1ea3aaba2153ae.svg
chengluo
user
wdlctc
67b547f555d0424a31b9c385
Zefan Cai
admin_assigned
2025-02-19T09:40:47.077Z
64b15284372d4340772a3dca
/avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
Zefan Cai
user
ZefanCai
67b547f555d0424a31b9c386
Hanshi Sun
67b547f555d0424a31b9c387
Jinqi Xiao
admin_assigned
2025-02-19T09:41:01.931Z
64c15c5bea792b1950e302e4
/avatars/51f84365cc08a1dcd5da70968389aed2.svg
Jinqi Xiao
user
jinqixiao
67b547f555d0424a31b9c388
Bo Yuan
67b547f555d0424a31b9c389
Wen Xiao
67b547f555d0424a31b9c38a
Junjie Hu
admin_assigned
2025-02-19T09:41:18.304Z
675f8271a63fff7b5bcbc478
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9tJn7NyzLMreCJVH4wRho.png
Junjie Hu
user
junjiehu
67b547f555d0424a31b9c38b
Jiawei Zhao
67b547f555d0424a31b9c38c
Beidi Chen
admin_assigned
2025-02-19T09:39:20.563Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
67b547f555d0424a31b9c38d
Anima Anandkumar
admin_assigned
2025-02-19T09:39:15.091Z
6532920b3e385cfc6002938d
/avatars/cb9cc6d2733031582c83f56dc6cd1dd5.svg
Anima Anandkumar
user
animakumar
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
Transformer-based large language models (LLMs) demonstrate impressive
performance in long context generation. Extending the context length has
disproportionately shifted the memory footprint of LLMs during inference to the
key-value cache (KV cache). In this paper, we propose HEADINFER, which offloads
the KV cache to CPU RAM while avoiding the need to fully store the KV cache for
any transformer layer on the GPU. HEADINFER employs a fine-grained, head-wise
offloading strategy, maintaining only selective attention heads KV cache on the
GPU while computing attention output dynamically. Through roofline analysis, we
demonstrate that HEADINFER maintains computational efficiency while
significantly reducing memory footprint. We evaluate HEADINFER on the
Llama-3-8B model with a 1-million-token sequence, reducing the GPU memory
footprint of the KV cache from 128 GB to 1 GB and the total GPU memory usage
from 207 GB to 17 GB, achieving a 92% reduction compared to BF16 baseline
inference. Notably, HEADINFER enables 4-million-token inference with an 8B
model on a single consumer GPU with 24GB memory (e.g., NVIDIA RTX 4090) without
approximation methods.
HEADINFER offloads the key-value cache to CPU RAM for transformer-based large language models, reducing GPU memory usage by maintaining only selective attention heads on the GPU.
Transformer-based large language models long context generation key-value cache KV cache HEADINFER fine-grained head-wise offloading strategy roofline analysis computational efficiency Llama-3-8B model NVIDIA RTX 4090