Cost-Optimal Grouped-Query Attention for Long-Context LLMs
Building effective and efficient Transformer-based large language models
(LLMs) has recently become a research focus, requiring maximizing model
language capabilities and minimizing training and deployment costs. Existing
efforts have primarily described complex relationships among model performance,
parameter size, and data size, as well as searched for the optimal compute
allocation to train LLMs. However, they overlook the impacts of context length
and attention head configuration (the number of query and key-value heads in
grouped-query attention) on training and inference. In this paper, we
systematically compare models with different parameter sizes, context lengths,
and attention head configurations in terms of model performance, computational
cost, and memory cost. Then, we extend the existing scaling methods, which are
based solely on parameter size and training compute, to guide the construction
of cost-optimal LLMs during both training and inference. Our quantitative
scaling studies show that, when processing sufficiently long sequences, a
larger model with fewer attention heads can achieve a lower loss while
incurring lower computational and memory costs. Our findings provide valuable
insights for developing practical LLMs, especially in long-context processing
scenarios. We will publicly release our code and data.
67d26ae40a955727b9687d8f
Yingfa Chen
claimed_verified
2025-03-13T08:24:03.831Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
67d26ae40a955727b9687d90
Yutong Wu
67d26ae40a955727b9687d91
Xu Han
67d26ae40a955727b9687d92
Zhiyuan Liu
claimed_verified
2025-06-02T07:52:30.729Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
67d26ae40a955727b9687d93
Maosong Sun
Cost-Optimal Grouped-Query Attention for Long-Context LLMs
Building effective and efficient Transformer-based large language models
(LLMs) has recently become a research focus, requiring maximizing model
language capabilities and minimizing training and deployment costs. Existing
efforts have primarily described complex relationships among model performance,
parameter size, and data size, as well as searched for the optimal compute
allocation to train LLMs. However, they overlook the impacts of context length
and attention head configuration (the number of query and key-value heads in
grouped-query attention) on training and inference. In this paper, we
systematically compare models with different parameter sizes, context lengths,
and attention head configurations in terms of model performance, computational
cost, and memory cost. Then, we extend the existing scaling methods, which are
based solely on parameter size and training compute, to guide the construction
of cost-optimal LLMs during both training and inference. Our quantitative
scaling studies show that, when processing sufficiently long sequences, a
larger model with fewer attention heads can achieve a lower loss while
incurring lower computational and memory costs. Our findings provide valuable
insights for developing practical LLMs, especially in long-context processing
scenarios. We will publicly release our code and data.
Research compares different configurations of Transformer-based large language models to optimize performance, cost, and memory usage, especially for long sequences.
Transformer-based large language models context length attention head configuration model performance computational cost memory cost parameter size grouped-query attention