Sigma: Differential Rescaling of Query, Key and Value for Efficient
  Language Models
We introduce Sigma, an efficient large language model specialized for the
system domain, empowered by a novel architecture including DiffQKV attention,
and pre-trained on our meticulously collected system domain data. DiffQKV
attention significantly enhances the inference efficiency of Sigma by
optimizing the Query (Q), Key (K), and Value (V) components in the attention
mechanism differentially, based on their varying impacts on the model
performance and efficiency indicators. Specifically, we (1) conduct extensive
experiments that demonstrate the model's varying sensitivity to the compression
of K and V components, leading to the development of differentially compressed
KV, and (2) propose augmented Q to expand the Q head dimension, which enhances
the model's representation capacity with minimal impacts on the inference
speed. Rigorous theoretical and empirical analyses reveal that DiffQKV
attention significantly enhances efficiency, achieving up to a 33.36%
improvement in inference speed over the conventional grouped-query attention
(GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various
sources, including 19.5B system domain data that we carefully collect and 1T
tokens of synthesized and rewritten data. In general domains, Sigma achieves
comparable performance to other state-of-arts models. In the system domain, we
introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates
remarkable performance across all tasks, significantly outperforming GPT-4 with
an absolute improvement up to 52.5%.
6792f8ed5e3ec6035dafb06a
Zhenghao Lin
admin_assigned
2025-01-24T09:14:52.584Z
63776f1806241efce1e7aae6
/avatars/d67d9dcd932934c630f407ac152f2ce6.svg
Zhenghao Lin
user
Lin0
6792f8ed5e3ec6035dafb06b
Zihao Tang
admin_assigned
2025-01-24T09:15:04.802Z
656c6bd8e0ff1cebe966aa35
/avatars/1083cb58bdb0bee72036953276d42e13.svg
tangzihao
user
tzh94588
6792f8ed5e3ec6035dafb06c
Xiao Liu
claimed_verified
2025-01-24T09:08:05.797Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
6792f8ed5e3ec6035dafb06d
Yeyun Gong
admin_assigned
2025-01-24T09:58:33.228Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
6792f8ed5e3ec6035dafb06e
Yi Cheng
6792f8ed5e3ec6035dafb06f
Qi Chen
6792f8ed5e3ec6035dafb070
Hang Li
admin_assigned
2025-01-24T10:03:59.680Z
61342a4b488458a484dee6c4
https://cdn-avatars.huggingface.co/v1/production/uploads/1630808595161-noauth.png
Hang Li
user
hanglics
6792f8ed5e3ec6035dafb071
Ying Xin
6792f8ed5e3ec6035dafb072
Ziyue Yang
admin_assigned
2025-01-24T10:04:09.709Z
62f6a9add3bdacb7eec0d4f5
https://cdn-avatars.huggingface.co/v1/production/uploads/1660332390183-noauth.jpeg
Ziyue Yang
user
ziyueyang37
6792f8ed5e3ec6035dafb073
Kailai Yang
admin_assigned
2025-01-24T10:04:16.033Z
646fc402e9c03ba436d5e93e
/avatars/870c86dc99fb1cb6a348a7a0385b1a04.svg
Kailai Yang
user
klyang
6792f8ed5e3ec6035dafb074
Yu Yan
6792f8ed5e3ec6035dafb075
Xiao Liang
6792f8ed5e3ec6035dafb076
Shuai Lu
6792f8ed5e3ec6035dafb077
Yiming Huang
6792f8ed5e3ec6035dafb078
Zheheng Luo
admin_assigned
2025-01-24T10:04:49.140Z
6443bb593c323e0918f61a96
/avatars/b9e1ba17f7798b5142bc0124fba95237.svg
zheheng luo
user
KenLuo
6792f8ed5e3ec6035dafb079
Lei Qu
6792f8ed5e3ec6035dafb07a
Xuan Feng
6792f8ed5e3ec6035dafb07b
Yaoxiang Wang
6792f8ed5e3ec6035dafb07c
Yuqing Xia
admin_assigned
2025-01-24T10:05:26.287Z
6369e01864aad59d4d4501ac
/avatars/bcbd3f9d0d194eeccd061c4fa6a6e283.svg
Yuqing Xia
user
yuqxia
6792f8ed5e3ec6035dafb07d
Feiyang Chen
admin_assigned
2025-01-24T10:05:34.991Z
673fd856a45b6f21829a3bf5
/avatars/deb8c5362fad22019cccaed6d03dea09.svg
Feiyang Chen
user
PhilipChen
6792f8ed5e3ec6035dafb07e
Yuting Jiang
admin_assigned
2025-01-24T10:05:41.151Z
64e85f4e5ddcace745bc0a55
/avatars/e316355b913c73104db530010ceedeb4.svg
Yuting Jiang
user
Stautinger
6792f8ed5e3ec6035dafb07f
Yasen Hu
6792f8ed5e3ec6035dafb080
Hao Ni
6792f8ed5e3ec6035dafb081
Binyang Li
admin_assigned
2025-01-24T10:06:03.263Z
6485714cfc41a0b97fe377cc
/avatars/0af8a3df9ad711a5eac739bce26c4c2a.svg
Li
user
Binyang
6792f8ed5e3ec6035dafb082
Guoshuai Zhao
admin_assigned
2025-01-24T10:05:17.780Z
663de80ca920d195191807da
/avatars/2437ce3fa073a07b971d370c26c7ab65.svg
Guoshuai Zhao
user
crayonshine
6792f8ed5e3ec6035dafb083
Jui-Hao Chiang
6792f8ed5e3ec6035dafb084
Zhongxin Guo
6792f8ed5e3ec6035dafb085
Chen Lin
6792f8ed5e3ec6035dafb086
Kun Kuang
6792f8ed5e3ec6035dafb087
Wenjie Li
admin_assigned
2025-01-24T10:06:22.951Z
66a3710a4ee2a4c936315a5a
/avatars/ef8da8fb1031695d77d34a5d365aa177.svg
Li
user
WenjieLi
6792f8ed5e3ec6035dafb088
Yelong Shen
admin_assigned
2025-01-24T10:06:30.109Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
6792f8ed5e3ec6035dafb089
Jian Jiao
6792f8ed5e3ec6035dafb08a
Peng Cheng
6792f8ed5e3ec6035dafb08b
Mao Yang
Sigma: Differential Rescaling of Query, Key and Value for Efficient
  Language Models
We introduce Sigma, an efficient large language model specialized for the
system domain, empowered by a novel architecture including DiffQKV attention,
and pre-trained on our meticulously collected system domain data. DiffQKV
attention significantly enhances the inference efficiency of Sigma by
optimizing the Query (Q), Key (K), and Value (V) components in the attention
mechanism differentially, based on their varying impacts on the model
performance and efficiency indicators. Specifically, we (1) conduct extensive
experiments that demonstrate the model's varying sensitivity to the compression
of K and V components, leading to the development of differentially compressed
KV, and (2) propose augmented Q to expand the Q head dimension, which enhances
the model's representation capacity with minimal impacts on the inference
speed. Rigorous theoretical and empirical analyses reveal that DiffQKV
attention significantly enhances efficiency, achieving up to a 33.36%
improvement in inference speed over the conventional grouped-query attention
(GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various
sources, including 19.5B system domain data that we carefully collect and 1T
tokens of synthesized and rewritten data. In general domains, Sigma achieves
comparable performance to other state-of-arts models. In the system domain, we
introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates
remarkable performance across all tasks, significantly outperforming GPT-4 with
an absolute improvement up to 52.5%.
Sigma, a large language model for the system domain, uses DiffQKV attention to enhance inference efficiency and achieves exceptional performance compared to GPT-4 on a new benchmark called AIMicius.
DiffQKV attention Query (Q) Key (K) Value (V) attention mechanism compressed KV augmented Q grouped-query attention (GQA) pre-training systematic domain data synthetic data rewritten data AIMicius benchmark