title: Sigma: Differential Rescaling of Query, Key and Value for Efficient
    Language Models
summary: We introduce Sigma, an efficient large language model specialized for the
  system domain, empowered by a novel architecture including DiffQKV attention,
  and pre-trained on our meticulously collected system domain data. DiffQKV
  attention significantly enhances the inference efficiency of Sigma by
  optimizing the Query (Q), Key (K), and Value (V) components in the attention
  mechanism differentially, based on their varying impacts on the model
  performance and efficiency indicators. Specifically, we (1) conduct extensive
  experiments that demonstrate the model's varying sensitivity to the compression
  of K and V components, leading to the development of differentially compressed
  KV, and (2) propose augmented Q to expand the Q head dimension, which enhances
  the model's representation capacity with minimal impacts on the inference
  speed. Rigorous theoretical and empirical analyses reveal that DiffQKV
  attention significantly enhances efficiency, achieving up to a 33.36%
  improvement in inference speed over the conventional grouped-query attention
  (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various
  sources, including 19.5B system domain data that we carefully collect and 1T
  tokens of synthesized and rewritten data. In general domains, Sigma achieves
  comparable performance to other state-of-arts models. In the system domain, we
  introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates
  remarkable performance across all tasks, significantly outperforming GPT-4 with
  an absolute improvement up to 52.5%.
paper_authors._id: 6792f8ed5e3ec6035dafb06a
paper_authors.name: Zhenghao Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T09:14:52.584Z
paper_authors._id: 63776f1806241efce1e7aae6
paper_authors.avatarUrl: /avatars/d67d9dcd932934c630f407ac152f2ce6.svg
paper_authors.fullname: Zhenghao Lin
paper_authors.type: user
paper_authors.user: Lin0
paper_authors._id: 6792f8ed5e3ec6035dafb06b
paper_authors.name: Zihao Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T09:15:04.802Z
paper_authors._id: 656c6bd8e0ff1cebe966aa35
paper_authors.avatarUrl: /avatars/1083cb58bdb0bee72036953276d42e13.svg
paper_authors.fullname: tangzihao
paper_authors.type: user
paper_authors.user: tzh94588
paper_authors._id: 6792f8ed5e3ec6035dafb06c
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:08:05.797Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 6792f8ed5e3ec6035dafb06d
paper_authors.name: Yeyun Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T09:58:33.228Z
paper_authors._id: 643f615aa16cd6d1f4c581de
paper_authors.avatarUrl: /avatars/47753a3e82b44f81881600c52e1e8495.svg
paper_authors.fullname: Yeyun Gong
paper_authors.type: user
paper_authors.user: yegong
paper_authors._id: 6792f8ed5e3ec6035dafb06e
paper_authors.name: Yi Cheng
paper_authors._id: 6792f8ed5e3ec6035dafb06f
paper_authors.name: Qi Chen
paper_authors._id: 6792f8ed5e3ec6035dafb070
paper_authors.name: Hang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:03:59.680Z
paper_authors._id: 61342a4b488458a484dee6c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1630808595161-noauth.png
paper_authors.fullname: Hang Li
paper_authors.type: user
paper_authors.user: hanglics
paper_authors._id: 6792f8ed5e3ec6035dafb071
paper_authors.name: Ying Xin
paper_authors._id: 6792f8ed5e3ec6035dafb072
paper_authors.name: Ziyue Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:04:09.709Z
paper_authors._id: 62f6a9add3bdacb7eec0d4f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660332390183-noauth.jpeg
paper_authors.fullname: Ziyue Yang
paper_authors.type: user
paper_authors.user: ziyueyang37
paper_authors._id: 6792f8ed5e3ec6035dafb073
paper_authors.name: Kailai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:04:16.033Z
paper_authors._id: 646fc402e9c03ba436d5e93e
paper_authors.avatarUrl: /avatars/870c86dc99fb1cb6a348a7a0385b1a04.svg
paper_authors.fullname: Kailai Yang
paper_authors.type: user
paper_authors.user: klyang
paper_authors._id: 6792f8ed5e3ec6035dafb074
paper_authors.name: Yu Yan
paper_authors._id: 6792f8ed5e3ec6035dafb075
paper_authors.name: Xiao Liang
paper_authors._id: 6792f8ed5e3ec6035dafb076
paper_authors.name: Shuai Lu
paper_authors._id: 6792f8ed5e3ec6035dafb077
paper_authors.name: Yiming Huang
paper_authors._id: 6792f8ed5e3ec6035dafb078
paper_authors.name: Zheheng Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:04:49.140Z
paper_authors._id: 6443bb593c323e0918f61a96
paper_authors.avatarUrl: /avatars/b9e1ba17f7798b5142bc0124fba95237.svg
paper_authors.fullname: zheheng luo
paper_authors.type: user
paper_authors.user: KenLuo
paper_authors._id: 6792f8ed5e3ec6035dafb079
paper_authors.name: Lei Qu
paper_authors._id: 6792f8ed5e3ec6035dafb07a
paper_authors.name: Xuan Feng
paper_authors._id: 6792f8ed5e3ec6035dafb07b
paper_authors.name: Yaoxiang Wang
paper_authors._id: 6792f8ed5e3ec6035dafb07c
paper_authors.name: Yuqing Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:05:26.287Z
paper_authors._id: 6369e01864aad59d4d4501ac
paper_authors.avatarUrl: /avatars/bcbd3f9d0d194eeccd061c4fa6a6e283.svg
paper_authors.fullname: Yuqing Xia
paper_authors.type: user
paper_authors.user: yuqxia
paper_authors._id: 6792f8ed5e3ec6035dafb07d
paper_authors.name: Feiyang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:05:34.991Z
paper_authors._id: 673fd856a45b6f21829a3bf5
paper_authors.avatarUrl: /avatars/deb8c5362fad22019cccaed6d03dea09.svg
paper_authors.fullname: Feiyang Chen
paper_authors.type: user
paper_authors.user: PhilipChen
paper_authors._id: 6792f8ed5e3ec6035dafb07e
paper_authors.name: Yuting Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:05:41.151Z
paper_authors._id: 64e85f4e5ddcace745bc0a55
paper_authors.avatarUrl: /avatars/e316355b913c73104db530010ceedeb4.svg
paper_authors.fullname: Yuting Jiang
paper_authors.type: user
paper_authors.user: Stautinger
paper_authors._id: 6792f8ed5e3ec6035dafb07f
paper_authors.name: Yasen Hu
paper_authors._id: 6792f8ed5e3ec6035dafb080
paper_authors.name: Hao Ni
paper_authors._id: 6792f8ed5e3ec6035dafb081
paper_authors.name: Binyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:06:03.263Z
paper_authors._id: 6485714cfc41a0b97fe377cc
paper_authors.avatarUrl: /avatars/0af8a3df9ad711a5eac739bce26c4c2a.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Binyang
paper_authors._id: 6792f8ed5e3ec6035dafb082
paper_authors.name: Guoshuai Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:05:17.780Z
paper_authors._id: 663de80ca920d195191807da
paper_authors.avatarUrl: /avatars/2437ce3fa073a07b971d370c26c7ab65.svg
paper_authors.fullname: Guoshuai Zhao
paper_authors.type: user
paper_authors.user: crayonshine
paper_authors._id: 6792f8ed5e3ec6035dafb083
paper_authors.name: Jui-Hao Chiang
paper_authors._id: 6792f8ed5e3ec6035dafb084
paper_authors.name: Zhongxin Guo
paper_authors._id: 6792f8ed5e3ec6035dafb085
paper_authors.name: Chen Lin
paper_authors._id: 6792f8ed5e3ec6035dafb086
paper_authors.name: Kun Kuang
paper_authors._id: 6792f8ed5e3ec6035dafb087
paper_authors.name: Wenjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:06:22.951Z
paper_authors._id: 66a3710a4ee2a4c936315a5a
paper_authors.avatarUrl: /avatars/ef8da8fb1031695d77d34a5d365aa177.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: WenjieLi
paper_authors._id: 6792f8ed5e3ec6035dafb088
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:06:30.109Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 6792f8ed5e3ec6035dafb089
paper_authors.name: Jian Jiao
paper_authors._id: 6792f8ed5e3ec6035dafb08a
paper_authors.name: Peng Cheng
paper_authors._id: 6792f8ed5e3ec6035dafb08b
paper_authors.name: Mao Yang
paper_title: Sigma: Differential Rescaling of Query, Key and Value for Efficient
    Language Models
paper_summary: We introduce Sigma, an efficient large language model specialized for the
  system domain, empowered by a novel architecture including DiffQKV attention,
  and pre-trained on our meticulously collected system domain data. DiffQKV
  attention significantly enhances the inference efficiency of Sigma by
  optimizing the Query (Q), Key (K), and Value (V) components in the attention
  mechanism differentially, based on their varying impacts on the model
  performance and efficiency indicators. Specifically, we (1) conduct extensive
  experiments that demonstrate the model's varying sensitivity to the compression
  of K and V components, leading to the development of differentially compressed
  KV, and (2) propose augmented Q to expand the Q head dimension, which enhances
  the model's representation capacity with minimal impacts on the inference
  speed. Rigorous theoretical and empirical analyses reveal that DiffQKV
  attention significantly enhances efficiency, achieving up to a 33.36%
  improvement in inference speed over the conventional grouped-query attention
  (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various
  sources, including 19.5B system domain data that we carefully collect and 1T
  tokens of synthesized and rewritten data. In general domains, Sigma achieves
  comparable performance to other state-of-arts models. In the system domain, we
  introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates
  remarkable performance across all tasks, significantly outperforming GPT-4 with
  an absolute improvement up to 52.5%.
paper_ai_summary: Sigma, a large language model for the system domain, uses DiffQKV attention to enhance inference efficiency and achieves exceptional performance compared to GPT-4 on a new benchmark called AIMicius.
paper_ai_keywords: DiffQKV attention Query (Q) Key (K) Value (V) attention mechanism compressed KV augmented Q grouped-query attention (GQA) pre-training systematic domain data synthetic data rewritten data AIMicius benchmark