title: ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training
    Multiplication-Less Reparameterization
summary: Large language models (LLMs) have shown impressive performance on language
  tasks but face challenges when deployed on resource-constrained devices due to
  their extensive parameters and reliance on dense multiplications, resulting in
  high memory demands and latency bottlenecks. Shift-and-add reparameterization
  offers a promising solution by replacing costly multiplications with
  hardware-friendly primitives in both the attention and multi-layer perceptron
  (MLP) layers of an LLM. However, current reparameterization techniques require
  training from scratch or full parameter fine-tuning to restore accuracy, which
  is resource-intensive for LLMs. To address this, we propose accelerating
  pretrained LLMs through post-training shift-and-add reparameterization,
  creating efficient multiplication-free models, dubbed ShiftAddLLM.
  Specifically, we quantize each weight matrix into binary matrices paired with
  group-wise scaling factors. The associated multiplications are reparameterized
  into (1) shifts between activations and scaling factors and (2) queries and
  adds according to the binary matrices. To reduce accuracy loss, we present a
  multi-objective optimization method to minimize both weight and output
  activation reparameterization errors. Additionally, based on varying
  sensitivity across layers to reparameterization, we develop an automated bit
  allocation strategy to further reduce memory usage and latency. Experiments on
  five LLM families and eight tasks consistently validate the effectiveness of
  ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points
  at comparable or lower latency compared to the most competitive quantized LLMs
  at 3 and 2 bits, respectively, and more than 80% memory and energy reductions
  over the original LLMs. Codes and models are available at
  https://github.com/GATECH-EIC/ShiftAddLLM.
paper_authors._id: 6667cef90c21d689e1936d2a
paper_authors.name: Haoran You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:37:56.023Z
paper_authors._id: 655fc82eb3c95b72253264e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655fc82eb3c95b72253264e1/EzimumET_Io5lo7e_xW_F.jpeg
paper_authors.fullname: Haoran You
paper_authors.type: user
paper_authors.user: hryou1998
paper_authors._id: 6667cef90c21d689e1936d2b
paper_authors.name: Yipin Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:38:02.554Z
paper_authors._id: 654329932cfe8660a394815e
paper_authors.avatarUrl: /avatars/fc47c383509b4a354a1f93e109ccb025.svg
paper_authors.fullname: Yipin Guo
paper_authors.type: user
paper_authors.user: kibnapper
paper_authors._id: 6667cef90c21d689e1936d2c
paper_authors.name: Yichao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:14:42.867Z
paper_authors._id: 62ba66296501b0ff15ba1075
paper_authors.avatarUrl: /avatars/de468727cb1240fd4b1f24a19fb237a6.svg
paper_authors.fullname: Yichao Fu
paper_authors.type: user
paper_authors.user: Viol2000
paper_authors._id: 6667cef90c21d689e1936d2d
paper_authors.name: Wei Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T08:01:19.742Z
paper_authors._id: 652dd0f4577d1e51cf57d5fe
paper_authors.avatarUrl: /avatars/2ab6cd122e7c137cd6d10f35269341ac.svg
paper_authors.fullname: Wei Zhou
paper_authors.type: user
paper_authors.user: weizhou03
paper_authors._id: 6667cef90c21d689e1936d2e
paper_authors.name: Huihong Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:38:25.937Z
paper_authors._id: 65ce9be34e5f19019db063b7
paper_authors.avatarUrl: /avatars/9e8b7ed2486a7c72e5b65a92017c75e5.svg
paper_authors.fullname: Huihong Shi
paper_authors.type: user
paper_authors.user: shihuihong
paper_authors._id: 6667cef90c21d689e1936d2f
paper_authors.name: Xiaofan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T16:05:25.779Z
paper_authors._id: 666870bfa2f41408ca503df8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666870bfa2f41408ca503df8/-4jeAGmoepwcezE0L-LvN.jpeg
paper_authors.fullname: Xiaofan Zhang
paper_authors.type: user
paper_authors.user: xiaofanzhang
paper_authors._id: 6667cef90c21d689e1936d30
paper_authors.name: Souvik Kundu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-11T04:13:46.788Z
paper_authors._id: 654026a57b2743b58fa1dc65
paper_authors.avatarUrl: /avatars/8d931fd1e582d35b3edc53216b2d0b75.svg
paper_authors.fullname: Souvik Kundu
paper_authors.type: user
paper_authors.user: souvikku
paper_authors._id: 6667cef90c21d689e1936d31
paper_authors.name: Amir Yazdanbakhsh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:38:46.873Z
paper_authors._id: 6352dc90a023833e59d6349a
paper_authors.avatarUrl: /avatars/76e5e7b51515b58e207c7d2a4b360d30.svg
paper_authors.fullname: Amir Yazdanbakhsh
paper_authors.type: user
paper_authors.user: ayazdan
paper_authors._id: 6667cef90c21d689e1936d32
paper_authors.name: Yingyan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T13:26:39.432Z
paper_authors._id: 659036962a0a886ef0246095
paper_authors.avatarUrl: /avatars/4262b790058296a3f13603fbcd3304af.svg
paper_authors.fullname: celine lin
paper_authors.type: user
paper_authors.user: violin2023
paper_title: ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training
    Multiplication-Less Reparameterization
paper_summary: Large language models (LLMs) have shown impressive performance on language
  tasks but face challenges when deployed on resource-constrained devices due to
  their extensive parameters and reliance on dense multiplications, resulting in
  high memory demands and latency bottlenecks. Shift-and-add reparameterization
  offers a promising solution by replacing costly multiplications with
  hardware-friendly primitives in both the attention and multi-layer perceptron
  (MLP) layers of an LLM. However, current reparameterization techniques require
  training from scratch or full parameter fine-tuning to restore accuracy, which
  is resource-intensive for LLMs. To address this, we propose accelerating
  pretrained LLMs through post-training shift-and-add reparameterization,
  creating efficient multiplication-free models, dubbed ShiftAddLLM.
  Specifically, we quantize each weight matrix into binary matrices paired with
  group-wise scaling factors. The associated multiplications are reparameterized
  into (1) shifts between activations and scaling factors and (2) queries and
  adds according to the binary matrices. To reduce accuracy loss, we present a
  multi-objective optimization method to minimize both weight and output
  activation reparameterization errors. Additionally, based on varying
  sensitivity across layers to reparameterization, we develop an automated bit
  allocation strategy to further reduce memory usage and latency. Experiments on
  five LLM families and eight tasks consistently validate the effectiveness of
  ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points
  at comparable or lower latency compared to the most competitive quantized LLMs
  at 3 and 2 bits, respectively, and more than 80% memory and energy reductions
  over the original LLMs. Codes and models are available at
  https://github.com/GATECH-EIC/ShiftAddLLM.
paper_ai_summary: Shift-and-add reparameterization of large language models reduces memory and latency without significant accuracy loss by converting weight matrices into binary matrices with group-wise scaling factors.
paper_ai_keywords: large language models LLMs shift-and-add reparameterization hardware-friendly primitives attention layers multi-layer perceptron MLP layers quantization weight matrices binary matrices group-wise scaling factors multi-objective optimization bit allocation strategy perplexity improvements