title: MobileQuant: Mobile-friendly Quantization for On-device Language Models
summary: Large language models (LLMs) have revolutionized language processing,
  delivering outstanding results across multiple applications. However, deploying
  LLMs on edge devices poses several challenges with respect to memory, energy,
  and compute costs, limiting their widespread use in devices such as mobile
  phones. A promising solution is to reduce the number of bits used to represent
  weights and activations. While existing works have found partial success at
  quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations
  beyond 16 bits often leads to large computational overheads due to poor
  on-device quantization support, or a considerable accuracy drop. Yet, 8-bit
  activations are very attractive for on-device deployment as they would enable
  LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units
  (NPUs). In this work, we make a first attempt to facilitate the on-device
  deployment of LLMs using integer-only quantization. We first investigate the
  limitations of existing quantization methods for on-device deployment, with a
  special focus on activation quantization. We then address these limitations by
  introducing a simple post-training quantization method, named MobileQuant, that
  extends previous weight equivalent transformation works by jointly optimizing
  the weight transformation and activation range parameters in an end-to-end
  manner. MobileQuant demonstrates superior capabilities over existing methods by
  1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2)
  reducing latency and energy consumption by 20\%-50\% compared to current
  on-device quantization strategies, 3) requiring limited compute budget, 4)
  being compatible with mobile-friendly compute units, e.g. NPU.
paper_authors._id: 66cd48dffc3640405cf4db90
paper_authors.name: Fuwen Tan
paper_authors._id: 66cd48dffc3640405cf4db91
paper_authors.name: Royson Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:14:19.341Z
paper_authors._id: 64ee0073913dc3d9e34dc83c
paper_authors.avatarUrl: /avatars/9c8aaaa5d379de504870a1dc20a57cf3.svg
paper_authors.fullname: Royson Lee
paper_authors.type: user
paper_authors.user: royson
paper_authors._id: 66cd48dffc3640405cf4db92
paper_authors.name: Łukasz Dudziak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:14:29.595Z
paper_authors._id: 66a40082f496b42dc094d213
paper_authors.avatarUrl: /avatars/6eab27ca0a3e4b30149153424bdd2bf6.svg
paper_authors.fullname: Łukasz Dudziak
paper_authors.type: user
paper_authors.user: lukaszd
paper_authors._id: 66cd48dffc3640405cf4db93
paper_authors.name: Shell Xu Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:14:35.286Z
paper_authors._id: 625d3b3de1176e6c77268df6
paper_authors.avatarUrl: /avatars/73deace40ca2879c3bae1897979fc63e.svg
paper_authors.fullname: Shell Xu Hu
paper_authors.type: user
paper_authors.user: hushell
paper_authors._id: 66cd48dffc3640405cf4db94
paper_authors.name: Sourav Bhattacharya
paper_authors._id: 66cd48dffc3640405cf4db95
paper_authors.name: Timothy Hospedales
paper_authors._id: 66cd48dffc3640405cf4db96
paper_authors.name: Georgios Tzimiropoulos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T13:15:04.647Z
paper_authors._id: 649db0dcd8f9b2e2cb47e4d7
paper_authors.avatarUrl: /avatars/e1fb64d5302271a3557ba306a85f2363.svg
paper_authors.fullname: Georgios Tzimiropoulos
paper_authors.type: user
paper_authors.user: tzimiro
paper_authors._id: 66cd48dffc3640405cf4db97
paper_authors.name: Brais Martinez
paper_title: MobileQuant: Mobile-friendly Quantization for On-device Language Models
paper_summary: Large language models (LLMs) have revolutionized language processing,
  delivering outstanding results across multiple applications. However, deploying
  LLMs on edge devices poses several challenges with respect to memory, energy,
  and compute costs, limiting their widespread use in devices such as mobile
  phones. A promising solution is to reduce the number of bits used to represent
  weights and activations. While existing works have found partial success at
  quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations
  beyond 16 bits often leads to large computational overheads due to poor
  on-device quantization support, or a considerable accuracy drop. Yet, 8-bit
  activations are very attractive for on-device deployment as they would enable
  LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units
  (NPUs). In this work, we make a first attempt to facilitate the on-device
  deployment of LLMs using integer-only quantization. We first investigate the
  limitations of existing quantization methods for on-device deployment, with a
  special focus on activation quantization. We then address these limitations by
  introducing a simple post-training quantization method, named MobileQuant, that
  extends previous weight equivalent transformation works by jointly optimizing
  the weight transformation and activation range parameters in an end-to-end
  manner. MobileQuant demonstrates superior capabilities over existing methods by
  1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2)
  reducing latency and energy consumption by 20\%-50\% compared to current
  on-device quantization strategies, 3) requiring limited compute budget, 4)
  being compatible with mobile-friendly compute units, e.g. NPU.
paper_ai_keywords: Large language models (LLMs) memory energy compute costs quantizing weights activations computational overheads integer-only quantization MobileQuant weight equivalent transformation activation range parameters end-to-end near-lossless quantization latency energy consumption limited compute budget mobile-friendly compute units NPU