MobileQuant: Mobile-friendly Quantization for On-device Language Models
Large language models (LLMs) have revolutionized language processing,
delivering outstanding results across multiple applications. However, deploying
LLMs on edge devices poses several challenges with respect to memory, energy,
and compute costs, limiting their widespread use in devices such as mobile
phones. A promising solution is to reduce the number of bits used to represent
weights and activations. While existing works have found partial success at
quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations
beyond 16 bits often leads to large computational overheads due to poor
on-device quantization support, or a considerable accuracy drop. Yet, 8-bit
activations are very attractive for on-device deployment as they would enable
LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units
(NPUs). In this work, we make a first attempt to facilitate the on-device
deployment of LLMs using integer-only quantization. We first investigate the
limitations of existing quantization methods for on-device deployment, with a
special focus on activation quantization. We then address these limitations by
introducing a simple post-training quantization method, named MobileQuant, that
extends previous weight equivalent transformation works by jointly optimizing
the weight transformation and activation range parameters in an end-to-end
manner. MobileQuant demonstrates superior capabilities over existing methods by
1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2)
reducing latency and energy consumption by 20\%-50\% compared to current
on-device quantization strategies, 3) requiring limited compute budget, 4)
being compatible with mobile-friendly compute units, e.g. NPU.
66cd48dffc3640405cf4db90
Fuwen Tan
66cd48dffc3640405cf4db91
Royson Lee
admin_assigned
2024-08-27T13:14:19.341Z
64ee0073913dc3d9e34dc83c
/avatars/9c8aaaa5d379de504870a1dc20a57cf3.svg
Royson Lee
user
royson
66cd48dffc3640405cf4db92
Łukasz Dudziak
admin_assigned
2024-08-27T13:14:29.595Z
66a40082f496b42dc094d213
https://cdn-avatars.huggingface.co/v1/production/uploads/66a40082f496b42dc094d213/uauXIpQ-fUvkE9-QW50X-.png
Łukasz Dudziak
user
lukaszd
66cd48dffc3640405cf4db93
Shell Xu Hu
admin_assigned
2024-08-27T13:14:35.286Z
625d3b3de1176e6c77268df6
/avatars/73deace40ca2879c3bae1897979fc63e.svg
Shell Xu Hu
user
hushell
66cd48dffc3640405cf4db94
Sourav Bhattacharya
66cd48dffc3640405cf4db95
Timothy Hospedales
66cd48dffc3640405cf4db96
Georgios Tzimiropoulos
admin_assigned
2024-08-27T13:15:04.647Z
649db0dcd8f9b2e2cb47e4d7
/avatars/e1fb64d5302271a3557ba306a85f2363.svg
Georgios Tzimiropoulos
user
tzimiro
66cd48dffc3640405cf4db97
Brais Martinez
MobileQuant: Mobile-friendly Quantization for On-device Language Models
Large language models (LLMs) have revolutionized language processing,
delivering outstanding results across multiple applications. However, deploying
LLMs on edge devices poses several challenges with respect to memory, energy,
and compute costs, limiting their widespread use in devices such as mobile
phones. A promising solution is to reduce the number of bits used to represent
weights and activations. While existing works have found partial success at
quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations
beyond 16 bits often leads to large computational overheads due to poor
on-device quantization support, or a considerable accuracy drop. Yet, 8-bit
activations are very attractive for on-device deployment as they would enable
LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units
(NPUs). In this work, we make a first attempt to facilitate the on-device
deployment of LLMs using integer-only quantization. We first investigate the
limitations of existing quantization methods for on-device deployment, with a
special focus on activation quantization. We then address these limitations by
introducing a simple post-training quantization method, named MobileQuant, that
extends previous weight equivalent transformation works by jointly optimizing
the weight transformation and activation range parameters in an end-to-end
manner. MobileQuant demonstrates superior capabilities over existing methods by
1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2)
reducing latency and energy consumption by 20\%-50\% compared to current
on-device quantization strategies, 3) requiring limited compute budget, 4)
being compatible with mobile-friendly compute units, e.g. NPU.
MobileQuant facilitates the on-device deployment of large language models using integer-only quantization, reducing latency and energy consumption while maintaining high accuracy.
large language models quantization bitwidths weights activations neural processing units NPUs post-training quantization weight equivalent transformation on-device deployment