T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on
  Edge
The deployment of Large Language Models (LLMs) on edge devices is
increasingly important to enhance on-device intelligence. Weight quantization
is crucial for reducing the memory footprint of LLMs on devices. However,
low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low
precision weights and high precision activations during inference. Existing
systems, lacking native support for mpGEMM, resort to dequantize weights for
high precision computation. Such an indirect way can lead to a significant
inference overhead.
  In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based
method designed for efficient low-bit LLM (i.e., weight-quantized LLM)
inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while
simultaneously eliminating multiplications and reducing additions required.
Specifically, T-MAC transforms the traditional data-type-centric multiplication
to bit-wise table lookup, and enables a unified and scalable mpGEMM solution.
  Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on
low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in
throughput and 70% reduction in energy consumption compared to llama.cpp. For
BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s
with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11
tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds
the adult average reading speed. T-MAC with LUT-based computing paradigm, paves
the way for the practical deployment of low-bit LLMs on resource-constrained
edge devices without compromising computational efficiency. The system is
open-sourced at https://github.com/microsoft/T-MAC.
66840ed73069bc14d4248b04
Jianyu Wei
admin_assigned
2024-07-03T15:17:15.878Z
64198c31060a651c415e1a65
/avatars/a4fb4a3c3296e0ee4762e711c00204c4.svg
Jianyu Wei
user
kalineid
66840ed73069bc14d4248b05
Shijie Cao
admin_assigned
2024-07-03T15:17:42.440Z
6562e526738013bf305dee79
/avatars/06513f6cae2a28a4660286831927e968.svg
Shijie Cao
user
Shijie4
66840ed73069bc14d4248b06
Ting Cao
claimed_verified
2024-09-26T08:26:19.137Z
662852bf8bfc90408a97acdf
/avatars/008e6faf51e1be4f0c0d17008dfe2f49.svg
cao
user
tingcao
66840ed73069bc14d4248b07
Lingxiao Ma
admin_assigned
2024-07-03T15:18:08.552Z
63d9074f89a9683b46d1c73d
/avatars/6dc070c7618f24b76d9df9882fcd5aed.svg
Lingxiao Ma
user
xysmlx
66840ed73069bc14d4248b08
Lei Wang
claimed_verified
2024-07-02T21:08:50.979Z
646c7b3849a37754bdeefe8e
https://cdn-avatars.huggingface.co/v1/production/uploads/646c7b3849a37754bdeefe8e/tejirZfCHm6EqhXOdnMUq.jpeg
Lei
user
LeiWang1999
66840ed73069bc14d4248b09
Yanyong Zhang
66840ed73069bc14d4248b0a
Mao Yang
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on
  Edge
The deployment of Large Language Models (LLMs) on edge devices is
increasingly important to enhance on-device intelligence. Weight quantization
is crucial for reducing the memory footprint of LLMs on devices. However,
low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low
precision weights and high precision activations during inference. Existing
systems, lacking native support for mpGEMM, resort to dequantize weights for
high precision computation. Such an indirect way can lead to a significant
inference overhead.
  In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based
method designed for efficient low-bit LLM (i.e., weight-quantized LLM)
inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while
simultaneously eliminating multiplications and reducing additions required.
Specifically, T-MAC transforms the traditional data-type-centric multiplication
to bit-wise table lookup, and enables a unified and scalable mpGEMM solution.
  Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on
low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in
throughput and 70% reduction in energy consumption compared to llama.cpp. For
BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s
with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11
tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds
the adult average reading speed. T-MAC with LUT-based computing paradigm, paves
the way for the practical deployment of low-bit LLMs on resource-constrained
edge devices without compromising computational efficiency. The system is
open-sourced at https://github.com/microsoft/T-MAC.
T-MAC is a lookup table-based method for efficient low-bit LLM inference on CPUs, supporting mixed precision matrix multiplication without dequantization and reducing computational overhead.
Large Language Models edge devices weight quantization mixed precision matrix multiplication lookup table inference scalability token generation throughput computational efficiency resource-constrained devices