title: T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on
    Edge
summary: The deployment of Large Language Models (LLMs) on edge devices is
  increasingly important to enhance on-device intelligence. Weight quantization
  is crucial for reducing the memory footprint of LLMs on devices. However,
  low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low
  precision weights and high precision activations during inference. Existing
  systems, lacking native support for mpGEMM, resort to dequantize weights for
  high precision computation. Such an indirect way can lead to a significant
  inference overhead.
    In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based
  method designed for efficient low-bit LLM (i.e., weight-quantized LLM)
  inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while
  simultaneously eliminating multiplications and reducing additions required.
  Specifically, T-MAC transforms the traditional data-type-centric multiplication
  to bit-wise table lookup, and enables a unified and scalable mpGEMM solution.
    Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on
  low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in
  throughput and 70% reduction in energy consumption compared to llama.cpp. For
  BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s
  with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11
  tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds
  the adult average reading speed. T-MAC with LUT-based computing paradigm, paves
  the way for the practical deployment of low-bit LLMs on resource-constrained
  edge devices without compromising computational efficiency. The system is
  open-sourced at https://github.com/microsoft/T-MAC.
paper_authors._id: 66840ed73069bc14d4248b04
paper_authors.name: Jianyu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:17:15.878Z
paper_authors._id: 64198c31060a651c415e1a65
paper_authors.avatarUrl: /avatars/a4fb4a3c3296e0ee4762e711c00204c4.svg
paper_authors.fullname: Jianyu Wei
paper_authors.type: user
paper_authors.user: kalineid
paper_authors._id: 66840ed73069bc14d4248b05
paper_authors.name: Shijie Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:17:42.440Z
paper_authors._id: 6562e526738013bf305dee79
paper_authors.avatarUrl: /avatars/06513f6cae2a28a4660286831927e968.svg
paper_authors.fullname: Shijie Cao
paper_authors.type: user
paper_authors.user: Shijie4
paper_authors._id: 66840ed73069bc14d4248b06
paper_authors.name: Ting Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:26:19.137Z
paper_authors._id: 662852bf8bfc90408a97acdf
paper_authors.avatarUrl: /avatars/008e6faf51e1be4f0c0d17008dfe2f49.svg
paper_authors.fullname: cao
paper_authors.type: user
paper_authors.user: tingcao
paper_authors._id: 66840ed73069bc14d4248b07
paper_authors.name: Lingxiao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:18:08.552Z
paper_authors._id: 63d9074f89a9683b46d1c73d
paper_authors.avatarUrl: /avatars/6dc070c7618f24b76d9df9882fcd5aed.svg
paper_authors.fullname: Lingxiao Ma
paper_authors.type: user
paper_authors.user: xysmlx
paper_authors._id: 66840ed73069bc14d4248b08
paper_authors.name: Lei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T21:08:50.979Z
paper_authors._id: 646c7b3849a37754bdeefe8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646c7b3849a37754bdeefe8e/tejirZfCHm6EqhXOdnMUq.jpeg
paper_authors.fullname: Lei
paper_authors.type: user
paper_authors.user: LeiWang1999
paper_authors._id: 66840ed73069bc14d4248b09
paper_authors.name: Yanyong Zhang
paper_authors._id: 66840ed73069bc14d4248b0a
paper_authors.name: Mao Yang
paper_title: T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on
    Edge
paper_summary: The deployment of Large Language Models (LLMs) on edge devices is
  increasingly important to enhance on-device intelligence. Weight quantization
  is crucial for reducing the memory footprint of LLMs on devices. However,
  low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low
  precision weights and high precision activations during inference. Existing
  systems, lacking native support for mpGEMM, resort to dequantize weights for
  high precision computation. Such an indirect way can lead to a significant
  inference overhead.
    In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based
  method designed for efficient low-bit LLM (i.e., weight-quantized LLM)
  inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while
  simultaneously eliminating multiplications and reducing additions required.
  Specifically, T-MAC transforms the traditional data-type-centric multiplication
  to bit-wise table lookup, and enables a unified and scalable mpGEMM solution.
    Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on
  low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in
  throughput and 70% reduction in energy consumption compared to llama.cpp. For
  BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s
  with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11
  tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds
  the adult average reading speed. T-MAC with LUT-based computing paradigm, paves
  the way for the practical deployment of low-bit LLMs on resource-constrained
  edge devices without compromising computational efficiency. The system is
  open-sourced at https://github.com/microsoft/T-MAC.
paper_ai_keywords: weight quantization mixed precision matrix multiplications (mpGEMM) lookup table (LUT) T-MAC low-bit LLM bit-wise table-lookup unified and scalable mpGEMM solution data-type-centric multiplication throughput energy consumption token generation throughput