title: Fast Matrix Multiplications for Lookup Table-Quantized LLMs
summary: The deployment of large language models (LLMs) is often constrained by memory
  bandwidth, where the primary bottleneck is the cost of transferring model
  parameters from the GPU's global memory to its registers. When coupled with
  custom kernels that fuse the dequantization and matmul operations, weight-only
  quantization can thus enable faster inference by reducing the amount of memory
  movement. However, developing high-performance kernels for weight-quantized
  LLMs presents substantial challenges, especially when the weights are
  compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform,
  lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup
  table engine for LUT-quantized LLMs, which uses offline restructuring of the
  quantized weight matrix to minimize bit manipulations associated with
  unpacking, and vectorization and duplication of the lookup table to mitigate
  shared memory bandwidth constraints. At batch sizes < 32 and quantization group
  size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster
  than existing GEMM kernels. As an application of FLUTE, we explore a simple
  extension to lookup table-based NormalFloat quantization and apply it to
  quantize LLaMA3 to various configurations, obtaining competitive quantization
  performance against strong baselines while obtaining an end-to-end throughput
  increase of 1.5 to 2 times.
paper_authors._id: 66996c35990749deca4f7c4f
paper_authors.name: Han Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:38:23.048Z
paper_authors._id: 617c0b780fdb1ffed4b10094
paper_authors.avatarUrl: /avatars/c9e18b435c35194732ee8e45b474a7b3.svg
paper_authors.fullname: Han Guo
paper_authors.type: user
paper_authors.user: hanguo
paper_authors._id: 66996c35990749deca4f7c50
paper_authors.name: William Brandon
paper_authors._id: 66996c35990749deca4f7c51
paper_authors.name: Radostin Cholakov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:22.096Z
paper_authors._id: 629bbadee1af194c64236565
paper_authors.avatarUrl: /avatars/2297243fae6456e031e9c3ff6997e8b2.svg
paper_authors.fullname: Radostin Cholakov
paper_authors.type: user
paper_authors.user: radi-cho
paper_authors._id: 66996c35990749deca4f7c52
paper_authors.name: Jonathan Ragan-Kelley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:37:29.985Z
paper_authors._id: 6304fd879aef62c40140f3fd
paper_authors.avatarUrl: /avatars/98550c142398aeda25cc165f8f5f83d5.svg
paper_authors.fullname: Jonathan Ragan-Kelley
paper_authors.type: user
paper_authors.user: jrk
paper_authors._id: 66996c35990749deca4f7c53
paper_authors.name: Eric P. Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T10:37:39.183Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 66996c35990749deca4f7c54
paper_authors.name: Yoon Kim
paper_title: Fast Matrix Multiplications for Lookup Table-Quantized LLMs
paper_summary: The deployment of large language models (LLMs) is often constrained by memory
  bandwidth, where the primary bottleneck is the cost of transferring model
  parameters from the GPU's global memory to its registers. When coupled with
  custom kernels that fuse the dequantization and matmul operations, weight-only
  quantization can thus enable faster inference by reducing the amount of memory
  movement. However, developing high-performance kernels for weight-quantized
  LLMs presents substantial challenges, especially when the weights are
  compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform,
  lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup
  table engine for LUT-quantized LLMs, which uses offline restructuring of the
  quantized weight matrix to minimize bit manipulations associated with
  unpacking, and vectorization and duplication of the lookup table to mitigate
  shared memory bandwidth constraints. At batch sizes < 32 and quantization group
  size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster
  than existing GEMM kernels. As an application of FLUTE, we explore a simple
  extension to lookup table-based NormalFloat quantization and apply it to
  quantize LLaMA3 to various configurations, obtaining competitive quantization
  performance against strong baselines while obtaining an end-to-end throughput
  increase of 1.5 to 2 times.
paper_ai_keywords: large language models (LLMs) memory bandwidth model parameters GPU's global memory registers custom kernels dequantization matmul operations weight-only quantization inference memory movement high-performance kernels non-evenly-divisible bit widths lookup table (LUT) quantization FLUTE flexible lookup table engine quantized weight matrix bit manipulations unpacking vectorization duplication of the lookup table shared memory bandwidth constraints GEMM kernels lookup table-based Normal Float quantization LLaMA3 quantization performance end-to-end throughput