title: LLM-FP4: 4-Bit Floating-Point Quantized Transformers
summary: We propose LLM-FP4 for quantizing both weights and activations in large
  language models (LLMs) down to 4-bit floating-point values, in a post-training
  manner. Existing post-training quantization (PTQ) solutions are primarily
  integer-based and struggle with bit widths below 8 bits. Compared to integer
  quantization, floating-point (FP) quantization is more flexible and can better
  handle long-tail or bell-shaped distributions, and it has emerged as a default
  choice in many hardware platforms. One characteristic of FP quantization is
  that its performance largely depends on the choice of exponent bits and
  clipping range. In this regard, we construct a strong FP-PTQ baseline by
  searching for the optimal quantization parameters. Furthermore, we observe a
  high inter-channel variance and low intra-channel variance pattern in
  activation distributions, which adds activation quantization difficulty. We
  recognize this pattern to be consistent across a spectrum of transformer models
  designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models.
  To tackle this, we propose per-channel activation quantization and show that
  these additional scaling factors can be reparameterized as exponential biases
  of weights, incurring a negligible cost. Our method, for the first time, can
  quantize both weights and activations in the LLaMA-13B to only 4-bit and
  achieves an average score of 63.1 on the common sense zero-shot reasoning
  tasks, which is only 5.8 lower than the full-precision model, significantly
  outperforming the previous state-of-the-art by 12.7 points. Code is available
  at: https://github.com/nbasyl/LLM-FP4.
paper_authors._id: 6539ddfefc8dea94ef71abad
paper_authors.name: Shih-yang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T10:05:18.816Z
paper_authors._id: 62b58c68a1bae3c711c41321
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b58c68a1bae3c711c41321/FGQ1ifsPpmRi8P0ElxV5J.png
paper_authors.fullname: LIU Shih-yang
paper_authors.type: user
paper_authors.user: sliuau
paper_authors._id: 6539ddfefc8dea94ef71abae
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:26:43.650Z
paper_authors._id: 620ae55fe190de4b8f27ba9d
paper_authors.avatarUrl: /avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zechun
paper_authors._id: 6539ddfefc8dea94ef71abaf
paper_authors.name: Xijie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:26:54.393Z
paper_authors._id: 6411a4f39457280bcdd29cf7
paper_authors.avatarUrl: /avatars/b0c85362be1edbfc40035e7e841fcbeb.svg
paper_authors.fullname: Xijie Huang
paper_authors.type: user
paper_authors.user: ScarletAce
paper_authors._id: 6539ddfefc8dea94ef71abb0
paper_authors.name: Pingcheng Dong
paper_authors._id: 6539ddfefc8dea94ef71abb1
paper_authors.name: Kwang-Ting Cheng
paper_title: LLM-FP4: 4-Bit Floating-Point Quantized Transformers
paper_summary: We propose LLM-FP4 for quantizing both weights and activations in large
  language models (LLMs) down to 4-bit floating-point values, in a post-training
  manner. Existing post-training quantization (PTQ) solutions are primarily
  integer-based and struggle with bit widths below 8 bits. Compared to integer
  quantization, floating-point (FP) quantization is more flexible and can better
  handle long-tail or bell-shaped distributions, and it has emerged as a default
  choice in many hardware platforms. One characteristic of FP quantization is
  that its performance largely depends on the choice of exponent bits and
  clipping range. In this regard, we construct a strong FP-PTQ baseline by
  searching for the optimal quantization parameters. Furthermore, we observe a
  high inter-channel variance and low intra-channel variance pattern in
  activation distributions, which adds activation quantization difficulty. We
  recognize this pattern to be consistent across a spectrum of transformer models
  designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models.
  To tackle this, we propose per-channel activation quantization and show that
  these additional scaling factors can be reparameterized as exponential biases
  of weights, incurring a negligible cost. Our method, for the first time, can
  quantize both weights and activations in the LLaMA-13B to only 4-bit and
  achieves an average score of 63.1 on the common sense zero-shot reasoning
  tasks, which is only 5.8 lower than the full-precision model, significantly
  outperforming the previous state-of-the-art by 12.7 points. Code is available
  at: https://github.com/nbasyl/LLM-FP4.
paper_ai_keywords: LLM-FP4 quantizing both weights and activations large language models (LLMs) 4-bit floating-point values post-training quantization (PTQ) integer-based floating-point (FP) quantization bit widths below 8 bits exponent bits and clipping range per-channel activation quantization exponential biases inter-channel variance intra-channel variance transformer models LLa-13B common zero-shot reasoning tasks