title: SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion
    Models
summary: Diffusion models have been proven highly effective at generating high-quality
  images. However, as these models grow larger, they require significantly more
  memory and suffer from higher latency, posing substantial challenges for
  deployment. In this work, we aim to accelerate diffusion models by quantizing
  their weights and activations to 4 bits. At such an aggressive level, both
  weights and activations are highly sensitive, where conventional post-training
  quantization methods for large language models like smoothing become
  insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit
  quantization paradigm. Different from smoothing which redistributes outliers
  between weights and activations, our approach absorbs these outliers using a
  low-rank branch. We first consolidate the outliers by shifting them from
  activations to weights, then employ a high-precision low-rank branch to take in
  the weight outliers with Singular Value Decomposition (SVD). This process eases
  the quantization on both sides. However, na\"{\i}vely running the low-rank
  branch independently incurs significant overhead due to extra data movement of
  activations, negating the quantization speedup. To address this, we co-design
  an inference engine Nunchaku that fuses the kernels of the low-rank branch into
  those of the low-bit branch to cut off redundant memory access. It can also
  seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for
  re-quantization. Extensive experiments on SDXL, PixArt-Sigma, and FLUX.1
  validate the effectiveness of SVDQuant in preserving image quality. We reduce
  the memory usage for the 12B FLUX.1 models by 3.5times, achieving
  3.0times speedup over the 4-bit weight-only quantized baseline on the 16GB
  laptop 4090 GPU, paving the way for more interactive applications on PCs. Our
  quantization library and inference engine are open-sourced.
paper_authors._id: 672da3b22e3dbc3d3a186cfd
paper_authors.name: Muyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:36.481Z
paper_authors._id: 63129589bbaa385279d1826e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
paper_authors.fullname: Muyang Li
paper_authors.type: user
paper_authors.user: Lmxyy
paper_authors._id: 672da3b22e3dbc3d3a186cfe
paper_authors.name: Yujun Lin
paper_authors._id: 672da3b22e3dbc3d3a186cff
paper_authors.name: Zhekai Zhang
paper_authors._id: 672da3b22e3dbc3d3a186d00
paper_authors.name: Tianle Cai
paper_authors._id: 672da3b22e3dbc3d3a186d01
paper_authors.name: Xiuyu Li
paper_authors._id: 672da3b22e3dbc3d3a186d02
paper_authors.name: Junxian Guo
paper_authors._id: 672da3b22e3dbc3d3a186d03
paper_authors.name: Enze Xie
paper_authors._id: 672da3b22e3dbc3d3a186d04
paper_authors.name: Chenlin Meng
paper_authors._id: 672da3b22e3dbc3d3a186d05
paper_authors.name: Jun-Yan Zhu
paper_authors._id: 672da3b22e3dbc3d3a186d06
paper_authors.name: Song Han
paper_title: SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion
    Models
paper_summary: Diffusion models have been proven highly effective at generating high-quality
  images. However, as these models grow larger, they require significantly more
  memory and suffer from higher latency, posing substantial challenges for
  deployment. In this work, we aim to accelerate diffusion models by quantizing
  their weights and activations to 4 bits. At such an aggressive level, both
  weights and activations are highly sensitive, where conventional post-training
  quantization methods for large language models like smoothing become
  insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit
  quantization paradigm. Different from smoothing which redistributes outliers
  between weights and activations, our approach absorbs these outliers using a
  low-rank branch. We first consolidate the outliers by shifting them from
  activations to weights, then employ a high-precision low-rank branch to take in
  the weight outliers with Singular Value Decomposition (SVD). This process eases
  the quantization on both sides. However, na\"{\i}vely running the low-rank
  branch independently incurs significant overhead due to extra data movement of
  activations, negating the quantization speedup. To address this, we co-design
  an inference engine Nunchaku that fuses the kernels of the low-rank branch into
  those of the low-bit branch to cut off redundant memory access. It can also
  seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for
  re-quantization. Extensive experiments on SDXL, PixArt-Sigma, and FLUX.1
  validate the effectiveness of SVDQuant in preserving image quality. We reduce
  the memory usage for the 12B FLUX.1 models by 3.5times, achieving
  3.0times speedup over the 4-bit weight-only quantized baseline on the 16GB
  laptop 4090 GPU, paving the way for more interactive applications on PCs. Our
  quantization library and inference engine are open-sourced.
paper_ai_summary: By quantizing the weights and activations of diffusion models to 4 bits using the SVDQuant paradigm and integrating a fused inference engine, memory usage and speed are significantly improved without compromising image quality.
paper_ai_keywords: diffusion models quantization weights activations SVDQuant low-rank branch Singular Value Decomposition (SVD) inference engine memory usage speedup SDXL PixArt-$\Sigma$ FLUX.1