ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation
<|reference_start|>ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation: Diffusion transformers (DiTs) have exhibited remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, larger model sizes and multi-frame processing for video generation lead to increased computational and memory costs, posing challenges for practical deployment on edge devices. Post-Training Quantization (PTQ) is an effective method for reducing memory costs and computational complexity. When quantizing diffusion transformers, we find that applying existing diffusion quantization methods designed for U-Net faces challenges in preserving quality. After analyzing the major challenges for quantizing diffusion transformers, we design an improved quantization scheme: "ViDiT-Q": Video and Image Diffusion Transformer Quantization) to address these issues. Furthermore, we identify highly sensitive layers and timesteps hinder quantization for lower bit-widths. To tackle this, we improve ViDiT-Q with a novel metric-decoupled mixed-precision quantization method (ViDiT-Q-MP). We validate the effectiveness of ViDiT-Q across a variety of text-to-image and video models. While baseline quantization methods fail at W8A8 and produce unreadable content at W4A8, ViDiT-Q achieves lossless W8A8 quantization. ViDiTQ-MP achieves W4A8 with negligible visual quality degradation, resulting in a 2.5x memory optimization and a 1.5x latency speedup.<|reference_end|>
@article{zhao2024vidit-q:,
    title={ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers
  for Image and Video Generation},
    author={Tianchen Zhao, Tongcheng Fang, Enshu Liu, Rui Wan, Widyadewi
  Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang,
  Xuefei Ning, Yu Wang},
    journal={arXiv preprint arXiv:2406.02540},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.02540},
    primaryClass={cs.CV}
}