title: On Architectural Compression of Text-to-Image Diffusion Models
summary: Exceptional text-to-image (T2I) generation results of Stable Diffusion models
  (SDMs) come with substantial computational demands. To resolve this issue,
  recent research on efficient SDMs has prioritized reducing the number of
  sampling steps and utilizing network quantization. Orthogonal to these
  directions, this study highlights the power of classical architectural
  compression for general-purpose T2I synthesis by introducing block-removed
  knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
  blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
  parameters, MACs per sampling step, and latency. We conduct distillation-based
  pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
  pairs) on a single A100 GPU. Despite being trained with limited resources, our
  compact models can imitate the original SDM by benefiting from transferred
  knowledge and achieve competitive results against larger multi-billion
  parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
  the applicability of our lightweight pretrained models in personalized
  generation with DreamBooth finetuning.
paper_authors._id: 64701658850a938d6c56653c
paper_authors.name: Bo-Kyeong Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T13:57:46.473Z
paper_authors._id: 64374f10cd93f4c9a34c1239
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64374f10cd93f4c9a34c1239/tdkdcgl18rKXLvyTEhfnG.jpeg
paper_authors.fullname: Bo-Kyeong Kim
paper_authors.type: user
paper_authors.user: bokyeong1015
paper_authors._id: 64701658850a938d6c56653d
paper_authors.name: Hyoung-Kyu Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T08:07:18.582Z
paper_authors._id: 61f44bab7eba274ea80b74ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61f44bab7eba274ea80b74ce/BRbKX1jephdZ7D44FATl4.jpeg
paper_authors.fullname: Hyoung-Kyu Song
paper_authors.type: user
paper_authors.user: deepkyu
paper_authors._id: 64701658850a938d6c56653e
paper_authors.name: Thibault Castells
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T13:57:50.370Z
paper_authors._id: 64709c9390482b0e0f670e80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8hZF_LZQ6ejoS_u_knXFV.png
paper_authors.fullname: Thibault Castells
paper_authors.type: user
paper_authors.user: thibcastells
paper_authors._id: 64701658850a938d6c56653f
paper_authors.name: Shinkook Choi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-29T08:43:19.101Z
paper_authors._id: 642f740f83e266575cf1f61d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642f740f83e266575cf1f61d/230E5P-CR7DFpCchGgnnw.jpeg
paper_authors.fullname: shinkookchoi
paper_authors.type: user
paper_authors.user: skchoi
paper_title: On Architectural Compression of Text-to-Image Diffusion Models
paper_summary: Exceptional text-to-image (T2I) generation results of Stable Diffusion models
  (SDMs) come with substantial computational demands. To resolve this issue,
  recent research on efficient SDMs has prioritized reducing the number of
  sampling steps and utilizing network quantization. Orthogonal to these
  directions, this study highlights the power of classical architectural
  compression for general-purpose T2I synthesis by introducing block-removed
  knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
  blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
  parameters, MACs per sampling step, and latency. We conduct distillation-based
  pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
  pairs) on a single A100 GPU. Despite being trained with limited resources, our
  compact models can imitate the original SDM by benefiting from transferred
  knowledge and achieve competitive results against larger multi-billion
  parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
  the applicability of our lightweight pretrained models in personalized
  generation with DreamBooth finetuning.
paper_ai_keywords: U-Net sampling steps network quantization classical architectural compression Discrimin-removed knowledge-distilled SDMs (BK-SDMs) residual blocks attention blocks parameters MACs (multiply-accumulate operations) latency distillation-based LAION pairs few-shot MS-COCO benchmark lightweight pretrained models personalized pretrained models fine-tuning