title: Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image
    Diffusion Models
summary: Text-to-image (T2I) diffusion models have demonstrated impressive image
  generation capabilities. Still, their computational intensity prohibits
  resource-constrained organizations from deploying T2I models after fine-tuning
  them on their internal target data. While pruning techniques offer a potential
  solution to reduce the computational burden of T2I models, static pruning
  methods use the same pruned model for all input prompts, overlooking the
  varying capacity requirements of different prompts. Dynamic pruning addresses
  this issue by utilizing a separate sub-network for each prompt, but it prevents
  batch parallelism on GPUs. To overcome these limitations, we introduce Adaptive
  Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed
  for T2I diffusion models. Central to our approach is a prompt router model,
  which learns to determine the required capacity for an input text prompt and
  routes it to an architecture code, given a total desired compute budget for
  prompts. Each architecture code represents a specialized model tailored to the
  prompts assigned to it, and the number of codes is a hyperparameter. We train
  the prompt router and architecture codes using contrastive learning, ensuring
  that similar prompts are mapped to nearby codes. Further, we employ optimal
  transport to prevent the codes from collapsing into a single one. We
  demonstrate APTP's effectiveness by pruning Stable Diffusion (SD) V2.1 using
  CC3M and COCO as target datasets. APTP outperforms the single-model pruning
  baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters
  learned by APTP reveals they are semantically meaningful. We also show that
  APTP can automatically discover previously empirically found challenging
  prompts for SD, e.g., prompts for generating text images, assigning them to
  higher capacity codes.
paper_authors._id: 66723abd2ff693dab0208a18
paper_authors.name: Alireza Ganjdanesh
paper_authors._id: 66723abd2ff693dab0208a19
paper_authors.name: Reza Shirkavand
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:20:53.785Z
paper_authors._id: 65466e8b4d1931dc93dbf9a6
paper_authors.avatarUrl: /avatars/8626417bac65f431bc50345a024dd570.svg
paper_authors.fullname: Reza Shirkavand
paper_authors.type: user
paper_authors.user: rezashkv
paper_authors._id: 66723abd2ff693dab0208a1a
paper_authors.name: Shangqian Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:34:37.120Z
paper_authors._id: 6673b68e27be15fd5fa84aaf
paper_authors.avatarUrl: /avatars/4584d95c05dd035b7184eec713a751bd.svg
paper_authors.fullname: Shangqian Gao
paper_authors.type: user
paper_authors.user: sgao84
paper_authors._id: 66723abd2ff693dab0208a1b
paper_authors.name: Heng Huang
paper_title: Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image
    Diffusion Models
paper_summary: Text-to-image (T2I) diffusion models have demonstrated impressive image
  generation capabilities. Still, their computational intensity prohibits
  resource-constrained organizations from deploying T2I models after fine-tuning
  them on their internal target data. While pruning techniques offer a potential
  solution to reduce the computational burden of T2I models, static pruning
  methods use the same pruned model for all input prompts, overlooking the
  varying capacity requirements of different prompts. Dynamic pruning addresses
  this issue by utilizing a separate sub-network for each prompt, but it prevents
  batch parallelism on GPUs. To overcome these limitations, we introduce Adaptive
  Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed
  for T2I diffusion models. Central to our approach is a prompt router model,
  which learns to determine the required capacity for an input text prompt and
  routes it to an architecture code, given a total desired compute budget for
  prompts. Each architecture code represents a specialized model tailored to the
  prompts assigned to it, and the number of codes is a hyperparameter. We train
  the prompt router and architecture codes using contrastive learning, ensuring
  that similar prompts are mapped to nearby codes. Further, we employ optimal
  transport to prevent the codes from collapsing into a single one. We
  demonstrate APTP's effectiveness by pruning Stable Diffusion (SD) V2.1 using
  CC3M and COCO as target datasets. APTP outperforms the single-model pruning
  baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters
  learned by APTP reveals they are semantically meaningful. We also show that
  APTP can automatically discover previously empirically found challenging
  prompts for SD, e.g., prompts for generating text images, assigning them to
  higher capacity codes.
paper_ai_summary: Adaptive prompt-tailored pruning optimizes T2I diffusion models by dynamically adjusting the model capacity per text prompt to enhance efficiency and performance.
paper_ai_keywords: diffusion models text-to-image computational intensity resource-constrained fine-tuning pruning techniques static pruning dynamic pruning prompt router model architecture code prompt-based pruning contrastive learning optimal transport Stable Diffusion FID CLIP CMMD scores semantically meaningful clusters