Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image
  Diffusion Models
Text-to-image (T2I) diffusion models have demonstrated impressive image
generation capabilities. Still, their computational intensity prohibits
resource-constrained organizations from deploying T2I models after fine-tuning
them on their internal target data. While pruning techniques offer a potential
solution to reduce the computational burden of T2I models, static pruning
methods use the same pruned model for all input prompts, overlooking the
varying capacity requirements of different prompts. Dynamic pruning addresses
this issue by utilizing a separate sub-network for each prompt, but it prevents
batch parallelism on GPUs. To overcome these limitations, we introduce Adaptive
Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed
for T2I diffusion models. Central to our approach is a prompt router model,
which learns to determine the required capacity for an input text prompt and
routes it to an architecture code, given a total desired compute budget for
prompts. Each architecture code represents a specialized model tailored to the
prompts assigned to it, and the number of codes is a hyperparameter. We train
the prompt router and architecture codes using contrastive learning, ensuring
that similar prompts are mapped to nearby codes. Further, we employ optimal
transport to prevent the codes from collapsing into a single one. We
demonstrate APTP's effectiveness by pruning Stable Diffusion (SD) V2.1 using
CC3M and COCO as target datasets. APTP outperforms the single-model pruning
baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters
learned by APTP reveals they are semantically meaningful. We also show that
APTP can automatically discover previously empirically found challenging
prompts for SD, e.g., prompts for generating text images, assigning them to
higher capacity codes.
66723abd2ff693dab0208a18
Alireza Ganjdanesh
66723abd2ff693dab0208a19
Reza Shirkavand
claimed_verified
2024-06-19T20:20:53.785Z
65466e8b4d1931dc93dbf9a6
/avatars/8626417bac65f431bc50345a024dd570.svg
Reza Shirkavand
user
rezashkv
66723abd2ff693dab0208a1a
Shangqian Gao
claimed_verified
2024-06-20T07:34:37.120Z
6673b68e27be15fd5fa84aaf
/avatars/4584d95c05dd035b7184eec713a751bd.svg
Shangqian Gao
user
sgao84
66723abd2ff693dab0208a1b
Heng Huang
Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image
  Diffusion Models
Text-to-image (T2I) diffusion models have demonstrated impressive image
generation capabilities. Still, their computational intensity prohibits
resource-constrained organizations from deploying T2I models after fine-tuning
them on their internal target data. While pruning techniques offer a potential
solution to reduce the computational burden of T2I models, static pruning
methods use the same pruned model for all input prompts, overlooking the
varying capacity requirements of different prompts. Dynamic pruning addresses
this issue by utilizing a separate sub-network for each prompt, but it prevents
batch parallelism on GPUs. To overcome these limitations, we introduce Adaptive
Prompt-Tailored Pruning (APTP), a novel prompt-based pruning method designed
for T2I diffusion models. Central to our approach is a prompt router model,
which learns to determine the required capacity for an input text prompt and
routes it to an architecture code, given a total desired compute budget for
prompts. Each architecture code represents a specialized model tailored to the
prompts assigned to it, and the number of codes is a hyperparameter. We train
the prompt router and architecture codes using contrastive learning, ensuring
that similar prompts are mapped to nearby codes. Further, we employ optimal
transport to prevent the codes from collapsing into a single one. We
demonstrate APTP's effectiveness by pruning Stable Diffusion (SD) V2.1 using
CC3M and COCO as target datasets. APTP outperforms the single-model pruning
baselines in terms of FID, CLIP, and CMMD scores. Our analysis of the clusters
learned by APTP reveals they are semantically meaningful. We also show that
APTP can automatically discover previously empirically found challenging
prompts for SD, e.g., prompts for generating text images, assigning them to
higher capacity codes.
Adaptive prompt-tailored pruning optimizes T2I diffusion models by dynamically adjusting the model capacity per text prompt to enhance efficiency and performance.
diffusion models text-to-image computational intensity resource-constrained fine-tuning pruning techniques static pruning dynamic pruning prompt router model architecture code prompt-based pruning contrastive learning optimal transport Stable Diffusion FID CLIP CMMD scores semantically meaningful clusters