Distilling Diversity and Control in Diffusion Models
Distilled diffusion models suffer from a critical limitation: reduced sample
diversity compared to their base counterparts. In this work, we uncover that
despite this diversity loss, distilled models retain the fundamental concept
representations of base models. We demonstrate control distillation - where
control mechanisms like Concept Sliders and LoRAs trained on base models can be
seamlessly transferred to distilled models and vice-versa, effectively
distilling control without any retraining. This preservation of
representational structure prompted our investigation into the mechanisms of
diversity collapse during distillation. To understand how distillation affects
diversity, we introduce Diffusion Target (DT) Visualization, an analysis and
debugging tool that reveals how models predict final outputs at intermediate
steps. Through DT-Visualization, we identify generation artifacts,
inconsistencies, and demonstrate that initial diffusion timesteps
disproportionately determine output diversity, while later steps primarily
refine details. Based on these insights, we introduce diversity distillation -
a hybrid inference approach that strategically employs the base model for only
the first critical timestep before transitioning to the efficient distilled
model. Our experiments demonstrate that this simple modification not only
restores the diversity capabilities from base to distilled models but
surprisingly exceeds it, while maintaining nearly the computational efficiency
of distilled inference, all without requiring additional training or model
modifications. Our code and data are available at
https://distillation.baulab.info
67d3a2a6977358f62157977d
Rohit Gandikota
claimed_verified
2025-03-14T08:56:16.089Z
636daf1b56c0762cfda074b5
/avatars/f44be5eb110acfa2efbd09de6b416239.svg
Rohit Gandikota
user
RohitGandikota
67d3a2a6977358f62157977e
David Bau
Distilling Diversity and Control in Diffusion Models
Distilled diffusion models suffer from a critical limitation: reduced sample
diversity compared to their base counterparts. In this work, we uncover that
despite this diversity loss, distilled models retain the fundamental concept
representations of base models. We demonstrate control distillation - where
control mechanisms like Concept Sliders and LoRAs trained on base models can be
seamlessly transferred to distilled models and vice-versa, effectively
distilling control without any retraining. This preservation of
representational structure prompted our investigation into the mechanisms of
diversity collapse during distillation. To understand how distillation affects
diversity, we introduce Diffusion Target (DT) Visualization, an analysis and
debugging tool that reveals how models predict final outputs at intermediate
steps. Through DT-Visualization, we identify generation artifacts,
inconsistencies, and demonstrate that initial diffusion timesteps
disproportionately determine output diversity, while later steps primarily
refine details. Based on these insights, we introduce diversity distillation -
a hybrid inference approach that strategically employs the base model for only
the first critical timestep before transitioning to the efficient distilled
model. Our experiments demonstrate that this simple modification not only
restores the diversity capabilities from base to distilled models but
surprisingly exceeds it, while maintaining nearly the computational efficiency
of distilled inference, all without requiring additional training or model
modifications. Our code and data are available at
https://distillation.baulab.info
Diversity distillation using a hybrid approach restores and enhances the diversity of distilled diffusion models without additional training or modifications.
distilled diffusion models sample diversity concept representations control distillation concept sliders LoRAs diffusion target visualization generation artifacts diversity collapse diversity distillation computational efficiency