Improved Training Technique for Latent Consistency Models
Consistency models are a new family of generative models capable of producing
high-quality samples in either a single step or multiple steps. Recently,
consistency models have demonstrated impressive performance, achieving results
on par with diffusion models in the pixel space. However, the success of
scaling consistency training to large-scale datasets, particularly for
text-to-image and video generation tasks, is determined by performance in the
latent space. In this work, we analyze the statistical differences between
pixel and latent spaces, discovering that latent data often contains highly
impulsive outliers, which significantly degrade the performance of iCT in the
latent space. To address this, we replace Pseudo-Huber losses with Cauchy
losses, effectively mitigating the impact of outliers. Additionally, we
introduce a diffusion loss at early timesteps and employ optimal transport (OT)
coupling to further enhance performance. Lastly, we introduce the adaptive
scaling-c scheduler to manage the robust training process and adopt
Non-scaling LayerNorm in the architecture to better capture the statistics of
the features and reduce outlier impact. With these strategies, we successfully
train latent consistency models capable of high-quality sampling with one or
two steps, significantly narrowing the performance gap between latent
consistency and diffusion models. The implementation is released here:
https://github.com/quandao10/sLCT/
67a189e8fbbab3ce03462fb3
Quan Dao
claimed_verified
2025-02-04T09:39:30.529Z
63e083e6f351dc0745745d17
https://cdn-avatars.huggingface.co/v1/production/uploads/63e083e6f351dc0745745d17/N0GE4uLrkm14blAQMnm2E.jpeg
Quan Dao
user
quandao10
67a189e8fbbab3ce03462fb4
Khanh Doan
67a189e8fbbab3ce03462fb5
Di Liu
67a189e8fbbab3ce03462fb6
Trung Le
extracted_pending
2025-02-04T03:30:50.175Z
66db7db231e772c5ec4c5576
/avatars/aa0eb054bd6c881054431a22daf1aea1.svg
Trung Le
user
trungleuc
67a189e8fbbab3ce03462fb7
Dimitris Metaxas
Improved Training Technique for Latent Consistency Models
Consistency models are a new family of generative models capable of producing
high-quality samples in either a single step or multiple steps. Recently,
consistency models have demonstrated impressive performance, achieving results
on par with diffusion models in the pixel space. However, the success of
scaling consistency training to large-scale datasets, particularly for
text-to-image and video generation tasks, is determined by performance in the
latent space. In this work, we analyze the statistical differences between
pixel and latent spaces, discovering that latent data often contains highly
impulsive outliers, which significantly degrade the performance of iCT in the
latent space. To address this, we replace Pseudo-Huber losses with Cauchy
losses, effectively mitigating the impact of outliers. Additionally, we
introduce a diffusion loss at early timesteps and employ optimal transport (OT)
coupling to further enhance performance. Lastly, we introduce the adaptive
scaling-c scheduler to manage the robust training process and adopt
Non-scaling LayerNorm in the architecture to better capture the statistics of
the features and reduce outlier impact. With these strategies, we successfully
train latent consistency models capable of high-quality sampling with one or
two steps, significantly narrowing the performance gap between latent
consistency and diffusion models. The implementation is released here:
https://github.com/quandao10/sLCT/
Latent consistency models achieve high-quality sampling by addressing outliers with Cauchy losses, diffusion losses, optimal transport, and adaptive scaling-$c$ scheduler, narrowing the gap with diffusion models.
consistency models diffusion models pixel space latent space Pseudo-Huber losses Cauchy losses diffusion loss optimal transport adaptive scaling-$c$ scheduler Non-scaling LayerNorm