title: Improved Training Technique for Latent Consistency Models
summary: Consistency models are a new family of generative models capable of producing
  high-quality samples in either a single step or multiple steps. Recently,
  consistency models have demonstrated impressive performance, achieving results
  on par with diffusion models in the pixel space. However, the success of
  scaling consistency training to large-scale datasets, particularly for
  text-to-image and video generation tasks, is determined by performance in the
  latent space. In this work, we analyze the statistical differences between
  pixel and latent spaces, discovering that latent data often contains highly
  impulsive outliers, which significantly degrade the performance of iCT in the
  latent space. To address this, we replace Pseudo-Huber losses with Cauchy
  losses, effectively mitigating the impact of outliers. Additionally, we
  introduce a diffusion loss at early timesteps and employ optimal transport (OT)
  coupling to further enhance performance. Lastly, we introduce the adaptive
  scaling-c scheduler to manage the robust training process and adopt
  Non-scaling LayerNorm in the architecture to better capture the statistics of
  the features and reduce outlier impact. With these strategies, we successfully
  train latent consistency models capable of high-quality sampling with one or
  two steps, significantly narrowing the performance gap between latent
  consistency and diffusion models. The implementation is released here:
  https://github.com/quandao10/sLCT/
paper_authors._id: 67a189e8fbbab3ce03462fb3
paper_authors.name: Quan Dao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:30.529Z
paper_authors._id: 63e083e6f351dc0745745d17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e083e6f351dc0745745d17/N0GE4uLrkm14blAQMnm2E.jpeg
paper_authors.fullname: Quan Dao
paper_authors.type: user
paper_authors.user: quandao10
paper_authors._id: 67a189e8fbbab3ce03462fb4
paper_authors.name: Khanh Doan
paper_authors._id: 67a189e8fbbab3ce03462fb5
paper_authors.name: Di Liu
paper_authors._id: 67a189e8fbbab3ce03462fb6
paper_authors.name: Trung Le
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-04T03:30:50.175Z
paper_authors._id: 66db7db231e772c5ec4c5576
paper_authors.avatarUrl: /avatars/aa0eb054bd6c881054431a22daf1aea1.svg
paper_authors.fullname: Trung Le
paper_authors.type: user
paper_authors.user: trungleuc
paper_authors._id: 67a189e8fbbab3ce03462fb7
paper_authors.name: Dimitris Metaxas
paper_title: Improved Training Technique for Latent Consistency Models
paper_summary: Consistency models are a new family of generative models capable of producing
  high-quality samples in either a single step or multiple steps. Recently,
  consistency models have demonstrated impressive performance, achieving results
  on par with diffusion models in the pixel space. However, the success of
  scaling consistency training to large-scale datasets, particularly for
  text-to-image and video generation tasks, is determined by performance in the
  latent space. In this work, we analyze the statistical differences between
  pixel and latent spaces, discovering that latent data often contains highly
  impulsive outliers, which significantly degrade the performance of iCT in the
  latent space. To address this, we replace Pseudo-Huber losses with Cauchy
  losses, effectively mitigating the impact of outliers. Additionally, we
  introduce a diffusion loss at early timesteps and employ optimal transport (OT)
  coupling to further enhance performance. Lastly, we introduce the adaptive
  scaling-c scheduler to manage the robust training process and adopt
  Non-scaling LayerNorm in the architecture to better capture the statistics of
  the features and reduce outlier impact. With these strategies, we successfully
  train latent consistency models capable of high-quality sampling with one or
  two steps, significantly narrowing the performance gap between latent
  consistency and diffusion models. The implementation is released here:
  https://github.com/quandao10/sLCT/
paper_ai_summary: Latent consistency models achieve high-quality sampling by addressing outliers with Cauchy losses, diffusion losses, optimal transport, and adaptive scaling-$c$ scheduler, narrowing the gap with diffusion models.
paper_ai_keywords: consistency models diffusion models pixel space latent space Pseudo-Huber losses Cauchy losses diffusion loss optimal transport adaptive scaling-$c$ scheduler Non-scaling LayerNorm