title: Pixel-Space Post-Training of Latent Diffusion Models
summary: Latent diffusion models (LDMs) have made significant advancements in the
  field of image generation in recent years. One major advantage of LDMs is their
  ability to operate in a compressed latent space, allowing for more efficient
  training and deployment. However, despite these advantages, challenges with
  LDMs still remain. For example, it has been observed that LDMs often generate
  high-frequency details and complex compositions imperfectly. We hypothesize
  that one reason for these flaws is due to the fact that all pre- and
  post-training of LDMs are done in latent space, which is typically 8 times 8
  lower spatial-resolution than the output images. To address this issue, we
  propose adding pixel-space supervision in the post-training process to better
  preserve high-frequency details. Experimentally, we show that adding a
  pixel-space objective significantly improves both supervised quality
  fine-tuning and preference-based post-training by a large margin on a
  state-of-the-art DiT transformer and U-Net diffusion models in both visual
  quality and visual flaw metrics, while maintaining the same text alignment
  quality.
paper_authors._id: 66f63450c6cef3d86eec706d
paper_authors.name: Christina Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-15T18:06:57.589Z
paper_authors._id: 67d5c0b3179ad2756611daae
paper_authors.avatarUrl: /avatars/0902468598b8042a1269d1381c5ba12c.svg
paper_authors.fullname: Christina Zhang
paper_authors.type: user
paper_authors.user: christinaz21
paper_authors._id: 66f63450c6cef3d86eec706e
paper_authors.name: Simran Motwani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:38:13.498Z
paper_authors._id: 6515bb7e99e8060f1e792047
paper_authors.avatarUrl: /avatars/9c6400f83c3165de3154b0e6bee2e5a1.svg
paper_authors.fullname: Simran Motwani
paper_authors.type: user
paper_authors.user: smotwani
paper_authors._id: 66f63450c6cef3d86eec706f
paper_authors.name: Matthew Yu
paper_authors._id: 66f63450c6cef3d86eec7070
paper_authors.name: Ji Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-13T19:28:21.228Z
paper_authors._id: 63084636f0dc38fb47c2dc6b
paper_authors.avatarUrl: /avatars/04edff10a048b5d93981e6f5af3ecf40.svg
paper_authors.fullname: Ji Hou
paper_authors.type: user
paper_authors.user: jhou90
paper_authors._id: 66f63450c6cef3d86eec7071
paper_authors.name: Felix Juefei-Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:40.291Z
paper_authors._id: 6417cf37dce1e4c0229f17b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
paper_authors.fullname: Felix Xu
paper_authors.type: user
paper_authors.user: katanaxu
paper_authors._id: 66f63450c6cef3d86eec7072
paper_authors.name: Sam Tsai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T15:31:33.926Z
paper_authors._id: 67127b753a058517519fabe8
paper_authors.avatarUrl: /avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
paper_authors.fullname: Sam Tsai
paper_authors.type: user
paper_authors.user: sstsai-adl
paper_authors._id: 66f63450c6cef3d86eec7073
paper_authors.name: Peter Vajda
paper_authors._id: 66f63450c6cef3d86eec7074
paper_authors.name: Zijian He
paper_authors._id: 66f63450c6cef3d86eec7075
paper_authors.name: Jialiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:38:30.788Z
paper_authors._id: 65300c8a5da070af779cf047
paper_authors.avatarUrl: /avatars/8e24a2a1d7a8ced05a4c8e25b3d1be5a.svg
paper_authors.fullname: Jialiang Wang
paper_authors.type: user
paper_authors.user: jwhuggingf
paper_title: Pixel-Space Post-Training of Latent Diffusion Models
paper_summary: Latent diffusion models (LDMs) have made significant advancements in the
  field of image generation in recent years. One major advantage of LDMs is their
  ability to operate in a compressed latent space, allowing for more efficient
  training and deployment. However, despite these advantages, challenges with
  LDMs still remain. For example, it has been observed that LDMs often generate
  high-frequency details and complex compositions imperfectly. We hypothesize
  that one reason for these flaws is due to the fact that all pre- and
  post-training of LDMs are done in latent space, which is typically 8 times 8
  lower spatial-resolution than the output images. To address this issue, we
  propose adding pixel-space supervision in the post-training process to better
  preserve high-frequency details. Experimentally, we show that adding a
  pixel-space objective significantly improves both supervised quality
  fine-tuning and preference-based post-training by a large margin on a
  state-of-the-art DiT transformer and U-Net diffusion models in both visual
  quality and visual flaw metrics, while maintaining the same text alignment
  quality.
paper_ai_summary: Adding pixel-space supervision to latent diffusion models improves high-frequency detail preservation during post-training without compromising text alignment quality.
paper_ai_keywords: latent diffusion models pixel-space supervision high-frequency details latent space DiT transformer U-Net diffusion models visual quality visual flaw metrics text alignment quality