Pixel-Space Post-Training of Latent Diffusion Models
Latent diffusion models (LDMs) have made significant advancements in the
field of image generation in recent years. One major advantage of LDMs is their
ability to operate in a compressed latent space, allowing for more efficient
training and deployment. However, despite these advantages, challenges with
LDMs still remain. For example, it has been observed that LDMs often generate
high-frequency details and complex compositions imperfectly. We hypothesize
that one reason for these flaws is due to the fact that all pre- and
post-training of LDMs are done in latent space, which is typically 8 times 8
lower spatial-resolution than the output images. To address this issue, we
propose adding pixel-space supervision in the post-training process to better
preserve high-frequency details. Experimentally, we show that adding a
pixel-space objective significantly improves both supervised quality
fine-tuning and preference-based post-training by a large margin on a
state-of-the-art DiT transformer and U-Net diffusion models in both visual
quality and visual flaw metrics, while maintaining the same text alignment
quality.
66f63450c6cef3d86eec706d
Christina Zhang
extracted_confirmed
2025-03-15T18:06:57.589Z
67d5c0b3179ad2756611daae
/avatars/0902468598b8042a1269d1381c5ba12c.svg
Christina Zhang
user
christinaz21
66f63450c6cef3d86eec706e
Simran Motwani
admin_assigned
2024-10-01T13:38:13.498Z
6515bb7e99e8060f1e792047
/avatars/9c6400f83c3165de3154b0e6bee2e5a1.svg
Simran Motwani
user
smotwani
66f63450c6cef3d86eec706f
Matthew Yu
66f63450c6cef3d86eec7070
Ji Hou
claimed_verified
2025-04-13T19:28:21.228Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
66f63450c6cef3d86eec7071
Felix Juefei-Xu
claimed_verified
2024-12-16T16:55:40.291Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
user
katanaxu
66f63450c6cef3d86eec7072
Sam Tsai
claimed_verified
2024-10-18T15:31:33.926Z
67127b753a058517519fabe8
/avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
Sam Tsai
user
sstsai-adl
66f63450c6cef3d86eec7073
Peter Vajda
66f63450c6cef3d86eec7074
Zijian He
66f63450c6cef3d86eec7075
Jialiang Wang
admin_assigned
2024-10-01T13:38:30.788Z
65300c8a5da070af779cf047
/avatars/8e24a2a1d7a8ced05a4c8e25b3d1be5a.svg
Jialiang Wang
user
jwhuggingf
Pixel-Space Post-Training of Latent Diffusion Models
Latent diffusion models (LDMs) have made significant advancements in the
field of image generation in recent years. One major advantage of LDMs is their
ability to operate in a compressed latent space, allowing for more efficient
training and deployment. However, despite these advantages, challenges with
LDMs still remain. For example, it has been observed that LDMs often generate
high-frequency details and complex compositions imperfectly. We hypothesize
that one reason for these flaws is due to the fact that all pre- and
post-training of LDMs are done in latent space, which is typically 8 times 8
lower spatial-resolution than the output images. To address this issue, we
propose adding pixel-space supervision in the post-training process to better
preserve high-frequency details. Experimentally, we show that adding a
pixel-space objective significantly improves both supervised quality
fine-tuning and preference-based post-training by a large margin on a
state-of-the-art DiT transformer and U-Net diffusion models in both visual
quality and visual flaw metrics, while maintaining the same text alignment
quality.
Adding pixel-space supervision to latent diffusion models improves high-frequency detail preservation during post-training without compromising text alignment quality.
latent diffusion models pixel-space supervision high-frequency details latent space DiT transformer U-Net diffusion models visual quality visual flaw metrics text alignment quality