title: Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
summary: Recently, diffusion models have made remarkable progress in text-to-image
  (T2I) generation, synthesizing images with high fidelity and diverse contents.
  Despite this advancement, latent space smoothness within diffusion models
  remains largely unexplored. Smooth latent spaces ensure that a perturbation on
  an input latent corresponds to a steady change in the output image. This
  property proves beneficial in downstream tasks, including image interpolation,
  inversion, and editing. In this work, we expose the non-smoothness of diffusion
  latent spaces by observing noticeable visual fluctuations resulting from minor
  latent variations. To tackle this issue, we propose Smooth Diffusion, a new
  category of diffusion models that can be simultaneously high-performing and
  smooth. Specifically, we introduce Step-wise Variation Regularization to
  enforce the proportion between the variations of an arbitrary input latent and
  that of the output image is a constant at any diffusion training step. In
  addition, we devise an interpolation standard deviation (ISTD) metric to
  effectively assess the latent space smoothness of a diffusion model. Extensive
  quantitative and qualitative experiments demonstrate that Smooth Diffusion
  stands out as a more desirable solution not only in T2I generation but also
  across various downstream tasks. Smooth Diffusion is implemented as a
  plug-and-play Smooth-LoRA to work with various community models. Code is
  available at https://github.com/SHI-Labs/Smooth-Diffusion.
paper_authors._id: 6572949959773fd21b605fc2
paper_authors.name: Jiayi Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:01.016Z
paper_authors._id: 6417c111dce1e4c0229ec182
paper_authors.avatarUrl: /avatars/de65da1f02b136b1af57862d9d0c48f2.svg
paper_authors.fullname: Jiayi Guo
paper_authors.type: user
paper_authors.user: JiayiGuo821
paper_authors._id: 6572949959773fd21b605fc3
paper_authors.name: Xingqian Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:07.349Z
paper_authors._id: 6341a3ebc5565a4b8d45db7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668583035141-6341a3ebc5565a4b8d45db7f.jpeg
paper_authors.fullname: Xingqian Xu
paper_authors.type: user
paper_authors.user: JamesXu
paper_authors._id: 6572949959773fd21b605fc4
paper_authors.name: Yifan Pu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:13.836Z
paper_authors._id: 647e83257f9ad5e44babe82a
paper_authors.avatarUrl: /avatars/2d9593775c49856fe5dfa5bd23dfcda7.svg
paper_authors.fullname: yifan pu
paper_authors.type: user
paper_authors.user: yifanpu001
paper_authors._id: 6572949959773fd21b605fc5
paper_authors.name: Zanlin Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:19.983Z
paper_authors._id: 63987ffb2ceb55aabe0852f3
paper_authors.avatarUrl: /avatars/343b796ff6b8906203904e8c620d7eb5.svg
paper_authors.fullname: Zanlin Ni
paper_authors.type: user
paper_authors.user: nzl-thu
paper_authors._id: 6572949959773fd21b605fc6
paper_authors.name: Chaofei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:26.667Z
paper_authors._id: 651c3def7d3cfa8bcd768339
paper_authors.avatarUrl: /avatars/a45cb0f7acf35cf40fe18e6dfc1db112.svg
paper_authors.fullname: Chaofei Wang
paper_authors.type: user
paper_authors.user: wangcf
paper_authors._id: 6572949959773fd21b605fc7
paper_authors.name: Manushree Vasu
paper_authors._id: 6572949959773fd21b605fc8
paper_authors.name: Shiji Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:33:46.576Z
paper_authors._id: 6497a53dbcde24ff442391b0
paper_authors.avatarUrl: /avatars/2e4957f8426223c9233f45414737b7c0.svg
paper_authors.fullname: 宋世杰
paper_authors.type: user
paper_authors.user: songshijie
paper_authors._id: 6572949959773fd21b605fc9
paper_authors.name: Gao Huang
paper_authors._id: 6572949959773fd21b605fca
paper_authors.name: Humphrey Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:34:07.019Z
paper_authors._id: 61e1188afc27c0f5e3641eb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
paper_authors.fullname: Humphrey Shi
paper_authors.type: user
paper_authors.user: Humphrey
paper_title: Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
paper_summary: Recently, diffusion models have made remarkable progress in text-to-image
  (T2I) generation, synthesizing images with high fidelity and diverse contents.
  Despite this advancement, latent space smoothness within diffusion models
  remains largely unexplored. Smooth latent spaces ensure that a perturbation on
  an input latent corresponds to a steady change in the output image. This
  property proves beneficial in downstream tasks, including image interpolation,
  inversion, and editing. In this work, we expose the non-smoothness of diffusion
  latent spaces by observing noticeable visual fluctuations resulting from minor
  latent variations. To tackle this issue, we propose Smooth Diffusion, a new
  category of diffusion models that can be simultaneously high-performing and
  smooth. Specifically, we introduce Step-wise Variation Regularization to
  enforce the proportion between the variations of an arbitrary input latent and
  that of the output image is a constant at any diffusion training step. In
  addition, we devise an interpolation standard deviation (ISTD) metric to
  effectively assess the latent space smoothness of a diffusion model. Extensive
  quantitative and qualitative experiments demonstrate that Smooth Diffusion
  stands out as a more desirable solution not only in T2I generation but also
  across various downstream tasks. Smooth Diffusion is implemented as a
  plug-and-play Smooth-LoRA to work with various community models. Code is
  available at https://github.com/SHI-Labs/Smooth-Diffusion.
paper_ai_summary: Smooth Diffusion addresses non-smooth latent spaces in diffusion models by enforcing consistent variation ratios, improving tasks like text-to-image generation, interpolation, inversion, and editing.
paper_ai_keywords: diffusion models text-to-image generation latent space smoothness Step-wise Variation Regularization interpolation standard deviation (ISTD) Smooth-LoRA