Alias-Free Latent Diffusion Models:Improving Fractional Shift
  Equivariance of Diffusion Latent Space
Latent Diffusion Models (LDMs) are known to have an unstable generation
process, where even small perturbations or shifts in the input noise can lead
to significantly different outputs. This hinders their applicability in
applications requiring consistent results. In this work, we redesign LDMs to
enhance consistency by making them shift-equivariant. While introducing
anti-aliasing operations can partially improve shift-equivariance, significant
aliasing and inconsistency persist due to the unique challenges in LDMs,
including 1) aliasing amplification during VAE training and multiple U-Net
inferences, and 2) self-attention modules that inherently lack
shift-equivariance. To address these issues, we redesign the attention modules
to be shift-equivariant and propose an equivariance loss that effectively
suppresses the frequency bandwidth of the features in the continuous domain.
The resulting alias-free LDM (AF-LDM) achieves strong shift-equivariance and is
also robust to irregular warping. Extensive experiments demonstrate that AF-LDM
produces significantly more consistent results than vanilla LDM across various
applications, including video editing and image-to-image translation. Code is
available at: https://github.com/SingleZombie/AFLDM
67d27032825fbe674d2e4109
Yifan Zhou
claimed_verified
2025-03-13T08:24:00.485Z
64a63f9449b08110f761cd73
/avatars/61860202fc818b105ef24e74dd4f7d3c.svg
Yifan Zhou
user
SingleZombie
67d27032825fbe674d2e410a
Zeqi Xiao
67d27032825fbe674d2e410b
Shuai Yang
67d27032825fbe674d2e410c
Xingang Pan
Alias-Free Latent Diffusion Models:Improving Fractional Shift
  Equivariance of Diffusion Latent Space
Latent Diffusion Models (LDMs) are known to have an unstable generation
process, where even small perturbations or shifts in the input noise can lead
to significantly different outputs. This hinders their applicability in
applications requiring consistent results. In this work, we redesign LDMs to
enhance consistency by making them shift-equivariant. While introducing
anti-aliasing operations can partially improve shift-equivariance, significant
aliasing and inconsistency persist due to the unique challenges in LDMs,
including 1) aliasing amplification during VAE training and multiple U-Net
inferences, and 2) self-attention modules that inherently lack
shift-equivariance. To address these issues, we redesign the attention modules
to be shift-equivariant and propose an equivariance loss that effectively
suppresses the frequency bandwidth of the features in the continuous domain.
The resulting alias-free LDM (AF-LDM) achieves strong shift-equivariance and is
also robust to irregular warping. Extensive experiments demonstrate that AF-LDM
produces significantly more consistent results than vanilla LDM across various
applications, including video editing and image-to-image translation. Code is
available at: https://github.com/SingleZombie/AFLDM
The redesigned LDM, considered alias-free LDM (AF-LDM), enhances consistency and robustness to perturbations by focusing on shift-equivariant attention modules and introducing an equivariance loss, thereby improving performance in video editing and image-to-image translation.
Latent Diffusion Models LDMs shift-equivariant anti-aliasing VAE U-Net self-attention equivariance loss AF-LDM alias-free video editing image-to-image translation