Designing a Better Asymmetric VQGAN for StableDiffusion
StableDiffusion is a revolutionary text-to-image generator that is causing a
stir in the world of image generation and editing. Unlike traditional methods
that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
model in the latent space via a VQGAN, ensuring both efficiency and quality. It
not only supports image generation tasks, but also enables image editing for
real images, such as image inpainting and local editing. However, we have
observed that the vanilla VQGAN used in StableDiffusion leads to significant
information loss, causing distortion artifacts even in non-edited image
regions. To this end, we propose a new asymmetric VQGAN with two simple
designs. Firstly, in addition to the input from the encoder, the decoder
contains a conditional branch that incorporates information from task-specific
priors, such as the unmasked image region in inpainting. Secondly, the decoder
is much heavier than the encoder, allowing for more detailed recovery while
only slightly increasing the total inference cost. The training cost of our
asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
local editing methods. Extensive experiments demonstrate that it can
significantly improve the inpainting and editing performance, while maintaining
the original text-to-image capability. The code is available at
https://github.com/buxiangzhiren/Asymmetric_VQGAN.
64813ccb5409aa3e3bbf9799
Zixin Zhu
admin_assigned
2023-06-12T13:51:50.013Z
645c8d2ea19f3e64bbecc616
/avatars/eceafc799002b57eaad01e638cec7aff.svg
zhuzixing
user
Zzxxxxx
64813ccb5409aa3e3bbf979a
Xuelu Feng
64813ccb5409aa3e3bbf979b
Dongdong Chen
64813ccb5409aa3e3bbf979c
Jianmin Bao
claimed_verified
2023-06-08T16:07:22.919Z
6481e690f9ed842838a2b106
/avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
Jianmin Bao
user
jianmin-ustc
64813ccb5409aa3e3bbf979d
Le Wang
64813ccb5409aa3e3bbf979e
Yinpeng Chen
64813ccb5409aa3e3bbf979f
Lu Yuan
64813ccb5409aa3e3bbf97a0
Gang Hua
Designing a Better Asymmetric VQGAN for StableDiffusion
StableDiffusion is a revolutionary text-to-image generator that is causing a
stir in the world of image generation and editing. Unlike traditional methods
that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
model in the latent space via a VQGAN, ensuring both efficiency and quality. It
not only supports image generation tasks, but also enables image editing for
real images, such as image inpainting and local editing. However, we have
observed that the vanilla VQGAN used in StableDiffusion leads to significant
information loss, causing distortion artifacts even in non-edited image
regions. To this end, we propose a new asymmetric VQGAN with two simple
designs. Firstly, in addition to the input from the encoder, the decoder
contains a conditional branch that incorporates information from task-specific
priors, such as the unmasked image region in inpainting. Secondly, the decoder
is much heavier than the encoder, allowing for more detailed recovery while
only slightly increasing the total inference cost. The training cost of our
asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
local editing methods. Extensive experiments demonstrate that it can
significantly improve the inpainting and editing performance, while maintaining
the original text-to-image capability. The code is available at
https://github.com/buxiangzhiren/Asymmetric_VQGAN.
An asymmetric VQGAN enhances StableDiffusion by reducing information loss and improving inpainting and editing performance without altering the original model architecture.
StableDiffusion diffusion model latent space VQGAN asymmetric VQGAN conditional branch task-specific priors image inpainting local editing encoder decoder inference cost training cost text-to-image capability