title: Designing a Better Asymmetric VQGAN for StableDiffusion
summary: StableDiffusion is a revolutionary text-to-image generator that is causing a
  stir in the world of image generation and editing. Unlike traditional methods
  that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
  model in the latent space via a VQGAN, ensuring both efficiency and quality. It
  not only supports image generation tasks, but also enables image editing for
  real images, such as image inpainting and local editing. However, we have
  observed that the vanilla VQGAN used in StableDiffusion leads to significant
  information loss, causing distortion artifacts even in non-edited image
  regions. To this end, we propose a new asymmetric VQGAN with two simple
  designs. Firstly, in addition to the input from the encoder, the decoder
  contains a conditional branch that incorporates information from task-specific
  priors, such as the unmasked image region in inpainting. Secondly, the decoder
  is much heavier than the encoder, allowing for more detailed recovery while
  only slightly increasing the total inference cost. The training cost of our
  asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
  while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
  asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
  local editing methods. Extensive experiments demonstrate that it can
  significantly improve the inpainting and editing performance, while maintaining
  the original text-to-image capability. The code is available at
  https://github.com/buxiangzhiren/Asymmetric_VQGAN.
paper_authors._id: 64813ccb5409aa3e3bbf9799
paper_authors.name: Zixin Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:51:50.013Z
paper_authors._id: 645c8d2ea19f3e64bbecc616
paper_authors.avatarUrl: /avatars/eceafc799002b57eaad01e638cec7aff.svg
paper_authors.fullname: zhuzixing
paper_authors.type: user
paper_authors.user: Zzxxxxx
paper_authors._id: 64813ccb5409aa3e3bbf979a
paper_authors.name: Xuelu Feng
paper_authors._id: 64813ccb5409aa3e3bbf979b
paper_authors.name: Dongdong Chen
paper_authors._id: 64813ccb5409aa3e3bbf979c
paper_authors.name: Jianmin Bao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:07:22.919Z
paper_authors._id: 6481e690f9ed842838a2b106
paper_authors.avatarUrl: /avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
paper_authors.fullname: Jianmin Bao
paper_authors.type: user
paper_authors.user: jianmin-ustc
paper_authors._id: 64813ccb5409aa3e3bbf979d
paper_authors.name: Le Wang
paper_authors._id: 64813ccb5409aa3e3bbf979e
paper_authors.name: Yinpeng Chen
paper_authors._id: 64813ccb5409aa3e3bbf979f
paper_authors.name: Lu Yuan
paper_authors._id: 64813ccb5409aa3e3bbf97a0
paper_authors.name: Gang Hua
paper_title: Designing a Better Asymmetric VQGAN for StableDiffusion
paper_summary: StableDiffusion is a revolutionary text-to-image generator that is causing a
  stir in the world of image generation and editing. Unlike traditional methods
  that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
  model in the latent space via a VQGAN, ensuring both efficiency and quality. It
  not only supports image generation tasks, but also enables image editing for
  real images, such as image inpainting and local editing. However, we have
  observed that the vanilla VQGAN used in StableDiffusion leads to significant
  information loss, causing distortion artifacts even in non-edited image
  regions. To this end, we propose a new asymmetric VQGAN with two simple
  designs. Firstly, in addition to the input from the encoder, the decoder
  contains a conditional branch that incorporates information from task-specific
  priors, such as the unmasked image region in inpainting. Secondly, the decoder
  is much heavier than the encoder, allowing for more detailed recovery while
  only slightly increasing the total inference cost. The training cost of our
  asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
  while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
  asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
  local editing methods. Extensive experiments demonstrate that it can
  significantly improve the inpainting and editing performance, while maintaining
  the original text-to-image capability. The code is available at
  https://github.com/buxiangzhiren/Asymmetric_VQGAN.
paper_ai_summary: An asymmetric VQGAN enhances StableDiffusion by reducing information loss and improving inpainting and editing performance without altering the original model architecture.
paper_ai_keywords: StableDiffusion diffusion model latent space VQGAN asymmetric VQGAN conditional branch task-specific priors image inpainting local editing encoder decoder inference cost training cost text-to-image capability