title: Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
summary: Semantic image synthesis, i.e., generating images from user-provided semantic
  label maps, is an important conditional image generation task as it allows to
  control both the content as well as the spatial layout of generated images.
  Although diffusion models have pushed the state of the art in generative image
  modeling, the iterative nature of their inference process makes them
  computationally demanding. Other approaches such as GANs are more efficient as
  they only need a single feed-forward pass for generation, but the image quality
  tends to suffer on large and diverse datasets. In this work, we propose a new
  class of GAN discriminators for semantic image synthesis that generates highly
  realistic images by exploiting feature backbone networks pre-trained for tasks
  such as image classification. We also introduce a new generator architecture
  with better context modeling and using cross-attention to inject noise into
  latent variables, leading to more diverse generated images. Our model, which we
  dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
  consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
  surpassing recent diffusion models while requiring two orders of magnitude less
  compute for inference.
paper_authors._id: 658510d7c7918aa044940648
paper_authors.name: Tariq Berrada
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T12:33:50.167Z
paper_authors._id: 65858134e07563d421117279
paper_authors.avatarUrl: /avatars/08fb3f98cd96aa1828f5b9929b30cbb7.svg
paper_authors.fullname: Tariq Berrada
paper_authors.type: user
paper_authors.user: TariqBerrada
paper_authors._id: 658510d7c7918aa044940649
paper_authors.name: Jakob Verbeek
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-16T15:59:29.103Z
paper_authors._id: 670fe1c99e1efdfc2ea39665
paper_authors.avatarUrl: /avatars/78d777c0bd6c490a1e5ba7ad7d038335.svg
paper_authors.fullname: Jakob Verbeek
paper_authors.type: user
paper_authors.user: jjverbeek
paper_authors._id: 658510d7c7918aa04494064a
paper_authors.name: Camille Couprie
paper_authors._id: 658510d7c7918aa04494064b
paper_authors.name: Karteek Alahari
paper_title: Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
paper_summary: Semantic image synthesis, i.e., generating images from user-provided semantic
  label maps, is an important conditional image generation task as it allows to
  control both the content as well as the spatial layout of generated images.
  Although diffusion models have pushed the state of the art in generative image
  modeling, the iterative nature of their inference process makes them
  computationally demanding. Other approaches such as GANs are more efficient as
  they only need a single feed-forward pass for generation, but the image quality
  tends to suffer on large and diverse datasets. In this work, we propose a new
  class of GAN discriminators for semantic image synthesis that generates highly
  realistic images by exploiting feature backbone networks pre-trained for tasks
  such as image classification. We also introduce a new generator architecture
  with better context modeling and using cross-attention to inject noise into
  latent variables, leading to more diverse generated images. Our model, which we
  dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
  consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
  surpassing recent diffusion models while requiring two orders of magnitude less
  compute for inference.
paper_ai_keywords: diffusion models generative image modeling inference process GANs feed-forward pass discriminators feature backbone networks image classification context modeling cross-attention latent variables image quality consistency DP-SIMS ADE-20K COCO-Stuff Cityscapes