title: Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot
    Subject-Driven Image Generator
summary: Subject-driven text-to-image generation aims to produce images of a new
  subject within a desired context by accurately capturing both the visual
  characteristics of the subject and the semantic content of a text prompt.
  Traditional methods rely on time- and resource-intensive fine-tuning for
  subject alignment, while recent zero-shot approaches leverage on-the-fly image
  prompting, often sacrificing subject alignment. In this paper, we introduce
  Diptych Prompting, a novel zero-shot approach that reinterprets as an
  inpainting task with precise subject alignment by leveraging the emergent
  property of diptych generation in large-scale text-to-image models. Diptych
  Prompting arranges an incomplete diptych with the reference image in the left
  panel, and performs text-conditioned inpainting on the right panel. We further
  prevent unwanted content leakage by removing the background in the reference
  image and improve fine-grained details in the generated subject by enhancing
  attention weights between the panels during inpainting. Experimental results
  confirm that our approach significantly outperforms zero-shot image prompting
  methods, resulting in images that are visually preferred by users.
  Additionally, our method supports not only subject-driven generation but also
  stylized image generation and subject-driven image editing, demonstrating
  versatility across diverse image generation applications. Project page:
  https://diptychprompting.github.io/
paper_authors._id: 67453096586b9927fa10108e
paper_authors.name: Chaehun Shin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:14.802Z
paper_authors._id: 631842124d24c00131a66656
paper_authors.avatarUrl: /avatars/590633e163da425bf041e3988d8dd015.svg
paper_authors.fullname: shin
paper_authors.type: user
paper_authors.user: chaehun
paper_authors._id: 67453096586b9927fa10108f
paper_authors.name: Jooyoung Choi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:12.750Z
paper_authors._id: 62f9e54defd2de85492a4dde
paper_authors.avatarUrl: /avatars/9925de0d02b2f2ed03cbafda44fd7de4.svg
paper_authors.fullname: Jooyoung Choi
paper_authors.type: user
paper_authors.user: jychoi
paper_authors._id: 67453096586b9927fa101090
paper_authors.name: Heeseung Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T10:08:19.148Z
paper_authors._id: 6311ac44d5a165db5d1dae38
paper_authors.avatarUrl: /avatars/4ffc53da60e7cc0b74c22ebb319ebdc3.svg
paper_authors.fullname: Heeseung Kim
paper_authors.type: user
paper_authors.user: KHS
paper_authors._id: 67453096586b9927fa101091
paper_authors.name: Sungroh Yoon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T22:56:29.172Z
paper_authors._id: 64c99967d33e6a7f82c35f0b
paper_authors.avatarUrl: /avatars/70e77eb570d12e11c2807e10cd4d926d.svg
paper_authors.fullname: Sungroh Yoon
paper_authors.type: user
paper_authors.user: sryoon
paper_title: Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot
    Subject-Driven Image Generator
paper_summary: Subject-driven text-to-image generation aims to produce images of a new
  subject within a desired context by accurately capturing both the visual
  characteristics of the subject and the semantic content of a text prompt.
  Traditional methods rely on time- and resource-intensive fine-tuning for
  subject alignment, while recent zero-shot approaches leverage on-the-fly image
  prompting, often sacrificing subject alignment. In this paper, we introduce
  Diptych Prompting, a novel zero-shot approach that reinterprets as an
  inpainting task with precise subject alignment by leveraging the emergent
  property of diptych generation in large-scale text-to-image models. Diptych
  Prompting arranges an incomplete diptych with the reference image in the left
  panel, and performs text-conditioned inpainting on the right panel. We further
  prevent unwanted content leakage by removing the background in the reference
  image and improve fine-grained details in the generated subject by enhancing
  attention weights between the panels during inpainting. Experimental results
  confirm that our approach significantly outperforms zero-shot image prompting
  methods, resulting in images that are visually preferred by users.
  Additionally, our method supports not only subject-driven generation but also
  stylized image generation and subject-driven image editing, demonstrating
  versatility across diverse image generation applications. Project page:
  https://diptychprompting.github.io/
paper_ai_keywords: subject-driven text-to-image generation zero-shot approaches Diptych Prompting inpainting task diptych generation text-conditioned inpainting attention weights subject-driven generation stylized image generation subject-driven image editing