Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot
  Subject-Driven Image Generator
Subject-driven text-to-image generation aims to produce images of a new
subject within a desired context by accurately capturing both the visual
characteristics of the subject and the semantic content of a text prompt.
Traditional methods rely on time- and resource-intensive fine-tuning for
subject alignment, while recent zero-shot approaches leverage on-the-fly image
prompting, often sacrificing subject alignment. In this paper, we introduce
Diptych Prompting, a novel zero-shot approach that reinterprets as an
inpainting task with precise subject alignment by leveraging the emergent
property of diptych generation in large-scale text-to-image models. Diptych
Prompting arranges an incomplete diptych with the reference image in the left
panel, and performs text-conditioned inpainting on the right panel. We further
prevent unwanted content leakage by removing the background in the reference
image and improve fine-grained details in the generated subject by enhancing
attention weights between the panels during inpainting. Experimental results
confirm that our approach significantly outperforms zero-shot image prompting
methods, resulting in images that are visually preferred by users.
Additionally, our method supports not only subject-driven generation but also
stylized image generation and subject-driven image editing, demonstrating
versatility across diverse image generation applications. Project page:
https://diptychprompting.github.io/
67453096586b9927fa10108e
Chaehun Shin
claimed_verified
2024-11-26T21:21:14.802Z
631842124d24c00131a66656
/avatars/590633e163da425bf041e3988d8dd015.svg
shin
user
chaehun
67453096586b9927fa10108f
Jooyoung Choi
claimed_verified
2024-11-26T21:21:12.750Z
62f9e54defd2de85492a4dde
/avatars/9925de0d02b2f2ed03cbafda44fd7de4.svg
Jooyoung Choi
user
jychoi
67453096586b9927fa101090
Heeseung Kim
claimed_verified
2024-11-29T10:08:19.148Z
6311ac44d5a165db5d1dae38
/avatars/4ffc53da60e7cc0b74c22ebb319ebdc3.svg
Heeseung Kim
user
KHS
67453096586b9927fa101091
Sungroh Yoon
claimed_verified
2024-11-26T22:56:29.172Z
64c99967d33e6a7f82c35f0b
/avatars/70e77eb570d12e11c2807e10cd4d926d.svg
Sungroh Yoon
user
sryoon
Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot
  Subject-Driven Image Generator
Subject-driven text-to-image generation aims to produce images of a new
subject within a desired context by accurately capturing both the visual
characteristics of the subject and the semantic content of a text prompt.
Traditional methods rely on time- and resource-intensive fine-tuning for
subject alignment, while recent zero-shot approaches leverage on-the-fly image
prompting, often sacrificing subject alignment. In this paper, we introduce
Diptych Prompting, a novel zero-shot approach that reinterprets as an
inpainting task with precise subject alignment by leveraging the emergent
property of diptych generation in large-scale text-to-image models. Diptych
Prompting arranges an incomplete diptych with the reference image in the left
panel, and performs text-conditioned inpainting on the right panel. We further
prevent unwanted content leakage by removing the background in the reference
image and improve fine-grained details in the generated subject by enhancing
attention weights between the panels during inpainting. Experimental results
confirm that our approach significantly outperforms zero-shot image prompting
methods, resulting in images that are visually preferred by users.
Additionally, our method supports not only subject-driven generation but also
stylized image generation and subject-driven image editing, demonstrating
versatility across diverse image generation applications. Project page:
https://diptychprompting.github.io/
Diptych Prompting is a zero-shot text-to-image generation method that enhances subject alignment and attention between panels during inpainting, resulting in visually preferable images across various applications.
zero-shot approaches text-to-image generation subject alignment diptych generation inpainting attention weights stylized image generation subject-driven image editing