DreamMatcher: Appearance Matching Self-Attention for
  Semantically-Consistent Text-to-Image Personalization
The objective of text-to-image (T2I) personalization is to customize a
diffusion model to a user-provided reference concept, generating diverse images
of the concept aligned with the target prompts. Conventional methods
representing the reference concepts using unique text embeddings often fail to
accurately mimic the appearance of the reference. To address this, one solution
may be explicitly conditioning the reference images into the target denoising
process, known as key-value replacement. However, prior works are constrained
to local editing since they disrupt the structure path of the pre-trained T2I
model. To overcome this, we propose a novel plug-in method, called
DreamMatcher, which reformulates T2I personalization as semantic matching.
Specifically, DreamMatcher replaces the target values with reference values
aligned by semantic matching, while leaving the structure path unchanged to
preserve the versatile capability of pre-trained T2I models for generating
diverse structures. We also introduce a semantic-consistent masking strategy to
isolate the personalized concept from irrelevant regions introduced by the
target prompts. Compatible with existing T2I models, DreamMatcher shows
significant improvements in complex scenarios. Intensive analyses demonstrate
the effectiveness of our approach.
65ceea67e83461df6cbcca9c
Jisu Nam
65ceea67e83461df6cbcca9d
Heesu Kim
65ceea67e83461df6cbcca9e
DongJae Lee
65ceea67e83461df6cbcca9f
Siyoon Jin
admin_assigned
2024-02-16T14:25:37.321Z
640dd32a3c82bd463ee38e02
/avatars/0d02bb8245785b7ad7cbbee855e2cf1f.svg
Siyoon Jin
user
clwm01
65ceea67e83461df6cbccaa0
Seungryong Kim
claimed_verified
2024-02-16T14:32:00.849Z
65cf717450818a335a1d3021
/avatars/382a0e0f40f661cda1b2531e3e6ea2ee.svg
Seungryong Kim
user
seungryong
65ceea67e83461df6cbccaa1
Seunggyu Chang
DreamMatcher: Appearance Matching Self-Attention for
  Semantically-Consistent Text-to-Image Personalization
The objective of text-to-image (T2I) personalization is to customize a
diffusion model to a user-provided reference concept, generating diverse images
of the concept aligned with the target prompts. Conventional methods
representing the reference concepts using unique text embeddings often fail to
accurately mimic the appearance of the reference. To address this, one solution
may be explicitly conditioning the reference images into the target denoising
process, known as key-value replacement. However, prior works are constrained
to local editing since they disrupt the structure path of the pre-trained T2I
model. To overcome this, we propose a novel plug-in method, called
DreamMatcher, which reformulates T2I personalization as semantic matching.
Specifically, DreamMatcher replaces the target values with reference values
aligned by semantic matching, while leaving the structure path unchanged to
preserve the versatile capability of pre-trained T2I models for generating
diverse structures. We also introduce a semantic-consistent masking strategy to
isolate the personalized concept from irrelevant regions introduced by the
target prompts. Compatible with existing T2I models, DreamMatcher shows
significant improvements in complex scenarios. Intensive analyses demonstrate
the effectiveness of our approach.
DreamMatcher personalizes text-to-image (T2I) models by semantically matching reference concepts to target prompts, improving accuracy and diversity without altering the model's structure.
text-to-image (T2I) diffusion model key-value replacement semantic matching semantic-consistent masking strategy