title: DreamMatcher: Appearance Matching Self-Attention for
    Semantically-Consistent Text-to-Image Personalization
summary: The objective of text-to-image (T2I) personalization is to customize a
  diffusion model to a user-provided reference concept, generating diverse images
  of the concept aligned with the target prompts. Conventional methods
  representing the reference concepts using unique text embeddings often fail to
  accurately mimic the appearance of the reference. To address this, one solution
  may be explicitly conditioning the reference images into the target denoising
  process, known as key-value replacement. However, prior works are constrained
  to local editing since they disrupt the structure path of the pre-trained T2I
  model. To overcome this, we propose a novel plug-in method, called
  DreamMatcher, which reformulates T2I personalization as semantic matching.
  Specifically, DreamMatcher replaces the target values with reference values
  aligned by semantic matching, while leaving the structure path unchanged to
  preserve the versatile capability of pre-trained T2I models for generating
  diverse structures. We also introduce a semantic-consistent masking strategy to
  isolate the personalized concept from irrelevant regions introduced by the
  target prompts. Compatible with existing T2I models, DreamMatcher shows
  significant improvements in complex scenarios. Intensive analyses demonstrate
  the effectiveness of our approach.
paper_authors._id: 65ceea67e83461df6cbcca9c
paper_authors.name: Jisu Nam
paper_authors._id: 65ceea67e83461df6cbcca9d
paper_authors.name: Heesu Kim
paper_authors._id: 65ceea67e83461df6cbcca9e
paper_authors.name: DongJae Lee
paper_authors._id: 65ceea67e83461df6cbcca9f
paper_authors.name: Siyoon Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T14:25:37.321Z
paper_authors._id: 640dd32a3c82bd463ee38e02
paper_authors.avatarUrl: /avatars/0d02bb8245785b7ad7cbbee855e2cf1f.svg
paper_authors.fullname: Siyoon Jin
paper_authors.type: user
paper_authors.user: clwm01
paper_authors._id: 65ceea67e83461df6cbccaa0
paper_authors.name: Seungryong Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-16T14:32:00.849Z
paper_authors._id: 65cf717450818a335a1d3021
paper_authors.avatarUrl: /avatars/382a0e0f40f661cda1b2531e3e6ea2ee.svg
paper_authors.fullname: Seungryong Kim
paper_authors.type: user
paper_authors.user: seungryong
paper_authors._id: 65ceea67e83461df6cbccaa1
paper_authors.name: Seunggyu Chang
paper_title: DreamMatcher: Appearance Matching Self-Attention for
    Semantically-Consistent Text-to-Image Personalization
paper_summary: The objective of text-to-image (T2I) personalization is to customize a
  diffusion model to a user-provided reference concept, generating diverse images
  of the concept aligned with the target prompts. Conventional methods
  representing the reference concepts using unique text embeddings often fail to
  accurately mimic the appearance of the reference. To address this, one solution
  may be explicitly conditioning the reference images into the target denoising
  process, known as key-value replacement. However, prior works are constrained
  to local editing since they disrupt the structure path of the pre-trained T2I
  model. To overcome this, we propose a novel plug-in method, called
  DreamMatcher, which reformulates T2I personalization as semantic matching.
  Specifically, DreamMatcher replaces the target values with reference values
  aligned by semantic matching, while leaving the structure path unchanged to
  preserve the versatile capability of pre-trained T2I models for generating
  diverse structures. We also introduce a semantic-consistent masking strategy to
  isolate the personalized concept from irrelevant regions introduced by the
  target prompts. Compatible with existing T2I models, DreamMatcher shows
  significant improvements in complex scenarios. Intensive analyses demonstrate
  the effectiveness of our approach.
paper_ai_keywords: text-to-image (T2I) personalization denoising text embeddings key-value replacement semantic matching semantic-consistent masking strategy pre-trained T2I models search and rescue