title: ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image
    Generation
summary: Personalized text-to-image generation using diffusion models has recently
  been proposed and attracted lots of attention. Given a handful of images
  containing a novel concept (e.g., a unique toy), we aim to tune the generative
  model to capture fine visual details of the novel concept and generate
  photorealistic images following a text condition. We present a plug-in method,
  named ViCo, for fast and lightweight personalized generation. Specifically, we
  propose an image attention module to condition the diffusion process on the
  patch-wise visual semantics. We introduce an attention-based object mask that
  comes almost at no cost from the attention module. In addition, we design a
  simple regularization based on the intrinsic properties of text-image attention
  maps to alleviate the common overfitting degradation. Unlike many existing
  models, our method does not finetune any parameters of the original diffusion
  model. This allows more flexible and transferable model deployment. With only
  light parameter training (~6% of the diffusion U-Net), our method achieves
  comparable or even better performance than all state-of-the-art models both
  qualitatively and quantitatively.
paper_authors._id: 64797df550ff7001631a3906
paper_authors.name: Shaozhe Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T12:46:28.898Z
paper_authors._id: 63b67ab2877d94d5c04946e1
paper_authors.avatarUrl: /avatars/56863b04863969b9aaa35194e25d1e82.svg
paper_authors.fullname: Shaozhe Hao
paper_authors.type: user
paper_authors.user: haoosz
paper_authors._id: 64797df550ff7001631a3907
paper_authors.name: Kai Han
paper_authors._id: 64797df550ff7001631a3908
paper_authors.name: Shihao Zhao
paper_authors._id: 64797df550ff7001631a3909
paper_authors.name: Kwan-Yee K. Wong
paper_title: ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image
    Generation
paper_summary: Personalized text-to-image generation using diffusion models has recently
  been proposed and attracted lots of attention. Given a handful of images
  containing a novel concept (e.g., a unique toy), we aim to tune the generative
  model to capture fine visual details of the novel concept and generate
  photorealistic images following a text condition. We present a plug-in method,
  named ViCo, for fast and lightweight personalized generation. Specifically, we
  propose an image attention module to condition the diffusion process on the
  patch-wise visual semantics. We introduce an attention-based object mask that
  comes almost at no cost from the attention module. In addition, we design a
  simple regularization based on the intrinsic properties of text-image attention
  maps to alleviate the common overfitting degradation. Unlike many existing
  models, our method does not finetune any parameters of the original diffusion
  model. This allows more flexible and transferable model deployment. With only
  light parameter training (~6% of the diffusion U-Net), our method achieves
  comparable or even better performance than all state-of-the-art models both
  qualitatively and quantitatively.
paper_ai_keywords: diffusion models text-to-image generation generative model visual details photorealistic images image attention module patch-wise visual semantics attention-based object mask discrimination regularization overfitting degradation finetune diffusion U-Net state-of-the-art models