ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image
  Generation
Personalized text-to-image generation using diffusion models has recently
been proposed and attracted lots of attention. Given a handful of images
containing a novel concept (e.g., a unique toy), we aim to tune the generative
model to capture fine visual details of the novel concept and generate
photorealistic images following a text condition. We present a plug-in method,
named ViCo, for fast and lightweight personalized generation. Specifically, we
propose an image attention module to condition the diffusion process on the
patch-wise visual semantics. We introduce an attention-based object mask that
comes almost at no cost from the attention module. In addition, we design a
simple regularization based on the intrinsic properties of text-image attention
maps to alleviate the common overfitting degradation. Unlike many existing
models, our method does not finetune any parameters of the original diffusion
model. This allows more flexible and transferable model deployment. With only
light parameter training (~6% of the diffusion U-Net), our method achieves
comparable or even better performance than all state-of-the-art models both
qualitatively and quantitatively.
64797df550ff7001631a3906
Shaozhe Hao
claimed_verified
2023-06-02T12:46:28.898Z
63b67ab2877d94d5c04946e1
/avatars/56863b04863969b9aaa35194e25d1e82.svg
Shaozhe Hao
user
haoosz
64797df550ff7001631a3907
Kai Han
64797df550ff7001631a3908
Shihao Zhao
64797df550ff7001631a3909
Kwan-Yee K. Wong
ViCo: Detail-Preserving Visual Condition for Personalized Text-to-Image
  Generation
Personalized text-to-image generation using diffusion models has recently
been proposed and attracted lots of attention. Given a handful of images
containing a novel concept (e.g., a unique toy), we aim to tune the generative
model to capture fine visual details of the novel concept and generate
photorealistic images following a text condition. We present a plug-in method,
named ViCo, for fast and lightweight personalized generation. Specifically, we
propose an image attention module to condition the diffusion process on the
patch-wise visual semantics. We introduce an attention-based object mask that
comes almost at no cost from the attention module. In addition, we design a
simple regularization based on the intrinsic properties of text-image attention
maps to alleviate the common overfitting degradation. Unlike many existing
models, our method does not finetune any parameters of the original diffusion
model. This allows more flexible and transferable model deployment. With only
light parameter training (~6% of the diffusion U-Net), our method achieves
comparable or even better performance than all state-of-the-art models both
qualitatively and quantitatively.
ViCo, a plug-in method using diffusion models, achieves high-quality personalized text-to-image generation with minimal parameter training through an image attention module and attention-based object masks.
diffusion models ViCo image attention module patch-wise visual semantics attention-based object masks text-image attention maps overfitting degradation diffusion U-Net