title: CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
    Personalization
summary: Recent advances in text-to-image personalization have enabled high-quality
  and controllable image synthesis for user-provided concepts. However, existing
  methods still struggle to balance identity preservation with text alignment.
  Our approach is based on the fact that generating prompt-aligned images
  requires a precise semantic understanding of the prompt, which involves
  accurately processing the interactions between the new concept and its
  surrounding context tokens within the CLIP text encoder. To address this, we
  aim to embed the new concept properly into the input embedding space of the
  text encoder, allowing for seamless integration with existing tokens. We
  introduce Context Regularization (CoRe), which enhances the learning of the new
  concept's text embedding by regularizing its context tokens in the prompt. This
  is based on the insight that appropriate output vectors of the text encoder for
  the context tokens can only be achieved if the new concept's text embedding is
  correctly learned. CoRe can be applied to arbitrary prompts without requiring
  the generation of corresponding images, thus improving the generalization of
  the learned text embedding. Additionally, CoRe can serve as a test-time
  optimization technique to further enhance the generations for specific prompts.
  Comprehensive experiments demonstrate that our method outperforms several
  baseline methods in both identity preservation and text alignment. Code will be
  made publicly available.
paper_authors._id: 66d564d66839bd191d8c4138
paper_authors.name: Feize Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:16:16.037Z
paper_authors._id: 65af7c59bff3ae9f33990ed5
paper_authors.avatarUrl: /avatars/3de08df23883e1a0b7f36d2a991f9f24.svg
paper_authors.fullname: FeizeWu
paper_authors.type: user
paper_authors.user: FeizeWu
paper_authors._id: 66d564d66839bd191d8c4139
paper_authors.name: Yun Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:05:49.993Z
paper_authors._id: 64049b100ab5e22719f352fa
paper_authors.avatarUrl: /avatars/e11c190f3cb6aa0ba0e0c106feb1e09e.svg
paper_authors.fullname: Yun Pang
paper_authors.type: user
paper_authors.user: darkcser
paper_authors._id: 66d564d66839bd191d8c413a
paper_authors.name: Junyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T09:21:12.338Z
paper_authors._id: 65ec0ae3b1e3f90307a7a0d6
paper_authors.avatarUrl: /avatars/83594b42c965c2a3f6e3764077c40bb6.svg
paper_authors.fullname: Xiaoquan Zhang
paper_authors.type: user
paper_authors.user: Zhangjunyi
paper_authors._id: 66d564d66839bd191d8c413b
paper_authors.name: Lianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T09:21:50.257Z
paper_authors._id: 664df4c0f1f205ad9018cb3a
paper_authors.avatarUrl: /avatars/b52f784b0798c94863526e4d9682c456.svg
paper_authors.fullname: YunPang
paper_authors.type: user
paper_authors.user: BlurBlur
paper_authors._id: 66d564d66839bd191d8c413c
paper_authors.name: Jian Yin
paper_authors._id: 66d564d66839bd191d8c413d
paper_authors.name: Baoquan Zhao
paper_authors._id: 66d564d66839bd191d8c413e
paper_authors.name: Qing Li
paper_authors._id: 66d564d66839bd191d8c413f
paper_authors.name: Xudong Mao
paper_title: CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
    Personalization
paper_summary: Recent advances in text-to-image personalization have enabled high-quality
  and controllable image synthesis for user-provided concepts. However, existing
  methods still struggle to balance identity preservation with text alignment.
  Our approach is based on the fact that generating prompt-aligned images
  requires a precise semantic understanding of the prompt, which involves
  accurately processing the interactions between the new concept and its
  surrounding context tokens within the CLIP text encoder. To address this, we
  aim to embed the new concept properly into the input embedding space of the
  text encoder, allowing for seamless integration with existing tokens. We
  introduce Context Regularization (CoRe), which enhances the learning of the new
  concept's text embedding by regularizing its context tokens in the prompt. This
  is based on the insight that appropriate output vectors of the text encoder for
  the context tokens can only be achieved if the new concept's text embedding is
  correctly learned. CoRe can be applied to arbitrary prompts without requiring
  the generation of corresponding images, thus improving the generalization of
  the learned text embedding. Additionally, CoRe can serve as a test-time
  optimization technique to further enhance the generations for specific prompts.
  Comprehensive experiments demonstrate that our method outperforms several
  baseline methods in both identity preservation and text alignment. Code will be
  made publicly available.
paper_ai_keywords: Context Regularization (CoRe) CLIP text encoder prompt-aligned images semantic understanding context tokens input embedding space text embedding output vectors identity preservation text alignment generalization test-time optimization technique