An Image is Worth Multiple Words: Learning Object Level Concepts using
  Multi-Concept Prompt Learning
Textural Inversion, a prompt learning method, learns a singular embedding for
a new "word" to represent image style and appearance, allowing it to be
integrated into natural language sentences to generate novel synthesised
images. However, identifying and integrating multiple object-level concepts
within one scene poses significant challenges even when embeddings for
individual concepts are attainable. This is further confirmed by our empirical
tests. To address this challenge, we introduce a framework for Multi-Concept
Prompt Learning (MCPL), where multiple new "words" are simultaneously learned
from a single sentence-image pair. To enhance the accuracy of word-concept
correlation, we propose three regularisation techniques: Attention Masking
(AttnMask) to concentrate learning on relevant areas; Prompts Contrastive Loss
(PromptCL) to separate the embeddings of different concepts; and Bind adjective
(Bind adj.) to associate new "words" with known words. We evaluate via image
generation, editing, and attention visualisation with diverse images. Extensive
quantitative comparisons demonstrate that our method can learn more
semantically disentangled concepts with enhanced word-concept correlation.
Additionally, we introduce a novel dataset and evaluation protocol tailored for
this new task of learning object-level concepts.
6532117b829e1dc2f291438b
Chen Jin
claimed_verified
2023-10-20T12:44:32.606Z
62d691726279202d95f86bb6
/avatars/a52567e73e6b4e84219cb32c85ee8c1f.svg
Chen Jin
user
lxasqjc
6532117b829e1dc2f291438c
Ryutaro Tanno
6532117b829e1dc2f291438d
Amrutha Saseendran
6532117b829e1dc2f291438e
Tom Diethe
6532117b829e1dc2f291438f
Philip Teare
claimed_verified
2023-10-21T09:22:28.981Z
631da8c0f318ed8dfd32370f
/avatars/d012e65e83fee403a510dff3f032232d.svg
Philip Teare
user
evolu8
An Image is Worth Multiple Words: Learning Object Level Concepts using
  Multi-Concept Prompt Learning
Textural Inversion, a prompt learning method, learns a singular embedding for
a new "word" to represent image style and appearance, allowing it to be
integrated into natural language sentences to generate novel synthesised
images. However, identifying and integrating multiple object-level concepts
within one scene poses significant challenges even when embeddings for
individual concepts are attainable. This is further confirmed by our empirical
tests. To address this challenge, we introduce a framework for Multi-Concept
Prompt Learning (MCPL), where multiple new "words" are simultaneously learned
from a single sentence-image pair. To enhance the accuracy of word-concept
correlation, we propose three regularisation techniques: Attention Masking
(AttnMask) to concentrate learning on relevant areas; Prompts Contrastive Loss
(PromptCL) to separate the embeddings of different concepts; and Bind adjective
(Bind adj.) to associate new "words" with known words. We evaluate via image
generation, editing, and attention visualisation with diverse images. Extensive
quantitative comparisons demonstrate that our method can learn more
semantically disentangled concepts with enhanced word-concept correlation.
Additionally, we introduce a novel dataset and evaluation protocol tailored for
this new task of learning object-level concepts.
A novel Multi-Concept Prompt Learning framework (MCPL) learns multiple embeddings from a single sentence-image pair to represent object-level concepts in images, using regularisation techniques to enhance accuracy.
prompt learning Textural Inversion image style image appearance Multi-Concept Prompt Learning MCPL Attention Masking AttnMask Prompts Contrastive Loss PromptCL Bind adjective Bind adj. word-concept correlation image generation image editing attention visualisation semantic disentangled concepts dataset evaluation protocol