Knowledge Transfer Across Modalities with Natural Language Supervision
We present a way to learn novel concepts by only using their textual
description. We call this method Knowledge Transfer. Similarly to human
perception, we leverage cross-modal interaction to introduce new concepts. We
hypothesize that in a pre-trained visual encoder there are enough low-level
features already learned (e.g. shape, appearance, color) that can be used to
describe previously unknown high-level concepts. Provided with a textual
description of the novel concept, our method works by aligning the known
low-level features of the visual encoder to its high-level textual description.
We show that Knowledge Transfer can successfully introduce novel concepts in
multimodal models, in a very efficient manner, by only requiring a single
description of the target concept. Our approach is compatible with both
separate textual and visual encoders (e.g. CLIP) and shared parameters across
modalities. We also show that, following the same principle, Knowledge Transfer
can improve concepts already known by the model. Leveraging Knowledge Transfer
we improve zero-shot performance across different tasks such as classification,
segmentation, image-text retrieval, and captioning.
67458484aa42422dcbcb4381
Carlo Alberto Barbano
extracted_confirmed
2024-11-26T08:28:39.551Z
64f9f44c3c2a6f731f9ead10
/avatars/c53a47a2d321e9d8b1bdf5babdfb206c.svg
Carlo Alberto Barbano
user
carloalbertobarbano
67458484aa42422dcbcb4382
Luca Molinaro
extracted_confirmed
2024-11-26T08:27:48.920Z
666875d810b4ced2a569c089
/avatars/5080e0a0bb44b911f6621e16476d5cd2.svg
Luca Molinaro
user
luca-molinaro
67458484aa42422dcbcb4383
Emanuele Aiello
claimed_verified
2024-11-26T21:20:44.600Z
6515ac7f60757b8c8f6b8d31
/avatars/ed033f30f9f78ed68da976e6d01af0ea.svg
Emanuele Aiello
user
Ema97x
67458484aa42422dcbcb4384
Marco Grangetto
extracted_confirmed
2025-02-11T12:17:33.514Z
67ab2a39f2d5a5a57e4c9a60
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/MtfhbJTcb0rblQLLEy40v.png
Marco Grangetto
user
marcograngetto
Knowledge Transfer Across Modalities with Natural Language Supervision
We present a way to learn novel concepts by only using their textual
description. We call this method Knowledge Transfer. Similarly to human
perception, we leverage cross-modal interaction to introduce new concepts. We
hypothesize that in a pre-trained visual encoder there are enough low-level
features already learned (e.g. shape, appearance, color) that can be used to
describe previously unknown high-level concepts. Provided with a textual
description of the novel concept, our method works by aligning the known
low-level features of the visual encoder to its high-level textual description.
We show that Knowledge Transfer can successfully introduce novel concepts in
multimodal models, in a very efficient manner, by only requiring a single
description of the target concept. Our approach is compatible with both
separate textual and visual encoders (e.g. CLIP) and shared parameters across
modalities. We also show that, following the same principle, Knowledge Transfer
can improve concepts already known by the model. Leveraging Knowledge Transfer
we improve zero-shot performance across different tasks such as classification,
segmentation, image-text retrieval, and captioning.
Knowledge Transfer uses textual descriptions to efficiently introduce and enhance novel concepts in multimodal models, improving zero-shot performance.
cross-modal interaction visual encoder low-level features high-level concepts textual description multimodal models zero-shot performance classification segmentation image-text retrieval captioning CLIP parameter sharing