title: Knowledge Transfer Across Modalities with Natural Language Supervision
summary: We present a way to learn novel concepts by only using their textual
  description. We call this method Knowledge Transfer. Similarly to human
  perception, we leverage cross-modal interaction to introduce new concepts. We
  hypothesize that in a pre-trained visual encoder there are enough low-level
  features already learned (e.g. shape, appearance, color) that can be used to
  describe previously unknown high-level concepts. Provided with a textual
  description of the novel concept, our method works by aligning the known
  low-level features of the visual encoder to its high-level textual description.
  We show that Knowledge Transfer can successfully introduce novel concepts in
  multimodal models, in a very efficient manner, by only requiring a single
  description of the target concept. Our approach is compatible with both
  separate textual and visual encoders (e.g. CLIP) and shared parameters across
  modalities. We also show that, following the same principle, Knowledge Transfer
  can improve concepts already known by the model. Leveraging Knowledge Transfer
  we improve zero-shot performance across different tasks such as classification,
  segmentation, image-text retrieval, and captioning.
paper_authors._id: 67458484aa42422dcbcb4381
paper_authors.name: Carlo Alberto Barbano
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-26T08:28:39.551Z
paper_authors._id: 64f9f44c3c2a6f731f9ead10
paper_authors.avatarUrl: /avatars/c53a47a2d321e9d8b1bdf5babdfb206c.svg
paper_authors.fullname: Carlo Alberto Barbano
paper_authors.type: user
paper_authors.user: carloalbertobarbano
paper_authors._id: 67458484aa42422dcbcb4382
paper_authors.name: Luca Molinaro
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-26T08:27:48.920Z
paper_authors._id: 666875d810b4ced2a569c089
paper_authors.avatarUrl: /avatars/5080e0a0bb44b911f6621e16476d5cd2.svg
paper_authors.fullname: Luca Molinaro
paper_authors.type: user
paper_authors.user: luca-molinaro
paper_authors._id: 67458484aa42422dcbcb4383
paper_authors.name: Emanuele Aiello
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:20:44.600Z
paper_authors._id: 6515ac7f60757b8c8f6b8d31
paper_authors.avatarUrl: /avatars/ed033f30f9f78ed68da976e6d01af0ea.svg
paper_authors.fullname: Emanuele Aiello
paper_authors.type: user
paper_authors.user: Ema97x
paper_authors._id: 67458484aa42422dcbcb4384
paper_authors.name: Marco Grangetto
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-11T12:17:33.514Z
paper_authors._id: 67ab2a39f2d5a5a57e4c9a60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/MtfhbJTcb0rblQLLEy40v.png
paper_authors.fullname: Marco Grangetto
paper_authors.type: user
paper_authors.user: marcograngetto
paper_title: Knowledge Transfer Across Modalities with Natural Language Supervision
paper_summary: We present a way to learn novel concepts by only using their textual
  description. We call this method Knowledge Transfer. Similarly to human
  perception, we leverage cross-modal interaction to introduce new concepts. We
  hypothesize that in a pre-trained visual encoder there are enough low-level
  features already learned (e.g. shape, appearance, color) that can be used to
  describe previously unknown high-level concepts. Provided with a textual
  description of the novel concept, our method works by aligning the known
  low-level features of the visual encoder to its high-level textual description.
  We show that Knowledge Transfer can successfully introduce novel concepts in
  multimodal models, in a very efficient manner, by only requiring a single
  description of the target concept. Our approach is compatible with both
  separate textual and visual encoders (e.g. CLIP) and shared parameters across
  modalities. We also show that, following the same principle, Knowledge Transfer
  can improve concepts already known by the model. Leveraging Knowledge Transfer
  we improve zero-shot performance across different tasks such as classification,
  segmentation, image-text retrieval, and captioning.
paper_ai_keywords: Knowledge Transfer cross-modal interaction visual encoder low-level features high-level concepts textual description alignment multimodal models zero-shot performance classification segmentation image-text retrieval captioning