title: MyVLM: Personalizing VLMs for User-Specific Queries
summary: Recent large-scale vision-language models (VLMs) have demonstrated remarkable
  capabilities in understanding and generating textual descriptions for visual
  content. However, these models lack an understanding of user-specific concepts.
  In this work, we take a first step toward the personalization of VLMs, enabling
  them to learn and reason over user-provided concepts. For example, we explore
  whether these models can learn to recognize you in an image and communicate
  what you are doing, tailoring the model to reflect your personal experiences
  and relationships. To effectively recognize a variety of user-specific
  concepts, we augment the VLM with external concept heads that function as
  toggles for the model, enabling the VLM to identify the presence of specific
  target concepts in a given image. Having recognized the concept, we learn a new
  concept embedding in the intermediate feature space of the VLM. This embedding
  is tasked with guiding the language model to naturally integrate the target
  concept in its generated response. We apply our technique to BLIP-2 and LLaVA
  for personalized image captioning and further show its applicability for
  personalized visual question-answering. Our experiments demonstrate our ability
  to generalize to unseen images of learned concepts while preserving the model
  behavior on unrelated inputs.
paper_authors._id: 65fd0b9eeb491be2c7cfd661
paper_authors.name: Yuval Alaluf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:06:18.974Z
paper_authors._id: 6310c1e0e87051f3e3e3bcc1
paper_authors.avatarUrl: /avatars/61b24a2eace7da0e27fbc4369a046051.svg
paper_authors.fullname: Yuval Alaluf
paper_authors.type: user
paper_authors.user: yuvalalaluf
paper_authors._id: 65fd0b9eeb491be2c7cfd662
paper_authors.name: Elad Richardson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:06:28.343Z
paper_authors._id: 633f370c13e836a0fc513724
paper_authors.avatarUrl: /avatars/38f3f9ecc5b9f35026becfb0c5c94fdf.svg
paper_authors.fullname: Elad Richardson
paper_authors.type: user
paper_authors.user: eladrich
paper_authors._id: 65fd0b9eeb491be2c7cfd663
paper_authors.name: Sergey Tulyakov
paper_authors._id: 65fd0b9eeb491be2c7cfd664
paper_authors.name: Kfir Aberman
paper_authors._id: 65fd0b9eeb491be2c7cfd665
paper_authors.name: Daniel Cohen-Or
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-22T10:06:43.359Z
paper_authors._id: 628507161949ebcae8e24ec3
paper_authors.avatarUrl: /avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
paper_authors.fullname: Daniel Cohen-Or
paper_authors.type: user
paper_authors.user: cohenor
paper_title: MyVLM: Personalizing VLMs for User-Specific Queries
paper_summary: Recent large-scale vision-language models (VLMs) have demonstrated remarkable
  capabilities in understanding and generating textual descriptions for visual
  content. However, these models lack an understanding of user-specific concepts.
  In this work, we take a first step toward the personalization of VLMs, enabling
  them to learn and reason over user-provided concepts. For example, we explore
  whether these models can learn to recognize you in an image and communicate
  what you are doing, tailoring the model to reflect your personal experiences
  and relationships. To effectively recognize a variety of user-specific
  concepts, we augment the VLM with external concept heads that function as
  toggles for the model, enabling the VLM to identify the presence of specific
  target concepts in a given image. Having recognized the concept, we learn a new
  concept embedding in the intermediate feature space of the VLM. This embedding
  is tasked with guiding the language model to naturally integrate the target
  concept in its generated response. We apply our technique to BLIP-2 and LLaVA
  for personalized image captioning and further show its applicability for
  personalized visual question-answering. Our experiments demonstrate our ability
  to generalize to unseen images of learned concepts while preserving the model
  behavior on unrelated inputs.
paper_ai_summary: External concept heads augment VLMs to integrate user-specific concepts into image captioning and question-answering, demonstrating generalization to unseen images.
paper_ai_keywords: large-scale vision-language models VLMs personalization external concept heads concept embedding intermediate feature space BLIP-2 LLaVA image captioning visual question-answering