title: Meta-Personalizing Vision-Language Models to Find Named Instances in
    Video
summary: Large-scale vision-language models (VLM) have shown impressive results for
  language-guided search applications. While these models allow category-level
  queries, they currently struggle with personalized searches for moments in a
  video where a specific object instance such as ``My dog Biscuit'' appears. We
  present the following three contributions to address this problem. First, we
  describe a method to meta-personalize a pre-trained VLM, i.e., learning how to
  learn to personalize a VLM at test time to search in video. Our method extends
  the VLM's token vocabulary by learning novel word embeddings specific to each
  instance. To capture only instance-specific features, we represent each
  instance embedding as a combination of shared and learned global category
  features. Second, we propose to learn such personalization without explicit
  human supervision. Our approach automatically identifies moments of named
  visual instances in video using transcripts and vision-language similarity in
  the VLM's embedding space. Finally, we introduce This-Is-My, a personal video
  instance retrieval benchmark. We evaluate our approach on This-Is-My and
  DeepFashion2 and show that we obtain a 15% relative improvement over the state
  of the art on the latter dataset.
paper_authors._id: 649259bba820f46b2cbf85b0
paper_authors.name: Chun-Hsiao Yeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-05T18:56:51.289Z
paper_authors._id: 6499eca0685215f7247bd5ce
paper_authors.avatarUrl: /avatars/b6fea0c33c3c930c7314b99b414072a9.svg
paper_authors.fullname: Chun-Hsiao Yeh
paper_authors.type: user
paper_authors.user: danielchyeh
paper_authors._id: 649259bba820f46b2cbf85b1
paper_authors.name: Bryan Russell
paper_authors._id: 649259bba820f46b2cbf85b2
paper_authors.name: Josef Sivic
paper_authors._id: 649259bba820f46b2cbf85b3
paper_authors.name: Fabian Caba Heilbron
paper_authors._id: 649259bba820f46b2cbf85b4
paper_authors.name: Simon Jenni
paper_title: Meta-Personalizing Vision-Language Models to Find Named Instances in
    Video
paper_summary: Large-scale vision-language models (VLM) have shown impressive results for
  language-guided search applications. While these models allow category-level
  queries, they currently struggle with personalized searches for moments in a
  video where a specific object instance such as ``My dog Biscuit'' appears. We
  present the following three contributions to address this problem. First, we
  describe a method to meta-personalize a pre-trained VLM, i.e., learning how to
  learn to personalize a VLM at test time to search in video. Our method extends
  the VLM's token vocabulary by learning novel word embeddings specific to each
  instance. To capture only instance-specific features, we represent each
  instance embedding as a combination of shared and learned global category
  features. Second, we propose to learn such personalization without explicit
  human supervision. Our approach automatically identifies moments of named
  visual instances in video using transcripts and vision-language similarity in
  the VLM's embedding space. Finally, we introduce This-Is-My, a personal video
  instance retrieval benchmark. We evaluate our approach on This-Is-My and
  DeepFashion2 and show that we obtain a 15% relative improvement over the state
  of the art on the latter dataset.
paper_ai_keywords: vision-language models (VLM) token vocabulary word embeddings instance embedding shared and learned global category features vision-language similarity personal video instance retrieval benchmark This-Is-My DeepFashion2 personalization