title: VLog: Video-Language Models by Generative Retrieval of Narration
    Vocabulary
summary: Human daily activities can be concisely narrated as sequences of routine
  events (e.g., turning off an alarm) in video streams, forming an event
  vocabulary. Motivated by this, we introduce VLog, a novel video understanding
  framework that define video narrations as vocabulary, going beyond the typical
  subword vocabularies in existing generative video-language models. Built on the
  lightweight language model GPT-2, VLog feature three key innovations: (i) A
  generative retrieval model, marrying language model's complex reasoning
  capabilities with contrastive retrieval's efficient similarity search. (ii) A
  hierarchical vocabulary derived from large-scale video narrations using our
  narration pair encoding algorithm, enabling efficient indexing of specific
  events (e.g., cutting a tomato) by identifying broader scenarios (e.g.,
  kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary
  update strategy leveraging generative models to extend the vocabulary for novel
  events encountered during inference. To validate our approach, we introduce
  VidCap-Eval, a development set requiring concise narrations with reasoning
  relationships (e.g., before and after). Experiments on EgoSchema, COIN, and
  HiREST further demonstrate the effectiveness of VLog, highlighting its ability
  to generate concise, contextually accurate, and efficient narrations, offering
  a novel perspective on video understanding. Codes are released at
  https://github.com/showlab/VLog.
paper_authors._id: 67d27a9117d5cbff7c2519f6
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:23:53.060Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 67d27a9117d5cbff7c2519f7
paper_authors.name: Mike Zheng Shou
paper_title: VLog: Video-Language Models by Generative Retrieval of Narration
    Vocabulary
paper_summary: Human daily activities can be concisely narrated as sequences of routine
  events (e.g., turning off an alarm) in video streams, forming an event
  vocabulary. Motivated by this, we introduce VLog, a novel video understanding
  framework that define video narrations as vocabulary, going beyond the typical
  subword vocabularies in existing generative video-language models. Built on the
  lightweight language model GPT-2, VLog feature three key innovations: (i) A
  generative retrieval model, marrying language model's complex reasoning
  capabilities with contrastive retrieval's efficient similarity search. (ii) A
  hierarchical vocabulary derived from large-scale video narrations using our
  narration pair encoding algorithm, enabling efficient indexing of specific
  events (e.g., cutting a tomato) by identifying broader scenarios (e.g.,
  kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary
  update strategy leveraging generative models to extend the vocabulary for novel
  events encountered during inference. To validate our approach, we introduce
  VidCap-Eval, a development set requiring concise narrations with reasoning
  relationships (e.g., before and after). Experiments on EgoSchema, COIN, and
  HiREST further demonstrate the effectiveness of VLog, highlighting its ability
  to generate concise, contextually accurate, and efficient narrations, offering
  a novel perspective on video understanding. Codes are released at
  https://github.com/showlab/VLog.
paper_ai_summary: VLog, a generative video understanding framework, enhances vocabulary-based models by integrating generative retrieval, hierarchical scenario-based vocabulary, and dynamic vocabulary updates, improving narration accuracy and efficiency.
paper_ai_keywords: generative retrieval contrastive retrieval hierarchical vocabulary narration pair encoding vocabulary update generative models VidCap-Eval EgoSchema COIN HiREST