title: Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
    Large Language Models
summary: Video Large Language Models (Video-LLMs) have demonstrated remarkable
  capabilities in coarse-grained video understanding, however, they struggle with
  fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
  a novel Video-LLM adept at perceiving and reasoning over specific video moments
  in a fine-grained manner. We identify that current Video-LLMs have limitations
  for fine-grained video understanding since they lack effective temporal
  modeling and timestamp representation. In light of this, we sharpen our model
  by incorporating (1) an additional temporal stream to encode the relationships
  between frames and (2) discrete temporal tokens enriched with specific time
  knowledge to represent timestamps. To optimize the training of
  Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
  simple video-captioning tasks and progressively introducing video temporal
  grounding tasks of increasing complexity. To further enhance
  Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
  VideoQA dataset by an automatic annotation pipeline. Extensive experiments
  demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
  tasks such as temporal sentence grounding, dense video captioning, and grounded
  VideoQA, but also shows great potential as a versatile video assistant for
  general video understanding.
paper_authors._id: 6703aeb7400ad7197be3fcdf
paper_authors.name: Haibo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T09:08:43.190Z
paper_authors._id: 63fee47352441fe3e87b5088
paper_authors.avatarUrl: /avatars/c1df1899e3925aa6fdfc8ee0049fa8a7.svg
paper_authors.fullname: WANG HAIBO
paper_authors.type: user
paper_authors.user: WHB139426
paper_authors._id: 6703aeb7400ad7197be3fce0
paper_authors.name: Zhiyang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:07:21.689Z
paper_authors._id: 64b6c686cf5117d7962d8f62
paper_authors.avatarUrl: /avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
paper_authors.fullname: Zhiyang Xu
paper_authors.type: user
paper_authors.user: Zhiyang03
paper_authors._id: 6703aeb7400ad7197be3fce1
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T16:26:25.964Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 6703aeb7400ad7197be3fce2
paper_authors.name: Shizhe Diao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:07:42.547Z
paper_authors._id: 633bd54b00732349209a18fe
paper_authors.avatarUrl: /avatars/150aa5b8d9bcca24366402932bf2d049.svg
paper_authors.fullname: Shizhe Diao
paper_authors.type: user
paper_authors.user: shizhediao
paper_authors._id: 6703aeb7400ad7197be3fce3
paper_authors.name: Yufan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:08:13.416Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 6703aeb7400ad7197be3fce4
paper_authors.name: Yixin Cao
paper_authors._id: 6703aeb7400ad7197be3fce5
paper_authors.name: Qifan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:09:34.090Z
paper_authors._id: 64b65c84c991daa61404abd7
paper_authors.avatarUrl: /avatars/3592aa8774ea4e2836099e0701b4b78d.svg
paper_authors.fullname: WangQifan
paper_authors.type: user
paper_authors.user: Bayees
paper_authors._id: 6703aeb7400ad7197be3fce6
paper_authors.name: Weifeng Ge
paper_authors._id: 6703aeb7400ad7197be3fce7
paper_authors.name: Lifu Huang
paper_title: Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
    Large Language Models
paper_summary: Video Large Language Models (Video-LLMs) have demonstrated remarkable
  capabilities in coarse-grained video understanding, however, they struggle with
  fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
  a novel Video-LLM adept at perceiving and reasoning over specific video moments
  in a fine-grained manner. We identify that current Video-LLMs have limitations
  for fine-grained video understanding since they lack effective temporal
  modeling and timestamp representation. In light of this, we sharpen our model
  by incorporating (1) an additional temporal stream to encode the relationships
  between frames and (2) discrete temporal tokens enriched with specific time
  knowledge to represent timestamps. To optimize the training of
  Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
  simple video-captioning tasks and progressively introducing video temporal
  grounding tasks of increasing complexity. To further enhance
  Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
  VideoQA dataset by an automatic annotation pipeline. Extensive experiments
  demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
  tasks such as temporal sentence grounding, dense video captioning, and grounded
  VideoQA, but also shows great potential as a versatile video assistant for
  general video understanding.
paper_ai_summary: Grounded-VideoLLM enhances fine-grained temporal understanding in videos by adding a temporal stream and discrete temporal tokens, demonstrating superior performance in tasks like temporal sentence grounding and grounded VideoQA.
paper_ai_keywords: Video Large Language Models Grounded-VideoLLM temporal modeling timestamp representation temporal stream discrete temporal tokens multi-stage training grounded VideoQA