title: VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video
    Comprehension with Video-Text Duet Interaction Format
summary: Recent researches on video large language models (VideoLLM) predominantly
  focus on model architectures and training datasets, leaving the interaction
  format between the user and the model under-explored. In existing works, users
  often interact with VideoLLMs by using the entire video and a query as input,
  after which the model generates a response. This interaction format constrains
  the application of VideoLLMs in scenarios such as live-streaming comprehension
  where videos do not end and responses are required in a real-time manner, and
  also results in unsatisfactory performance on time-sensitive tasks that
  requires localizing video segments. In this paper, we focus on a video-text
  duet interaction format. This interaction format is characterized by the
  continuous playback of the video, and both the user and the model can insert
  their text messages at any position during the video playback. When a text
  message ends, the video continues to play, akin to the alternative of two
  performers in a duet. We construct MMDuetIT, a video-text training dataset
  designed to adapt VideoLLMs to video-text duet interaction format. We also
  introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to
  benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT,
  MMDuet demonstrates that adopting the video-text duet interaction format
  enables the model to achieve significant improvements in various time-sensitive
  tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights
  highlight detection and 25% R@0.5 on Charades-STA temporal video grounding)
  with minimal training efforts, and also enable VideoLLMs to reply in a
  real-time manner as the video plays. Code, data and demo are available at:
  https://github.com/yellow-binary-tree/MMDuet.
paper_authors._id: 6747ea29d59bc71f133e8d0a
paper_authors.name: Yueqian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:22:04.153Z
paper_authors._id: 643672220b2c0d86d0b0f639
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643672220b2c0d86d0b0f639/8nqZku8HxsSL_zR3XJPqK.png
paper_authors.fullname: Yueqian Wang
paper_authors.type: user
paper_authors.user: wangyueqian
paper_authors._id: 6747ea29d59bc71f133e8d0b
paper_authors.name: Xiaojun Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:44:57.246Z
paper_authors._id: 676bd26f295f85d93ec5dc16
paper_authors.avatarUrl: /avatars/b8c00e5b9730c735b9ff3af12ed921be.svg
paper_authors.fullname: Meng Jason
paper_authors.type: user
paper_authors.user: mengxj08
paper_authors._id: 6747ea29d59bc71f133e8d0c
paper_authors.name: Yuxuan Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-28T07:16:32.123Z
paper_authors._id: 60b9e6837946aff342f734ae
paper_authors.avatarUrl: /avatars/a711a6aa35757dfd7b78b26098a964fc.svg
paper_authors.fullname: Yuxuan Wang
paper_authors.type: user
paper_authors.user: ColorfulAI
paper_authors._id: 6747ea29d59bc71f133e8d0d
paper_authors.name: Jianxin Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:22:17.700Z
paper_authors._id: 64d245829db917d3f0a256d2
paper_authors.avatarUrl: /avatars/f7821992daa8d3a587b45b2ecbc4304e.svg
paper_authors.fullname: jianxin liang
paper_authors.type: user
paper_authors.user: ljx2000
paper_authors._id: 6747ea29d59bc71f133e8d0e
paper_authors.name: Jiansheng Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:22:30.128Z
paper_authors._id: 6348ad7826a24bfaf4ae7924
paper_authors.avatarUrl: /avatars/6af4ebc10cf6dffa18efc859144da841.svg
paper_authors.fullname: Wei JianSheng
paper_authors.type: user
paper_authors.user: jiansheng80
paper_authors._id: 6747ea29d59bc71f133e8d0f
paper_authors.name: Huishuai Zhang
paper_authors._id: 6747ea29d59bc71f133e8d10
paper_authors.name: Dongyan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:22:53.269Z
paper_authors._id: 66389190596ff1c3447cb88b
paper_authors.avatarUrl: /avatars/e1eabfa6892810a830d8a42c1b5ee928.svg
paper_authors.fullname: zhao dongyang
paper_authors.type: user
paper_authors.user: yang2016
paper_title: VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video
    Comprehension with Video-Text Duet Interaction Format
paper_summary: Recent researches on video large language models (VideoLLM) predominantly
  focus on model architectures and training datasets, leaving the interaction
  format between the user and the model under-explored. In existing works, users
  often interact with VideoLLMs by using the entire video and a query as input,
  after which the model generates a response. This interaction format constrains
  the application of VideoLLMs in scenarios such as live-streaming comprehension
  where videos do not end and responses are required in a real-time manner, and
  also results in unsatisfactory performance on time-sensitive tasks that
  requires localizing video segments. In this paper, we focus on a video-text
  duet interaction format. This interaction format is characterized by the
  continuous playback of the video, and both the user and the model can insert
  their text messages at any position during the video playback. When a text
  message ends, the video continues to play, akin to the alternative of two
  performers in a duet. We construct MMDuetIT, a video-text training dataset
  designed to adapt VideoLLMs to video-text duet interaction format. We also
  introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to
  benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT,
  MMDuet demonstrates that adopting the video-text duet interaction format
  enables the model to achieve significant improvements in various time-sensitive
  tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights
  highlight detection and 25% R@0.5 on Charades-STA temporal video grounding)
  with minimal training efforts, and also enable VideoLLMs to reply in a
  real-time manner as the video plays. Code, data and demo are available at:
  https://github.com/yellow-binary-tree/MMDuet.
paper_ai_summary: VideoLLMs using a video-text duet interaction format achieve improvements in time-sensitive tasks and real-time response capabilities with minimal training effort.
paper_ai_keywords: video-text duet MMDuetIT Multi-Answer Grounded Video Question Answering (MAGQA) YouCook2 dense video captioning QVHighlights highlight detection Charades-STA temporal video grounding