title: ViSpeak: Visual Instruction Feedback in Streaming Videos
summary: Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
  offline video understanding. Instead, streaming video understanding poses great
  challenges to recent models due to its time-sensitive, omni-modal and
  interactive characteristics. In this work, we aim to extend the streaming video
  understanding from a new perspective and propose a novel task named Visual
  Instruction Feedback in which models should be aware of visual contents and
  learn to extract instructions from them. For example, when users wave their
  hands to agents, agents should recognize the gesture and start conversations
  with welcome information. Thus, following instructions in visual modality
  greatly enhances user-agent interactions. To facilitate research, we define
  seven key subtasks highly relevant to visual modality and collect the
  ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
  Further, we propose the ViSpeak model, which is a SOTA streaming video
  understanding LMM with GPT-4o-level performance on various streaming video
  understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
  ViSpeak is equipped with basic visual instruction feedback ability, serving as
  a solid baseline for future research.
paper_authors._id: 67d8ded81a1b6ae91f79eb18
paper_authors.name: Shenghao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:26.291Z
paper_authors._id: 67067633351e0c16a5c27497
paper_authors.avatarUrl: /avatars/356aa3431198c8931b820a714bcfb19d.svg
paper_authors.fullname: Shenghao Fu
paper_authors.type: user
paper_authors.user: fushh7
paper_authors._id: 67d8ded81a1b6ae91f79eb19
paper_authors.name: Qize Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:23.971Z
paper_authors._id: 66a097801a26a2350395edc7
paper_authors.avatarUrl: /avatars/1e7e127cb7222df7d56e5bfda6bab519.svg
paper_authors.fullname: Qize Yang
paper_authors.type: user
paper_authors.user: PhilipC
paper_authors._id: 67d8ded81a1b6ae91f79eb1a
paper_authors.name: Yuan-Ming Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:20.167Z
paper_authors._id: 644fe6a9e1d7a97f3b66e906
paper_authors.avatarUrl: /avatars/ad1a45f0b1c8a4d03ba87f2a3ce5a8f8.svg
paper_authors.fullname: Yuanming-Li
paper_authors.type: user
paper_authors.user: Lymann
paper_authors._id: 67d8ded81a1b6ae91f79eb1b
paper_authors.name: Yi-Xing Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:18.304Z
paper_authors._id: 66b02f7405e2b2771bb431db
paper_authors.avatarUrl: /avatars/9d1ecb38e6cb2f0be33a5c1938bb1253.svg
paper_authors.fullname: Yi-Xing Peng
paper_authors.type: user
paper_authors.user: maybetomorrow
paper_authors._id: 67d8ded81a1b6ae91f79eb1c
paper_authors.name: Kun-Yu Lin
paper_authors._id: 67d8ded81a1b6ae91f79eb1d
paper_authors.name: Xihan Wei
paper_authors._id: 67d8ded81a1b6ae91f79eb1e
paper_authors.name: Jian-Fang Hu
paper_authors._id: 67d8ded81a1b6ae91f79eb1f
paper_authors.name: Xiaohua Xie
paper_authors._id: 67d8ded81a1b6ae91f79eb20
paper_authors.name: Wei-Shi Zheng
paper_title: ViSpeak: Visual Instruction Feedback in Streaming Videos
paper_summary: Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
  offline video understanding. Instead, streaming video understanding poses great
  challenges to recent models due to its time-sensitive, omni-modal and
  interactive characteristics. In this work, we aim to extend the streaming video
  understanding from a new perspective and propose a novel task named Visual
  Instruction Feedback in which models should be aware of visual contents and
  learn to extract instructions from them. For example, when users wave their
  hands to agents, agents should recognize the gesture and start conversations
  with welcome information. Thus, following instructions in visual modality
  greatly enhances user-agent interactions. To facilitate research, we define
  seven key subtasks highly relevant to visual modality and collect the
  ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
  Further, we propose the ViSpeak model, which is a SOTA streaming video
  understanding LMM with GPT-4o-level performance on various streaming video
  understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
  ViSpeak is equipped with basic visual instruction feedback ability, serving as
  a solid baseline for future research.
paper_ai_summary: A novel Visual Instruction Feedback task is introduced for streaming video understanding, using the ViSpeak model with GPT-4o-level performance on relevant benchmarks.
paper_ai_keywords: Large Multi-modal Models streaming video understanding visual instruction feedback ViSpeak-Instruct dataset ViSpeak-Bench ViSpeak model