title: Dispider: Enabling Video LLMs with Active Real-Time Interaction via
    Disentangled Perception, Decision, and Reaction
summary: Active Real-time interaction with video LLMs introduces a new paradigm for
  human-computer interaction, where the model not only understands user intent
  but also responds while continuously processing streaming video on the fly.
  Unlike offline video LLMs, which analyze the entire video before answering
  questions, active real-time interaction requires three capabilities: 1)
  Perception: real-time video monitoring and interaction capturing. 2) Decision:
  raising proactive interaction in proper situations, 3) Reaction: continuous
  interaction with users. However, inherent conflicts exist among the desired
  capabilities. The Decision and Reaction require a contrary Perception scale and
  grain, and the autoregressive decoding blocks the real-time Perception and
  Decision during the Reaction. To unify the conflicted capabilities within a
  harmonious system, we present Dispider, a system that disentangles Perception,
  Decision, and Reaction. Dispider features a lightweight proactive streaming
  video processing module that tracks the video stream and identifies optimal
  moments for interaction. Once the interaction is triggered, an asynchronous
  interaction module provides detailed responses, while the processing module
  continues to monitor the video in the meantime. Our disentangled and
  asynchronous design ensures timely, contextually accurate, and computationally
  efficient responses, making Dispider ideal for active real-time interaction for
  long-duration video streams. Experiments show that Dispider not only maintains
  strong performance in conventional video QA tasks, but also significantly
  surpasses previous online models in streaming scenario responses, thereby
  validating the effectiveness of our architecture. The code and model are
  released at https://github.com/Mark12Ding/Dispider.
paper_authors._id: 677cdc004804091804c654c6
paper_authors.name: Rui Qian
paper_authors._id: 677cdc004804091804c654c7
paper_authors.name: Shuangrui Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:47:38.104Z
paper_authors._id: 65a7c0335e79abfa2ec30c52
paper_authors.avatarUrl: /avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
paper_authors.fullname: Shuangrui Ding
paper_authors.type: user
paper_authors.user: Mar2Ding
paper_authors._id: 677cdc004804091804c654c8
paper_authors.name: Xiaoyi Dong
paper_authors._id: 677cdc004804091804c654c9
paper_authors.name: Pan Zhang
paper_authors._id: 677cdc004804091804c654ca
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:19.447Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 677cdc004804091804c654cb
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:47:16.734Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 677cdc004804091804c654cc
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:47:00.823Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 677cdc004804091804c654cd
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:46:54.181Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: Dispider: Enabling Video LLMs with Active Real-Time Interaction via
    Disentangled Perception, Decision, and Reaction
paper_summary: Active Real-time interaction with video LLMs introduces a new paradigm for
  human-computer interaction, where the model not only understands user intent
  but also responds while continuously processing streaming video on the fly.
  Unlike offline video LLMs, which analyze the entire video before answering
  questions, active real-time interaction requires three capabilities: 1)
  Perception: real-time video monitoring and interaction capturing. 2) Decision:
  raising proactive interaction in proper situations, 3) Reaction: continuous
  interaction with users. However, inherent conflicts exist among the desired
  capabilities. The Decision and Reaction require a contrary Perception scale and
  grain, and the autoregressive decoding blocks the real-time Perception and
  Decision during the Reaction. To unify the conflicted capabilities within a
  harmonious system, we present Dispider, a system that disentangles Perception,
  Decision, and Reaction. Dispider features a lightweight proactive streaming
  video processing module that tracks the video stream and identifies optimal
  moments for interaction. Once the interaction is triggered, an asynchronous
  interaction module provides detailed responses, while the processing module
  continues to monitor the video in the meantime. Our disentangled and
  asynchronous design ensures timely, contextually accurate, and computationally
  efficient responses, making Dispider ideal for active real-time interaction for
  long-duration video streams. Experiments show that Dispider not only maintains
  strong performance in conventional video QA tasks, but also significantly
  surpasses previous online models in streaming scenario responses, thereby
  validating the effectiveness of our architecture. The code and model are
  released at https://github.com/Mark12Ding/Dispider.
paper_ai_keywords: real-time interaction video LLMs human-computer interaction user intent streaming video proactive interaction autoregressive decoding Dispider lightweight proactive streaming video processing module asynchronous interaction module disentangled design contextually accurate responses computation efficiency